fix(agent): adoption cannot hide a node; Config Import on fresh installs (v1.11.1)

Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.

DISCOVERY REPORTS THAT NEVER REACHED THE SERVER

The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.

  - the report is cached only on a 2xx;
  - GET /agents/{name}/keepalived-config now reports whether the server actually
    holds a discovery for that agent, and the cache may only suppress while it
    says yes - which is what lets nodes stuck from earlier releases recover on
    their own, with nobody touching them;
  - the flag is parsed with has() + tostring, not `// empty`: jq's alternative
    operator returns the alternative for **false** as well as null, so the naive
    form could not tell "no record" from "older backend" and the recovery would
    have been completely inert;
  - a 400/413/422 records the refusal so identical bytes are not re-posted
    forever - 4xx and 5xx agent calls are never sampled out of the request log,
    so an unattended loop would write a row carrying the whole config every
    cycle - while 401 and 404 keep retrying, because here they mean a token
    rotation or an agent row briefly absent, not a bad payload;
  - the CLEAR path had the same exit-code defect, where it left a stale row
    offering a managed node for adoption with nothing to ever retry it.

CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS

check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.

DETERMINISTIC CONFIG PATH

A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.

Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.

Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.

.env.template

@@ -17,11 +17,34 @@ REDIS_URL=redis://redis:6379
 # Change this to a strong random string in production
 SECRET_KEY=your-secret-key-change-this-in-production
 
-# Optional: dedicated Fernet key for encrypting VRRP secrets of HA/VIP (Issue #27).
-# If unset, it is derived from SECRET_KEY (HKDF), exactly like MFA. Set an explicit
-# key (urlsafe-base64, 32 bytes) in production if you want independent key rotation.
+# ----------------------------------------------------------------------------
+# Optional per-purpose encryption keys.
+#
+# Every secret the application stores is encrypted at rest with Fernet. Each class
+# derives its own key, so rotating one never affects another. If a variable below is
+# unset, that class's key is derived from SECRET_KEY via HKDF — which works, but means
+# rotating SECRET_KEY makes the existing values of that class UNDECRYPTABLE. Set an
+# explicit key (urlsafe-base64, 32 bytes) in production if you want independent
+# rotation. Generate one with:
+#   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
+# ----------------------------------------------------------------------------
+
+# VRRP secrets for HA/VIP (Issue #27).
 # VIP_ENCRYPTION_KEY=
 
+# TOTP secrets for multi-factor authentication (Issue #18).
+# MFA_ENCRYPTION_KEY=
+
+# Per-account DNS provider credentials for ACME DNS-01 (Issue #35).
+# Rotating this without re-entering credentials makes DNS-01 renewals fail until the
+# affected accounts' credentials are re-saved in ACME Automation.
+# DNS_PROVIDER_ENCRYPTION_KEY=
+
+# Private keys of PENDING CSRs, held only until the signed certificate is imported
+# (Issue #53). Rotating this while CSRs are out for signature makes those CSRs
+# unusable — they must be deleted and re-created.
+# CSR_ENCRYPTION_KEY=
+
 # ============================================================================
 # PUBLIC URL CONFIGURATION
 # ============================================================================
@@ -36,6 +59,13 @@ PUBLIC_URL=http://localhost:8000
 
 # Management base URL (defaults to PUBLIC_URL if not set)
 # Override this if your management interface is on a different URL
+#
+# This is also the last-resort fallback for the ACME HTTP-01 challenge backend,
+# i.e. the address written into haproxy.cfg as `server _acme_mgmt <host>:<port>`.
+# That address is resolved BY HAPROXY, ON THE HAPROXY NODE. If HAProxy runs
+# anywhere other than this machine, localhost points at the wrong box and HTTP-01
+# validation fails while DNS-01 keeps working. Set a routable address, with the
+# port, e.g. MANAGEMENT_BASE_URL=http://10.90.1.4:8080
 MANAGEMENT_BASE_URL=http://localhost:8000
 
 # ============================================================================
@@ -59,6 +89,15 @@ AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
 # Config sync interval in seconds
 AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 
+# ============================================================================
+# BACKEND PERFORMANCE
+# ============================================================================
+# Number of uvicorn worker processes for the backend API (default: 1).
+# On multi-core hosts, setting this to the core count (e.g. 2) lets the API
+# use all cores. Safe to increase: background tasks are multi-replica safe
+# (the k8s deployment already runs 2+ replicas via HPA).
+UVICORN_WORKERS=1
+
 # ============================================================================
 # CORS CONFIGURATION
 # ============================================================================
@@ -67,6 +106,44 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 # Example: http://haproxy-manager.example.com,http://localhost:8080
 CORS_ORIGINS=
 
+# ============================================================================
+# REQUEST / RESPONSE LOG (v1.11.0)
+# ============================================================================
+# Records every inbound API call and every outbound HTTP call the backend makes
+# (ACME, DNS providers, agents) into the `request_logs` table, browsable under
+# "Request Log" in the UI.
+#
+# Only the four settings below are environment-level, because they decide
+# whether the middleware is registered at all and how much memory the writer
+# queue may hold. Everything an operator tunes day to day — retention windows,
+# body capture, sampling, excluded paths — lives in the database and is edited
+# in Settings -> Request Log.
+
+# Hard kill-switch. When false the logging middleware is NEVER added to the ASGI
+# stack and neither the writer nor the retention task starts: zero overhead, not
+# even a settings lookup. Requires a restart to change.
+# (The `enabled` toggle in Settings is the no-restart equivalent.)
+REQUEST_LOG_ENABLED=true
+
+# Per-worker in-process queue depth. When it fills, rows are DROPPED and counted
+# rather than blocking the request — the drop count is shown on the Request Log
+# page (per worker).
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Hard memory ceiling for that same queue, per worker. The row count above does
+# NOT bound memory on its own: `requestlog.max_body_bytes` is editable from
+# Settings up to 256 KB and a row can carry it twice, so at that ceiling a
+# 2000-row queue would hold ~1 GiB. Whichever limit is reached first stops the
+# queue. Raise this if you raise REQUEST_LOG_QUEUE_MAX.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Rows per batched INSERT. One connection is taken from the pool per batch, not
+# per request.
+REQUEST_LOG_BATCH_SIZE=100
+
+# Maximum wait before a partial batch is flushed, in milliseconds.
+REQUEST_LOG_FLUSH_MS=500
+
 # ============================================================================
 # FRONTEND CONFIGURATION (React)
 # ============================================================================

.github/workflows/docker-build.yml

@@ -21,27 +21,17 @@ jobs:
       - name: read product version
         id: prodversion
         run: |
-          VERSION=$(jq -r .version version.json)
+          VERSION=$(jq -r .version backend/version.json)
           if [ -z "$VERSION" ] || [ "$VERSION" = "null" ]; then
-            echo "Failed to read product version from version.json" >&2
+            echo "Failed to read product version from backend/version.json" >&2
             exit 1
           fi
           echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
 
-      # The backend image is built with `context: ./backend`, so the
-      # repo-root version.json is OUTSIDE the build context and never
-      # reaches the container. Backend `main.py` falls back to a
-      # compile-time constant when /app/version.json is missing, which
-      # caused a real production drift: a redeploy of the v1.5.2 tree
-      # silently still reported "v1.5.0" in `/api/version` because the
-      # constant in main.py had been bumped but the file was not
-      # available to read. Stage version.json into the backend
-      # context here so the canonical file IS shipped and the
-      # constant only serves as a defensive fallback. The staged file
-      # is gitignored to keep `git status` clean for developers.
-      - name: stage version.json into backend build context
-        run: cp version.json backend/version.json
-
+      # version.json now lives at backend/version.json (inside the ./backend build
+      # context), so `COPY . .` bakes it into the image directly — no staging step
+      # is needed and the backend reports the correct version in every deployment,
+      # not just this workflow's builds.
       - name: set up qemu
         uses: docker/setup-qemu-action@v3
 
@@ -91,7 +81,7 @@ jobs:
         run: |
           VERSION="${{ steps.prodversion.outputs.VERSION }}"
           TAG="v${VERSION}"
-          RELEASE_NAME=$(jq -r '.releaseName // empty' version.json)
+          RELEASE_NAME=$(jq -r '.releaseName // empty' backend/version.json)
           if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
             echo "Release $TAG already exists, skipping."
           else

.gitignore

@@ -39,11 +39,6 @@ venv.bak/
 *.sqlite
 *.sqlite3
 
-# Build-time staged version.json (CI `cp version.json backend/`).
-# The canonical file lives at repo root; this path is a transient
-# copy for the backend Docker build context.
-backend/version.json
-
 # IDE
 .vscode/
 .idea/

CONFIG.md

@@ -131,6 +131,66 @@ if (window.location) {
 }
 ```
 
+### REQUEST_LOG_ENABLED (v1.11.0)
+
+**Ne İşe Yarar**: Request/Response Log özelliğinin sert (hard) kill-switch'i. `false` yapıldığında
+loglama middleware'i ASGI zincirine **hiç eklenmez**, yazıcı ve retention görevleri başlatılmaz —
+yani sıfır ek yük, ayar okuması bile yapılmaz. Değişiklik için restart gerekir.
+
+**Örnekler**:
+```bash
+# Varsayılan: açık
+REQUEST_LOG_ENABLED=true
+
+# Tamamen kapat (ör. çok yüksek trafikli kurulum, veya regülasyon gereği)
+REQUEST_LOG_ENABLED=false
+```
+
+**Nasıl Kullanılır**:
+1. Restart gerektirmeden kapatmak isterseniz bunun yerine **Settings → Request Log → Enable request
+   log** anahtarını kullanın; o anında etkili olur.
+2. Retention süreleri, gövde (body) yakalama, örnekleme oranı ve hariç tutulan path'ler bu env
+   değişkeniyle değil, veritabanındaki `requestlog.*` ayarlarıyla yönetilir — arayüzden düzenlenir.
+3. Disk büyümesi asıl operasyonel konudur: sırasıyla `sample_rate`'i düşürün, `capture_get`'i
+   kapatın, `capture_bodies`'i kapatın, sonra `success_retention_days`'i kısaltın.
+
+### REQUEST_LOG_QUEUE_MAX / REQUEST_LOG_QUEUE_MAX_BYTES / REQUEST_LOG_BATCH_SIZE / REQUEST_LOG_FLUSH_MS (v1.11.0)
+
+**Ne İşe Yarar**: Log satırlarını yazan arka plan görevinin ayarları. Satırlar sınırlı bir kuyruğa
+konur ve toplu (batch) INSERT ile yazılır; böylece istek yolu asla veritabanını beklemez.
+
+**Örnekler**:
+```bash
+# Worker başına kuyruk derinliği. Dolduğunda satırlar DÜŞÜRÜLÜR (sayılır ve
+# Request Log sayfasında gösterilir), istek bloklanmaz.
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Aynı kuyruğun BAYT tavanı (worker başına). Satır sayısı tek başına belleği
+# sınırlamaz: `max_body_bytes` Settings'ten 256 KB'a kadar ayarlanabilir ve bir
+# satır bunu iki kez taşıyabilir, o tavanda 2000 satırlık kuyruk ~1 GiB tutar.
+# Hangi sınır önce dolarsa kuyruk orada durur.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Tek INSERT'te kaç satır yazılacağı (havuzdan istek başına değil, batch başına
+# bir bağlantı alınır)
+REQUEST_LOG_BATCH_SIZE=100
+
+# Yarım dolu bir batch'in en fazla ne kadar bekletileceği (ms)
+REQUEST_LOG_FLUSH_MS=500
+```
+
+**Nasıl Kullanılır**:
+1. Request Log sayfasında "rows dropped" uyarısı görüyorsanız önce `requestlog.max_body_bytes`
+   veya `sample_rate`'i düşürün. `REQUEST_LOG_QUEUE_MAX`'ı artırmak bu worker'ın tutabileceği
+   belleği de artırır; artıracaksanız `REQUEST_LOG_QUEUE_MAX_BYTES`'ı da birlikte artırın.
+2. Bu değerler worker başınadır — `UVICORN_WORKERS` arttıkça toplam bellek de o oranda artar.
+   `GET /api/request-logs/stats` içindeki sayaçlar da worker başınadır ve yanıtta öyle
+   etiketlenir; 4 worker'da gördüğünüz düşüş sayısı gerçeğin dörtte biridir.
+3. Büyük filolarda tek en etkili ayar `requestlog.capture_agent_success`'tir (varsayılan kapalı).
+   Açık olsaydı 200 düğümlük bir filo günde ~2M satır yazar ve 500.000 satır tavanına 6 saatte
+   ulaşırdı; yapılandırılmış 7 gün / 30 gün saklama o noktada birkaç saate iner. Başarısız ajan
+   çağrıları bu ayardan bağımsız olarak her zaman loglanır.
+
 ## 🚀 Deployment Senaryoları
 
 ### Docker Compose

README.md

@@ -105,8 +105,9 @@ This architecture provides better security (no inbound connections to HAProxy se
 ✅ **Version Control & Rollback** - Every change versioned with one-click restore capability  
 ✅ **Real-Time Monitoring** - Live stats, health checks, and performance dashboards  
 ✅ **SSL Certificate Management** - Centralized SSL with expiration tracking  
+✅ **CSR Creation** *(v1.9.0)* - Generate a private key + CSR in-app (RSA 2048/4096, ECDSA P-256/P-384, full subject + SANs), have it signed by any external CA, then import the signed certificate — the key never leaves the server  
 ✅ **ACME Auto SSL (Let's Encrypt)** - Automated certificate issuance, renewal, and deployment via ACME protocol  
-✅ **ACME DNS-01 Challenge** *(v1.8.0)* - TXT-record validation for internal/isolated clusters (no public port 80) and wildcard certificates; pluggable DNS providers (Manual + Cloudflare), opt-in, HTTP-01 unchanged  
+✅ **ACME DNS-01 Challenge** *(v1.8.0)* - TXT-record validation for internal/isolated clusters (no public port 80) and wildcard certificates; pluggable DNS providers (Manual + Cloudflare + GoDaddy *(v1.10.0)*), opt-in, HTTP-01 unchanged  
 ✅ **ACME Certificate Diagnostic Panel** - Automated preflight that checks agent readiness, DNS resolution, port 80 reachability, and ACME challenge ACL before issuing certificates  
 ✅ **WAF Rules** - Web Application Firewall management and deployment  
 ✅ **Agent Script Versioning** - Update agents via UI (Monaco editor) with auto-upgrade  
@@ -115,6 +116,7 @@ This architecture provides better security (no inbound connections to HAProxy se
 ✅ **HA / VIP (Keepalived) Management** - Create virtual IPs from the UI; the agent installs & configures Keepalived (unicast VRRP) with a HAProxy health-check so the VIP fails over automatically; live MASTER/BACKUP detection per node  
 ✅ **Role-Based User Management** - Admin and user roles with granular permissions and access control  
 ✅ **User Activity Audit Logs** - Complete audit trail of all system events  
+✅ **Request/Response Log** *(v1.11.0)* - Every inbound API call (GETs and errors included) and every outbound HTTP call the backend makes (ACME, Cloudflare, GoDaddy, agents) in one filterable timeline, with redacted, size-capped bodies and operator-configurable retention  
 ✅ **REST API** - Full programmatic access for automation and CI/CD integration
 
 
@@ -255,7 +257,7 @@ This architecture provides better security (no inbound connections to HAProxy se
 - **Stuck Order Detection** *(v1.4.0)*: Setup wizard surfaces orders that the CA has validated but not yet downloaded, with one-click `Complete` action and automatic 60-second retry
 - **Multi-Provider Support**: Configurable ACME directory URL supports Let's Encrypt, ZeroSSL, Google Trust Services, Buypass, and custom CAs
 - **HTTP-01 Challenge**: Built-in challenge responder with automatic HAProxy routing injection; reserved backend name `_acme_challenge_backend` is auto-managed and protected from manual edits / agent sync collisions
-- **DNS-01 Challenge** *(v1.8.0 — Issue #35)*: Validate via a DNS TXT record instead of HTTP on port 80, for **internal/isolated clusters with no public ingress** and for **wildcard** certificates (`*.example.com`). Pluggable per-account DNS provider (Manual + Cloudflare to start; credentials encrypted at rest and verified on save), same PENDING → APPLIED pipeline, bounded automatic retry on propagation lag, and a DNS-01 event timeline. Opt-in via a global setting; HTTP-01 behaviour is unchanged. (See the *DNS-01 Challenge* subsection under ACME Auto SSL below.)
+- **DNS-01 Challenge** *(v1.8.0 — Issue #35)*: Validate via a DNS TXT record instead of HTTP on port 80, for **internal/isolated clusters with no public ingress** and for **wildcard** certificates (`*.example.com`). Pluggable per-account DNS provider (Manual + Cloudflare + GoDaddy *(v1.10.0)*; credentials encrypted at rest and verified on save), same PENDING → APPLIED pipeline, bounded automatic retry on propagation lag, and a DNS-01 event timeline. Opt-in via a global setting; HTTP-01 behaviour is unchanged. (See the *DNS-01 Challenge* subsection under ACME Auto SSL below.)
 - **ACME Account Management**: Register, view, and deactivate ACME accounts from the UI
 - **Staging Mode**: Test certificate issuance with Let's Encrypt staging environment before production
 - **Custom Staging Endpoint** *(v1.4.0)*: Optional `staging_url_override` setting lets you point staging mode at a private ACME test CA (e.g. Pebble) without touching the production directory URL
@@ -778,6 +780,15 @@ User Updates SSL in UI → All Agents Poll Backend (30s)
   → Validate Config → Reload HAProxy (zero downtime)
 ```
 
+#### CSR Workflow (external / corporate CAs) — v1.9.0
+
+For certificates signed by an external or corporate CA, the **CSR tab** on the SSL Certificates page covers the whole flow without the private key ever leaving the server:
+
+1. **Create CSR**: pick a name (becomes the certificate name / on-agent file path), Common Name, optional SANs and subject fields (O/OU/L/ST/C/email), and a key algorithm (RSA 2048/4096 or ECDSA P-256/P-384). The backend generates the key + CSR; only the CSR PEM is shown (copy or download as `.csr`).
+2. **Get it signed**: submit the CSR to your Certificate Authority.
+3. **Import**: paste the signed certificate (+ optional chain), choose Global or cluster-specific scope and usage type. The backend verifies the certificate matches the stored key, rejects expired certs, warns on SAN drift, and creates a normal SSL certificate entry (source: `CSR`).
+4. **Deploy**: the imported certificate goes through the standard **PENDING → Apply Management → agent pull** pipeline like any other certificate.
+
 #### Key Features
 - **Certificate Upload**: PEM format certificate and private key upload
 - **ACME Automation**: Automatic certificate issuance and renewal via Let's Encrypt / ACME protocol (see [ACME Auto SSL](#acme-auto-ssl---automated-certificate-management))
@@ -979,9 +990,9 @@ DNS-01 is **opt-in** and fully backward compatible: it is disabled until an admi
 
 - **Enable it**: Settings → ACME / SSL Automation → **DNS-01 Challenge (advanced)** → turn on *Enable DNS-01 Challenge* and Save. While off, DNS-01 options are hidden and no DNS-01 orders can be created.
 - **Per-account provider**: in ACME Automation, create (or reconfigure) an ACME account with **Challenge Method = DNS-01** and a **DNS Provider**. Provider credentials are **verified before saving** and **encrypted at rest** (Fernet, mirroring the VRRP/MFA secret pattern); they are never returned by the API or written to logs.
-- **Supported providers**: **Manual** (publish the TXT record yourself in any DNS — including fully internal DNS — then click *Verify*; works everywhere but cannot auto-renew unattended) and **Cloudflare** (API token with `Zone:DNS:Edit` + `Zone:Read`; the TXT record is created and cleaned up automatically and renews unattended). The provider interface is pluggable — more providers can be added without changing the issuance flow.
+- **Supported providers**: **Manual** (publish the TXT record yourself in any DNS — including fully internal DNS — then click *Verify*; works everywhere but cannot auto-renew unattended), **Cloudflare** (API token with `Zone:DNS:Edit` + `Zone:Read`; the TXT record is created and cleaned up automatically and renews unattended), and **GoDaddy** *(v1.10.0)* (a **Production** API Key + Secret pair from `developer.godaddy.com/keys` — the first key that dashboard issues is an OTE/test key and is rejected; the zone must be in the same GoDaddy account, which needs at least one registered domain before GoDaddy allows DNS API access at all. A **Personal Access Token** works too: paste it as the API Key and leave the Secret blank — that is the forward path as GoDaddy retires the `sso-key` scheme. TXT records are created and cleaned up automatically and renew unattended). The provider interface is pluggable — more providers can be added without changing the issuance flow.
 - **Same pipeline**: after validation the certificate follows the normal PENDING → APPLIED flow (assign to clusters / Apply Management) and the agent serves it — identical to HTTP-01 from finalize onward, with **zero agent or rendered-config changes** for DNS-01.
-- **Manual flow**: the order detail shows the exact `_acme-challenge.<domain>` record name + TXT value (copyable); publish it and click *I've added the records — Verify*. For Cloudflare it is automatic.
+- **Manual flow**: the order detail shows the exact `_acme-challenge.<domain>` record name + TXT value (copyable); publish it and click *I've added the records — Verify*. For Cloudflare and GoDaddy it is automatic.
 - **Resilience**: a propagation-lag failure is recovered by a **bounded fresh-order retry chain** (1 original + 3 retries with increasing backoff, kept under Let's Encrypt's rate limits); any orphaned TXT record is cleaned up by a reconcile sweep. The order detail shows a DNS-01 event timeline (publish → validation → cleanup).
 - **Wildcards**: `*.example.com` is validated at `_acme-challenge.example.com`; it does **not** cover the apex — add `example.com` as a separate name if you need both (the providers handle the two coexisting TXT values automatically).
 - **Scope (this release)**: the Site Wizard remains HTTP-01-only; issue DNS-01 / wildcard certificates from **ACME Automation**.
@@ -1738,6 +1749,19 @@ Add new HAProxy clusters through the web interface or directly via API:
 }
 ```
 
+### Performance Tuning *(v1.8.6)*
+
+The backend API defaults to a **single uvicorn worker process**, which uses one CPU core. Agents poll the API every 30 seconds (heartbeat, config, pending-requests, upgrade checks), so larger fleets add a constant baseline load. Two ways to scale:
+
+- **Docker Compose — worker processes**: set `UVICORN_WORKERS` in your `.env` (default `1`). On a multi-core host, matching the core count (e.g. `UVICORN_WORKERS=2` on a 2-core machine) lets the API use all cores:
+  ```bash
+  echo "UVICORN_WORKERS=2" >> .env && docker-compose up -d backend
+  ```
+- **Kubernetes/OpenShift — replicas**: the shipped manifests already include an HPA for the backend (2→10 replicas, `k8s/manifests/13-hpa.yaml`); raise `minReplicas`/`maxReplicas` as needed.
+
+Both are safe: all background tasks (ACME completion, renewals, agent monitoring) are multi-replica safe by design (atomic claims via `FOR UPDATE SKIP LOCKED`, PostgreSQL advisory locks).
+
+**Diagnosing slow requests**: every API response carries an `X-Response-Time` header, and the backend logs `Slow request detected` (WARNING) for any request taking longer than 1 second — check those log lines to pinpoint slow endpoints before tuning anything else.
 
 ## API Reference
 
@@ -1843,6 +1867,42 @@ GET /api/backends?cluster_id=1
 GET /api/frontends?cluster_id=1
 ```
 
+### SSL CSR API (v1.9.0)
+```bash
+# Create a CSR (generates the private key server-side; response contains the
+# CSR PEM — the private key is never returned by any endpoint)
+POST /api/ssl/csrs
+Authorization: Bearer <token>
+{
+  "name": "www-example-com",
+  "common_name": "www.example.com",
+  "sans": ["api.example.com"],
+  "key_algorithm": "rsa-2048",   # rsa-2048 | rsa-4096 | ecdsa-p256 | ecdsa-p384
+  "organization": "Example Corp",
+  "country": "TR"
+}
+
+# List CSRs (metadata only, no PEM)
+GET /api/ssl/csrs
+
+# CSR detail (includes the CSR PEM)
+GET /api/ssl/csrs/{csr_id}
+
+# Import the CA-signed certificate for a pending CSR
+POST /api/ssl/csrs/{csr_id}/import
+{
+  "certificate_content": "-----BEGIN CERTIFICATE-----...",
+  "chain_content": "-----BEGIN CERTIFICATE-----...",   # optional
+  "usage_type": "frontend",                            # frontend | server
+  "is_global": false,
+  "cluster_ids": [1, 2]
+}
+
+# Delete a CSR (pending: permanently destroys the private key;
+# completed: removes history only — the imported certificate is unaffected)
+DELETE /api/ssl/csrs/{csr_id}
+```
+
 ### ACME / Let's Encrypt API
 ```bash
 # List ACME accounts
@@ -2069,7 +2129,7 @@ haproxy-openmanager/
 ├── docker-compose.yml                # Docker Compose configuration
 ├── docker-compose.localtest.yml      # Local development/testing overrides
 ├── docker-compose.test.yml           # Test environment
-├── version.json                      # Application version metadata
+├── backend/version.json              # Application version metadata (single source of truth)
 ├── build-images.sh                   # Build Docker images
 ├── pytest.ini                        # Pytest configuration
 ├── README.md                         # This file
@@ -2415,6 +2475,33 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.11.1** (2026-08-15) — **A node can no longer be hidden from adoption for good, and Config Import reaches a freshly installed agent**: the agent posts an unmanaged `keepalived.conf` for adoption and caches the hash of what it sent, so the file (which carries the VRRP password) is re-posted only when it changes. Delivery was judged by `curl`'s exit code, which is **0 for 5xx as well**, so a report the server *rejected* was recorded as delivered — and because a hand-maintained config does not change on its own, that node dropped out of *Unmanaged keepalived detected* permanently, curable only by deleting a cache file on the node by hand. Now: the report is cached only on a 2xx; `GET /agents/{name}/keepalived-config` tells the agent whether the server actually holds a discovery for it, so nodes stuck from earlier releases recover by themselves on the next poll; a **400/413/422** records the refusal so identical bytes are not re-posted forever (4xx and 5xx are never sampled out of the request log, so an unattended loop would write a row carrying the whole config every cycle), while **401 and 404 keep retrying** because in this system they mean a token rotation or an agent row briefly absent, not a bad payload. The same exit-code mistake in the *clear* path is fixed too, where it left a managed node still being offered for adoption. Separately, **Config Import silently did nothing on any agent that had never self-upgraded**: `check_config_requests` was defined in the installer and in the self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by `type`, so the operator asked a node for its `haproxy.cfg` and nothing arrived, with no error anywhere. And the cluster whose `keepalived_config_path` is handed to an agent is now resolved deterministically — a pool may hold several clusters, and the unordered join could return a different one between polls, pointing the agent at a file that does not exist. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema change.
+- **v1.11.0** (2026-08-14) — **Unified request/response log with configurable retention**: until now the only record of what happened was `user_activity_logs`, which stores non-GET **2xx** operations with no bodies — so when something failed you could see *that* the count went up, never *what was sent or what came back*. This release adds one queryable timeline covering **both directions**: every inbound API call (**including GETs and including 4xx/5xx**) with the user, client IP, status, duration and — redacted and size-capped — the request and response bodies; and every **outbound** HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows **inherit the inbound request's id**, so one operator action and the CA/DNS calls it triggered read as a single trace — opening a failed *Request Certificate* shows the exact `POST /acme/new-order` and the CA's `429` body underneath it. Capture is a **pure-ASGI middleware that tees** the request and response streams rather than draining them, so no downstream handler is affected (notably the raw-body agent heartbeat), and rows are written by a **batching background writer** with a bounded queue, so the request path never waits on the database and a saturated logger drops rows visibly instead of blocking. Secrets never land: headers are an allowlist (`Authorization`/`Cookie` reduced to a presence marker), body keys and value shapes are redacted (passwords, tokens, API keys, private-key PEMs, JWTs), the **ACME JWS request body is never stored** (a stored `protected`+`signature` pair is a replayable credential — a summary is logged instead), DNS-provider errors record only the exception **type**, and the ACME HTTP-01 challenge endpoint is excluded so `key_authorization` is never captured. **Retention is operator-configurable** in *Settings → Request Log*: separate day counts for successful and failed rows (defaults 7 and 30) plus a hard row cap (500 000), whichever is reached first, pruned in **batches** under a Postgres advisory lock so a multi-million-row table cannot time out the delete or have every replica scan it at once. New **Request Log** page (`requestlog.read`) and retention/purge permission (`requestlog.manage`); `super_admin` and `security_admin` get both, `operator` gets read, `viewer` gets neither. **Successful agent polls are not logged** (`capture_agent_success`, default off; failures always are), which is what keeps the table's size a function of operator activity rather than of node count: measured at 2 424 bytes/row, a 200-node fleet would otherwise write 2.0M rows/day and reach the row cap in six hours, silently reducing the configured 7-day/30-day retention to a few hours for everything in the table. Cost is measured, not estimated: 27.7 µs per request on the hot path, 18.8 µs per row on the writer task, **0.096 % of one core at 500 nodes**. Adds one new table (`request_logs`) and its settings seed — SCHEMA_VERSION 11 → 12 (not 11: that number was taken by v1.10.4 while this was in review, and the version gate would have skipped the migration entirely on every existing install), auto-migrated, no existing table altered, no agent or rendered-config change. Kill switches: `REQUEST_LOG_ENABLED=false` (environment — the middleware is then never registered and costs nothing) or the `enabled` toggle in Settings (no restart).
+- **v1.10.14** (2026-08-14) — **A converged node keeps acknowledging**: the deploy report is the server's only evidence that a member node applied its `keepalived.conf`, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return on every cycle and never reported again, so a **single lost report** — a backend restart, a 5xx, a network blip — left the VIP reading `SYNCING (0/n)` with an empty *Last ack* forever, while the node was demonstrably running the right config. Nothing would ever reconcile the two: the node was correct, the page was not, and the only way out was to change the rendered config so the agent wrote it again. The agent now re-asserts its state on the idempotent path too, which costs one request per node per ~2.5 minutes and touches nothing on the node — keepalived is not reloaded and the file is not rewritten. This is a long-standing gap from the original HA/VIP work, surfaced when acknowledgements were dropped for an unrelated reason in v1.10.12. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema or API change.
+- **v1.10.13** (2026-08-14) — **Agent deploy acknowledgements were silently dropped** (regression in v1.10.12, fix it before or with that release): the takeover-retirement clause added to `POST /agents/{name}/keepalived-status` in v1.10.12 reused one query placeholder for both the assignment `last_deploy_hash=$n` and the comparison inside its `CASE`. PostgreSQL deduces a type per **use**, so the same placeholder came out as `text` in one and `character varying` in the other, and asyncpg rejected the statement with `AmbiguousParameterError`. The failure was not partial: the whole UPDATE never ran, so **no member ever recorded an acknowledgement**. Every VIP sat at `SYNCING (0/n)` with an empty *Last ack*, even after the nodes had deployed the config successfully, and teardown acknowledgements were lost the same way. The hash is now bound to its own placeholder, which is only ever compared against the column and therefore unambiguous. Verified against a real PostgreSQL: both statements execute, a matching hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash` both leave it in place, and every case records the acknowledgement. A test now asserts every `$n` in these statements is bound exactly once and that the count matches the arguments passed. Backend only: no schema, agent or API-shape change.
+- **v1.10.12** (2026-08-14) — **A valid keepalived config is no longer rejected by its own warning**: before writing a rendered `keepalived.conf` the agent validates it with `keepalived -t` and, on failure, keeps the running config and does not restart keepalived. That fail-safe is right, but it treated **any** non-zero exit as invalid, and keepalived's config-test exit code does not separate fatal from benign. Measured on 2.2.8: a clean config exits 0, but `Truncating auth_pass to 8 characters` exits **5** and so does a missing `}` or an `Unknown keyword`. A VRRP password longer than eight characters was therefore enough to make every apply fail, including on nodes whose own running config produces the same warning and has been serving the VIP for weeks. The gate now judges the **output**: messages known to be benign are dropped and anything that remains still fails, so it fails **closed** and an unrecognised message is treated as fatal. Verified against real keepalived: a truncation warning passes while a missing brace, an unknown keyword and a `SECURITY VIOLATION` are all still refused. The agent also **reports what keepalived said** now, in the log and in the status the HA/VIP page shows; discarding it left a correct refusal with no way to act on it. Agent-script change: sync the script from Agent Management and let the agents upgrade for it to take effect. No schema or API change.
+- **v1.10.11** (2026-08-14) — **The *Adoptable* tag names the problem that actually blocks adoption**: the tag and the disabled *Adopt* button were computed separately and could disagree. A pair blocked because its peer's `keepalived.conf` could not be parsed was labelled **MASTER missing** — technically true, since the unreadable node's `state MASTER` had not been counted, but it pointed the operator at the wrong node while the real reason sat in the button's own tooltip. Both now come from one ordered decision, so the label, its colour and the tooltip always describe the condition that stops the adoption; a group held up by an unreadable or unreachable peer reads **blocked by peer**, and two MASTERs is now distinct from none. Display only: what the endpoint accepts or refuses is unchanged. On the public repo this is the first artifact carrying v1.10.4 through v1.10.10: none was released separately, because VIP adoption did not work end to end until these fixes landed.
+- **v1.10.10** (2026-08-14) — **Adoption blockers are listed once per instance**: with the instance-based panel a two-node pair reported the *same* problems about the *same* shared config twice, once per member, and the line numbers differ between the two files so plain de-duplication did not collapse them. Four issues on a pair read as eight, in both the *Adoptable* tooltip and the adopt dialog. They are now merged on the message text with the leading `line N:` ignored, so each distinct problem appears once. Display only: the endpoint already evaluated the combined set and its refusals are unchanged.
+- **v1.10.9** (2026-08-14) — **Adoption refuses to strand a node or silently normalise a peer's settings**: v1.10.8 adopted the whole VRRP instance, but it could only *match* a node it was able to read, that was enabled, and that sat in the same pool. Each of those was a door a real member of the group left through silently — the nodes that remained were rewritten while the one that left kept serving the same address from an unmanaged config. Seen on a live pool: one node of a pair had an unclosed `vrrp_instance` block, so it parsed to nothing while its partner parsed cleanly. Instead of guarding each door, adoption now asks the question directly — is there **any** reported `keepalived.conf` that mentions this virtual address and is not among the nodes being taken over — and refuses naming the node and the reason (unparseable, agent disabled, different pool). Separately, four VIP-level fields (`prefix_length`, unicast/multicast mode, HAProxy tracking and the VRRP password) are stored once and re-rendered onto **every** member, so taking them from whichever node was clicked imposed its settings on the others; the prefix length is the sharpest, because the design refuses to *guess* a netmask for a live VIP and copying one node's netmask onto another is that same change by another name. Adoption now requires the nodes to agree on all four, and compares the VRRP secret by decrypting each node's token (Fernet is non-deterministic, so the ciphertexts cannot be compared). Finally, the takeover authorisation is genuinely one-shot: `takeover_expected_hash` was written at adoption and never cleared, so it stayed valid for that file content indefinitely — it is now retired the moment a member acknowledges our rendered config, gated on the acked hash matching so a failed deploy never drops it. No schema change, no agent change.
+- **v1.10.8** (2026-08-13) — **VIP adoption takes the whole VRRP instance**: adoption used to take only the node whose row was clicked, which broke the exact case the feature exists for, a running HA pair. Adopting the **BACKUP** alone produced a VIP that could never be applied (*exactly one member must be MASTER*); adopting the **MASTER** alone left the peer unmanaged, and adopting it afterwards hit the VRID-collision guard with 409, so the pair could not be completed from the panel at all. Most serious, on a **unicast** instance the single-member render dropped the unicast block entirely — the renderer emits it only when it has peer addresses — so keepalived fell back to **multicast** on the adopted node while its peer stayed unicast: they stop seeing each other and **both** claim the VIP. Adoption now resolves the whole instance, keyed on `(virtual_router_id, virtual address)` exactly as keepalived groups nodes, and every participating node becomes a member with the role, priority and interface its own file declares and its own one-shot takeover hash. It refuses, with the reason, when the group does not have exactly one MASTER, when the nodes disagree on `advert_int`, when a declared unicast peer is not among the nodes being adopted, or when a node already belongs to a live VIP — a rule create/edit enforced and adoption did not. The panel now lists one row per **instance** instead of per node. Two further fixes: the Apply Management **View Change** diff did not recognise the `adopt` action, so it fell through to the generic HAProxy diff and rendered the cluster's entire `haproxy.cfg` as removed; and **rejecting** an adoption hid the node from the panel permanently, because `adopted_vip_id` is write-once, a VIP is only ever soft-deleted (so the column's `ON DELETE SET NULL` never fires) and the agent does not re-report an unchanged file — adoptability is now derived from whether the linked VIP is still active, which self-heals reject, undo-reject and approved teardown alike. No schema change, no agent change.
+- **v1.10.7** (2026-08-13) — **HA / VIP follows the selected cluster**: the page ignored the cluster picker in the header. On a multi-cluster install both the VIP table and the new *Unmanaged keepalived detected* panel listed every cluster's nodes at once and did not change when the selection did, so the panel appeared to be stuck on one cluster's keepalived. Both lists now send `cluster_id`, resolved to that cluster's pool exactly as the Apply Management view already did. The API parameter is **optional**: a caller that omits it still receives the whole fleet, so nothing outside the page changes. This is a deliberate behaviour change for the VIP table, which was fleet-wide before. Backend and frontend only: no schema, no agent change.
+- **v1.10.6** (2026-08-13) — **VIP adoption panel was unreachable**: v1.10.4's *Unmanaged keepalived detected* panel never appeared, even on a fleet where the agents had reported their configs correctly. `GET /discoveries` was declared **after** `GET /{vip_id}` in `routers/vip.py`, and FastAPI matches routes in declaration order, so every request for the discovery list was answered by the get-one-VIP handler, which takes `vip_id: int` and rejected `"discoveries"` with **422** before the real handler ran. Nothing surfaced the failure: the agents reported normally, the rows landed in `vip_discoveries`, and the HA/VIP page treats any non-OK response as "nothing to show" — so the whole feature was invisible with no error anywhere. The route is moved above the parameterised ones, and a static source scan now asserts that **no** literal path in **any** router is shadowed by an earlier parameterised one, so the class of bug cannot come back silently. Data reported under v1.10.4 is not lost: existing `vip_discoveries` rows appear as soon as the fixed backend is deployed, with no agent action needed. Backend-only fix. No schema, API-shape or agent change.
+- **v1.10.5** (2026-08-09) — **HTTP-01 challenge backend on split deployments**: on a deployment where the HAProxy nodes and the management stack are on different hosts, HTTP-01 issuance could fail silently for weeks while DNS-01 kept working — the rendered config pointed `server _acme_mgmt` at an address that resolves **on the HAProxy node**, defaulting to loopback, and every diagnostic still reported success. The per-cluster `acme_backend_url` now has a UI field, changing it actually mints a config version, and the value is validated where it is written. Three adjacent bugs are fixed with it: a config-generation failure was returned as `# Error ...` text and then stored as an APPLIED version and pushed to agents as the cluster's whole `haproxy.cfg` (both call sites now refuse with 422); a nullable `frontends.mode` was interpolated raw and emitted `mode None`, which HAProxy rejects and which takes down the entire cluster config; and cluster creation silently dropped the ACME fields. `docker-compose.yml` now interpolates `PUBLIC_URL` / `MANAGEMENT_BASE_URL` instead of hardcoding them, with the old literals as defaults. Diagnostics read the response body so an SPA answering 200 is no longer counted as healthy, and every new condition is a warning rather than a failure so no install is locked on upgrade. No schema, API-shape or agent change.
+- **v1.10.4** (2026-08-08) — **Adopt an existing keepalived VIP** (Issue #27 follow-up): on a fleet that already runs keepalived, the **HA / VIP** page came up empty, because the flow was one-way — VIPs were declared in OpenManager and pushed to the node, and nothing ever read what was already there. Agents now **report the `keepalived.conf` they find and do not own** (strictly read-only; the node is never touched), the page lists those nodes under *Unmanaged keepalived detected*, and **Adopt** turns one `vrrp_instance` into a managed VIP with the values from the file instead of retyping them. The heartbeat could not drive this: it carries the VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs **eleven** fields, and guessing them is not cosmetic — a wrong `virtual_router_id` puts the nodes in separate VRRP domains and a wrong `auth_pass` makes them reject each other, so both would claim the VIP. Because adoption **replaces** the operator's file with OpenManager's render, the parser reports every directive it cannot reproduce — a `notify_master` hook, an LVS `virtual_server` section, a `vrrp_sync_group`, a second address in one instance, a custom `track_script` — and **refuses** while any remain; the operator can waive that class explicitly, but a value that is simply *unknown* (an absent VRID or prefix length) can never be waived, only supplied. keepalived's own documented defaults (`state BACKUP`, `priority 100`, `advert_int 1`) are applied and shown as assumed. The agent's ownership guard is **not** weakened: adoption authorises exactly **one** takeover of exactly the file that was analysed, pinned to its hash, so a config edited between adoption and Apply is still refused. The adopted VIP is created **PENDING** like any other, so nothing reaches the node until it is applied from Apply Management. VRRP passwords are Fernet-encrypted at ingest and masked in the stored copy and the preview. Schema change: one new table `vip_discoveries` plus two additive columns (SCHEMA_VERSION 10 → 11, auto-migrated, no existing table altered) — **see the upgrade notes: this bump re-seeds the four built-in roles, and the Linux agent script must reach the nodes before discovery starts**.
+- **v1.10.3** (2026-08-08) — **Multi-account ACME: the certificate wizard honours the account you pick**: with more than one ACME account registered, picking an **HTTP-01** account in *Request ACME Certificate* still produced a **DNS-01** request. Three faults compounded. (1) `Form.useWatch` reports only fields that are currently **rendered**, and the account `Select` lives on the *Configuration* step — so as soon as the wizard advanced to *Review* the watch read `undefined` and the wizard silently reverted to the default account, even though the value was still in the form store; the watches now pass `preserve: true`. The same fault disabled the **wildcard guard** on *Review*, the one step where Submit lives. (2) The UI and the backend disagreed on which account is the *default*: the backend takes the **newest** valid account (`ORDER BY created_at DESC`), the UI took the **oldest** entry of a list ordered by id — the opposite account whenever the two differ. The wizard now resolves the same one, and sends `account_id` **explicitly** so there is no guess left to disagree about. (3) `account_id` was read from the form store while `challenge_type` came from the reverted account object, so the request asked for DNS-01 validation on an HTTP-01 account and the API answered `The selected ACME account has no DNS provider configured for DNS-01.` — both are now derived from one resolved account. The *Review* step also showed the default account's address instead of the chosen one, and Submit stayed enabled for a deactivated account; both fixed. Frontend only — no schema, API-shape, agent or rendered-config changes, and single-account installations behave exactly as before.
+- **v1.10.2** (2026-08-08) — **Dark mode fixes on Apply Management**: several panels on the Apply Management page were painted with light-mode colour literals, so in dark mode the **Pending Changes** box rendered as a cream panel with light text on it — measured contrast **1.03:1**, effectively unreadable, now **11.50:1**. The same bug affected the added/removed rows in the *View Change* diff (2.21:1 and 2.99:1, now 5.49:1 and 4.01:1), the ACME and pending-version panels, the VIP pending-delete row, and the agent-error recommendation box; all now derive from theme tokens. Separately, **static confirm dialogs came up white in dark mode**: in Ant Design 5 the static `Modal.confirm` / `message` / `notification` APIs render into their own detached root and never see the app's `ConfigProvider`, so they always used the light algorithm. Registering `ConfigProvider.config({ holderRender })` once at the app root fixes **every** static dialog in the application (12 components use them), not only this page. Light mode is byte-identical — each token resolves under the default algorithm to exactly the literal it replaced. Frontend only: no schema, API, environment or agent change.
+- **v1.10.1** (2026-08-08) — **CSR private key encrypted at rest** (Issue #53): the private key of a **pending** CSR is now Fernet-encrypted in the database instead of stored as PEM. It is the one key in the system worth protecting this way — it sits idle for the entire signing window (days to weeks), is never transmitted to an agent, and is destroyed the moment the signed certificate is imported; `ssl_certificates.private_key_content` and the ACME order keys are unchanged, because agents must receive those in plaintext on every poll. The token replaces the PEM in the **same column**, so there is **no schema change and no `SCHEMA_VERSION` bump** (and therefore no re-seed of the built-in roles). CSRs created before this release keep a raw PEM and are still read transparently, so anything already out for signature imports normally with no data migration. The key derives from `SECRET_KEY` via HKDF with its own info string, independent of the VIP/MFA/DNS keys, and an optional `CSR_ENCRYPTION_KEY` enables independent rotation — rotating `SECRET_KEY` without it makes pending CSR keys unrecoverable, which now fails with an explicit "delete and re-create this CSR" error rather than a misleading key-mismatch. `.env.template` now documents all four per-purpose encryption keys. No API, UI or agent change.
+- **v1.10.0** (2026-08-07) — **GoDaddy DNS provider for DNS-01** (Issue #35 follow-up): DNS-01 challenges can now be published and cleaned up automatically through **GoDaddy**, alongside the existing Manual and Cloudflare providers, so wildcard and internal-cluster certificates on GoDaddy-hosted zones **renew unattended**. Credentials are a **Production API Key + Secret** pair from `developer.godaddy.com/keys` (a **Personal Access Token** also works — paste it as the Key and leave the Secret blank, which is the forward path as GoDaddy retires `sso-key`); they are **verified against the GoDaddy API before being saved** and **encrypted at rest** (Fernet, the same path as Cloudflare), and are never returned by the API, logged, or written to an order event. GoDaddy's v1 API has **no per-value TXT write** — `PUT` replaces an entire RRset — so add/remove are read-modify-write with sibling values merged back, empty-`data` tombstones filtered out, and `DELETE` used for the last value (`PUT []` is rejected); this is what keeps the **apex + wildcard** case (two TXT values at one `_acme-challenge` name) working, and the record path is hard-gated so it can never collapse onto the zone-wide endpoint that would wipe SPF/DKIM/DMARC. Zone lookup probes the records API rather than the domain listing, so **delegated sub-zones** resolve and small accounts are not falsely rejected. Registry-only addition: one new provider module plus one registry line — no frontend change (the credential form is schema-driven). No schema, API-shape, agent, or rendered-config changes; Manual, Cloudflare and HTTP-01 are unaffected.
+- **v1.9.0** (2026-08-04) — **CSR creation** (in-app key + CSR generation and signed-certificate import): a new **CSR tab** on the SSL Certificates page generates a private key and Certificate Signing Request server-side (RSA 2048/4096 or ECDSA P-256/P-384; full subject — O/OU/L/ST/C/email — plus DNS SANs with wildcard support), for certificates signed by an **external or corporate CA**. The operator downloads/copies the CSR PEM, has it signed, then imports the signed certificate (+ optional chain): the backend verifies the certificate against the stored key (hard gate), rejects expired certs, warns on SAN drift, and creates a normal SSL certificate entry (source `CSR`) that flows through the standard **PENDING → Apply Management → agent pull** pipeline. The private key **never leaves the server** — no CSR endpoint returns it, and after import the CSR row's key copy is destroyed (the key then lives only on the certificate, like every other key). Additive schema change: one new table `ssl_csrs` (SCHEMA_VERSION 9 → 10, auto-migrated, no existing table altered); key generation runs off the event loop and is rate-limited per user; existing `ssl.*` permissions govern all new endpoints. No agent or rendered-config changes.
+- **v1.8.10** (2026-07-20) — **Security hardening** (GHSA-7rhv-c5pc-69r8, GHSA-3p5c-m5m4-mjpx, GHSA-3vh4): three advisory classes remediated, backend-only, no agent changes. (1) **RCE**: the agent script-template read/write endpoints now require the `agents.version` permission on top of authentication — a poisoned template is executed as root on every HAProxy node, so authentication alone was insufficient. (2) **Missing authentication**: operator/UI endpoints that were served without a JWT (dashboard stats, pool/cluster listings, agent inventory, WAF rules, config validate/optimize, SSL config-versions, health deep/agents/clusters) are now gated by a `require_authenticated_user` dependency, and agent data-plane endpoints that treated the `X-API-Key` header as *optional* (heartbeat, config, ssl-certificates, upgrade-status, pending-requests) now hard-reject a missing key. In every case the auth check was moved **ahead of** the handler's `try:` block so a 401 can no longer be rewritten into a 500 by the generic exception handler. (3) **SSRF**: a new `utils/ssrf_guard.py` (https-only, IPv4-pinned connector, all resolved addresses must be public, no redirects) protects the ACME directory fetch, the signed-request target and the ACME connection test, which accept operator- or DB-supplied URLs; the connection test also stopped reflecting arbitrary upstream JSON. Frontend dependency advisories patched in the same release. No schema, API-shape or rendered-config changes.
+- **v1.8.9** (2026-07-13) — **ACL `-f` pattern-file support** (Issue #38 follow-up): ACL definitions that reference a host-side pattern file (`acl … -f /etc/haproxy/lists/blocked.lst`) are accepted on import and edit instead of being rejected. The referenced file lives on the HAProxy node and cannot be validated from the manager, so the manager emits an **advisory warning** rather than a hard rejection and lets the agent's `haproxy -c` check be the fail-safe gate (a broken reference fails validation on the node and the previous config is restored). Consistent with the SPOE handling introduced in v1.8.8.
+- **v1.8.8** (2026-07-10) — **SPOE filter and frontend `log-format` preserved on import/edit** (Issue #38): importing an existing `haproxy.cfg` or editing a frontend silently dropped `filter spoe …` directives and custom `log-format` lines, so the next Apply pushed a config that had lost them. Both are now round-tripped through import and edit. As with `-f` pattern files, the SPOE engine config is a host-side file the manager cannot read, so it is preserved verbatim and reported as an advisory rather than validated centrally.
+- **v1.8.7** (2026-07-09) — **Version reporting single-source fix**: the version shown in the UI (backend-sourced via `/api/version`) could lag behind the real release. The canonical version lived in the repo-root `version.json`, but the backend image is built from the `./backend` context, so that file did not reach the container in every pipeline; the backend then fell back to a hardcoded constant in `main.py` that had to be bumped by hand and had drifted (it reported 1.8.4 after 1.8.5/1.8.6 shipped). The version now lives in a single file, `backend/version.json`, baked into every image automatically, and `main.py` no longer carries a real version literal (its fallback is a neutral "unknown"). A new test enforces that the version stays single-source and cannot drift. No functional or API change.
+- **v1.8.6** (2026-07-06) — **Performance: opt-in API workers + heartbeat micro-optimization** (Issue #35 follow-up): the backend container can now run multiple uvicorn worker processes via the new `UVICORN_WORKERS` environment variable (default **1** — behavior unchanged unless you opt in), letting the API use all cores on multi-core hosts; background tasks were already multi-replica safe, as exercised by the Kubernetes HPA deployment. The agent heartbeat handler now reads the agent's `status`/`version`/`upgrade_status` in one query instead of three (one round-trip per heartbeat, per agent, every 30s). Added a *Performance Tuning* section to the README (worker/replica scaling and how to use the `X-Response-Time` header and `Slow request detected` logs to pinpoint slow endpoints). Zero-risk release: no schema, API, or agent changes; defaults preserve existing behavior exactly.
+- **v1.8.5** (2026-07-03) — **ACME completion-task SQL fix** (Issue #35 follow-up): the background order-completion task (`complete_pending_acme_orders`, runs every 60s) died on **every cycle** with `syntax error at or near ")"` — an extra closing parenthesis introduced in v1.8.0's bounded DNS-01 retry claim query. Because that query is the task's first database call, **no background ACME work ran at all from v1.8.0 through v1.8.4**: orders were never claimed for finalize/download, the DNS-01 TXT record was never published (so DNS-01 with an automated provider such as Cloudflare could never validate), Site Wizard staged orders never left `wizard_staged`, and DNS-01 retry/TXT-cleanup never executed. The stray parenthesis is removed and a regression test now scans all ACME modules' SQL for unbalanced parentheses (the unit suite mocks the database, which is why a raw-SQL syntax error could slip through). One-line backend query fix; no schema, API, or agent changes — fully backward compatible.
+- **v1.8.4** (2026-06-27) — **Agent installer self-kill fix** (Issue #31): the Linux/macOS agent installer could abort during "pre-installation cleanup" (terminal showed `Killing processes matching: haproxy-agent` then `Killed`) when the install script's own filename contained "haproxy-agent". The cleanup killed processes by matching the bare string "haproxy-agent" against full command lines, which also matched the running installer (and a `sudo`/PAM ancestor the self-exclusion did not cover), so the installer terminated itself. Cleanup now targets only the installed agent (the `$INSTALL_DIR/haproxy-agent` binary and the agent service), never the bare string, and the UI now names the downloaded scripts `install-agent-<platform>.sh` / `uninstall-agent-<platform>.sh`. Installer-only change; the running agent and its privilege model (it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade) are unchanged.
+- **v1.8.3** (2026-06-25) — **Agent heartbeat JSON fix** (Issue #31): a self-hosted agent could fail every heartbeat with `HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host, leaving a stray comma in the hand-built heartbeat JSON. The agent script now substitutes a valid placeholder when that block is empty so it can no longer emit a stray comma, and the backend heartbeat endpoint now parses valid payloads as-is and, only when a body fails to parse, tolerates that specific malformed pattern (a leading or doubled comma) so an already-deployed agent recovers on its next heartbeat after this build is deployed. Backend + agent-script only; healthy agents of every version are byte-for-byte unaffected.
+- **v1.8.2** (2026-06-25) — **ACME nonce fix** (Issue #35 follow-up): the ACME client now scopes the anti-replay nonce **per certificate authority** so a nonce issued by one CA is never sent to another. This fixes ZeroSSL/Google account registration failing with `malformed: The Replay Nonce could not be base64url-decoded` (the client previously shared one nonce across CAs and only auto-retried on `badNonce`). Account registration now always uses a fresh nonce from the target CA, and the retry covers this case too. Backend-only; HTTP-01 and Let's Encrypt are unaffected.
 - **v1.8.1** (2026-06-24) — **ACME DNS-01 fixes** (Issue #35 follow-up): Cloudflare API tokens are now sanitized so a pasted token with quotes/spaces no longer fails with "Invalid request headers"; ZeroSSL/Google **External Account Binding (EAB)** can be entered per-account in the register dialog and EAB-required failures show a clear message; and **Apply Management** now categorizes cluster ACME enable/disable changes under their own "ACME Challenge Routing" section and **Apply/Reject All** correctly process them (previously "Rejected 0 HA/VIP change(s)"), consistent with every other entity. Fully backward compatible.
 - **v1.8.0** (2026-06-23) — **ACME DNS-01 challenge support** (Issue #35): Auto SSL can now validate via a **DNS TXT record** (`_acme-challenge.<domain>`) instead of HTTP-01 on port 80, enabling certificates for **internal/isolated clusters with no public ingress** and **wildcard** certificates (`*.example.com`). Pluggable **per-account DNS provider** (Manual + Cloudflare to start; credentials verified on save and **encrypted at rest**, never returned by the API or logged), the same **PENDING → APPLIED** pipeline, a **bounded automatic retry** on propagation lag, and a **DNS-01 event timeline** in the order detail. **Opt-in** via Settings → ACME (global switch, default off); **HTTP-01 is byte-for-byte unchanged**, with **zero agent or rendered-config changes**. Manual DNS-01 certificates cannot auto-renew unattended; the UI states this and disables auto-renew for them.
 - **v1.7.8** (2026-06-07) — HA / VIP apply progress now shows **per-node** convergence: a multi-node VIP's apply popup reads "Syncing HA/VIP… 1/2 node(s) converged" (matching the HA/VIP table) instead of a coarse per-change count. Frontend-only.

UPGRADE_GUIDE.md

@@ -1,3 +1,604 @@
+# Upgrade Notes — v1.11.1 (adoption cannot hide a node; Config Import on fresh installs)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. After deploying, sync the Linux agent script from **Agent Management** and let
+the agents upgrade, or none of this reaches the nodes.
+
+- **A node that never appeared under *Unmanaged keepalived detected* now recovers by itself.**
+  The agent caches the hash of its last discovery report and skips re-posting while it matches.
+  Delivery was judged by `curl`'s exit code, which is 0 for 5xx too, so a rejected report was
+  cached as delivered and a hand-maintained config — which never changes on its own — kept that
+  node hidden. The report is now cached only on a 2xx, and the config endpoint reports whether
+  the server actually holds a discovery for that agent, so the cache can only suppress while the
+  server agrees. **No access to the nodes is needed**; affected nodes reappear within one poll
+  cycle (~2.5 min) after the agents pick up the new script.
+- **Permanent refusals do not loop.** A 400, 413 or 422 means the payload itself is unacceptable,
+  so the refusal is recorded and the same bytes are not re-posted; fixing the file releases the
+  brake, because it is keyed to the content hash. **401 and 404 keep retrying** — in this system
+  they mean a token rotation or an agent row briefly absent while it re-registers, and braking on
+  them would have re-created the very failure above. This matters beyond noise: 4xx and 5xx agent
+  calls are never sampled out of the request log, so a loop would write a row carrying the whole
+  `keepalived.conf` every cycle on every affected node.
+- **The clear path had the same defect.** When a node becomes managed the agent tells the server
+  to drop the discovery; that too was judged by the exit code, so a rejected clear left a stale
+  row offering a **managed** node for adoption, with nothing to ever retry it.
+- **Config Import now works on a freshly installed agent.** `check_config_requests` uploads a
+  node's live `haproxy.cfg` when you ask for it. It was defined in the installer and in the
+  self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by
+  `type`, so on such a node the feature was a silent no-op: the request was made and nothing ever
+  arrived. Any agent that had self-upgraded at least once already had it, which is why it went
+  unnoticed. A freshly installed agent now polls the pending-requests endpoint once per cycle,
+  exactly as every upgraded agent already does — **no node running today changes behaviour**.
+- **The keepalived.conf path is resolved deterministically.** A pool may hold more than one
+  cluster and the join was unordered, so the path handed to an agent could differ between polls
+  whenever two clusters disagreed on it — the agent would inspect a file that is not there and the
+  node would never appear, intermittently. A customised path now wins over the shipped default,
+  then the lowest cluster id. With one cluster per pool, or when every cluster carries the
+  default, the value is byte-identical to before.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous behaviour, in which
+a rejected discovery report is never retried and Config Import is absent on fresh installs.
+
+---
+
+# Upgrade Notes — v1.11.0 (Unified request/response log)
+
+**Adds one new table and bumps `SCHEMA_VERSION` 11 → 12. The migration runs automatically on the
+first backend start.** No agent impact, no rendered-config change, no change to any existing API
+shape or response body.
+
+> **12, not 11.** The feature was developed against v1.10.3, where `SCHEMA_VERSION` was still 10, and
+> proposed 11. v1.10.4 took 11 in the meantime. Shipping it as 11 would have been silently inert:
+> `run_all_migrations()` returns early on `applied_version >= SCHEMA_VERSION`, so every database
+> already at 11 would have skipped the entire sequence and received neither `request_logs` nor the
+> `requestlog.*` permissions, while a fresh install would have received both. If you are coming from
+> a pre-release build that recorded 11 for this feature, no action is needed: the bump to 12 re-runs
+> the (idempotent) sequence and creates whatever is missing.
+
+- **New table `request_logs`** (BIGSERIAL primary key, 9 indexes). Created empty and starts filling
+  immediately. The shipped defaults keep 7 days of successful requests, 30 days of failed ones, and
+  at most 500 000 rows — whichever limit is reached first. Change any of it in
+  *Settings → Request Log*.
+- **Successful agent polls are NOT logged** (`requestlog.capture_agent_success`, default off).
+  Failed agent calls always are. This is what keeps the table's size a function of operator
+  activity rather than of how many nodes you run. Measured at 2 424 bytes/row on PostgreSQL 15
+  against the real schema and all nine indexes, with each agent issuing ~9 792 logged calls a day:
+
+  | fleet | if successful polls were logged | with the shipped default |
+  |---|---|---|
+  | 20 nodes  | 196k rows/day, 453 MB/day, row cap in 61 h | ~21k rows/day, 48 MB/day, cap in 24 d |
+  | 200 nodes | 2.0M rows/day, 4.5 GB/day, row cap in 6 h  | ~30k rows/day, 69 MB/day, cap in 17 d |
+  | 500 nodes | 4.9M rows/day, 11 GB/day, row cap in 2.5 h | ~44k rows/day, 103 MB/day, cap in 11 d |
+
+  The row cap always holds, but it holds by deleting — so without this default the configured
+  "7 days of successes, 30 days of failures" silently becomes a few hours of both, and the forensic
+  record the feature exists for is evicted by polling noise. Turn it on temporarily when debugging
+  a specific node, then turn it back off.
+- **Runtime cost is measured, not estimated.** The middleware adds 27.7 µs (p50) per request and
+  1.4 µs on an excluded path; redaction runs on the writer task, off the request path, at 18.8 µs
+  per row. At 500 nodes that is **0.096 % of one core** in total.
+- **New permissions `requestlog.read` and `requestlog.manage`.** Both are granted to `super_admin`
+  and `security_admin`; `operator` gets `requestlog.read` only; `viewer` gets neither, because
+  captured request/response bodies are a broader disclosure surface than the read-only configuration
+  views a viewer is meant to have. Custom roles can be granted either from **Users → Roles**.
+- **⚠️ Built-in roles are re-seeded to their defaults.** This is the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump, not something new in this release, but it bites here because this release
+  bumps: the version gate re-runs the whole sequence and
+  `update_system_roles_to_enterprise_rbac()` issues an unconditional
+  `UPDATE roles SET … permissions = <defaults> WHERE name = …` for the four **built-in** roles
+  (`super_admin`, `operator`, `security_admin`, `viewer`). **Any customisation you made to a
+  built-in role is reverted.** Roles you created yourself are untouched (the update matches on
+  name). To preserve customisation, export with `GET /api/roles` before upgrading and re-apply with
+  `PUT /api/roles/{id}`, or move the customisation into a custom role.
+- **Bodies are captured, redacted and capped at 8 KB.** Passwords, tokens, API keys, private-key
+  PEMs, JWT-shaped values, `Authorization` / `Cookie` headers, ACME JWS payloads and DNS-provider
+  credentials are never stored. Headers use an allowlist — anything not on it is dropped rather than
+  saved. Review *Settings → Request Log* before enabling body capture in a regulated environment;
+  `capture_bodies` can be turned off while still recording who called what, with what result.
+- **Excluded by default:** health checks, the API docs, the ACME HTTP-01 challenge endpoint (it
+  returns `key_authorization`), the agent heartbeat (the highest-volume POST in the system), static
+  assets, and the log viewer's own endpoints. The list is editable, except the log viewer itself,
+  which is a hard floor so the page cannot end up logging you reading it.
+- **Disk growth is the main operational consideration.** On a busy install, in order of bluntness:
+  leave `capture_agent_success` off (the single biggest lever on a large fleet), lower `sample_rate`
+  (errors are always kept at 100 %), turn off `capture_get`, turn off `capture_bodies`, or shorten
+  `success_retention_days`.
+- **`operator` can see agent rows.** A caller holding only `requestlog.read` sees their own inbound
+  requests plus the fleet's, and nothing belonging to another user. Agent rows are included because
+  an apply fails on the *node* and the node reports it over its own API key — scoping to own-rows
+  only would have hidden the diagnosis from exactly the role the grant exists for. Anonymous traffic
+  (failed logins and the usernames they carry, unauthenticated probes) is **not** agent traffic and
+  remains visible only to `requestlog.manage`.
+- **New environment variables**, all optional: `REQUEST_LOG_ENABLED` (default `true`),
+  `REQUEST_LOG_QUEUE_MAX` (2000 rows), `REQUEST_LOG_QUEUE_MAX_BYTES` (64 MiB),
+  `REQUEST_LOG_BATCH_SIZE` (100), `REQUEST_LOG_FLUSH_MS` (500). See `.env.template`.
+  `REQUEST_LOG_QUEUE_MAX_BYTES` is a hard memory ceiling per worker: the row count alone does not
+  bound memory, because `max_body_bytes` is operator-editable up to 256 KB and a row can carry it
+  twice — at the ceiling the default 2 000-row queue would hold ~1 GiB, which is the whole pod
+  limit. Whichever limit binds first stops the queue. If you raise `REQUEST_LOG_QUEUE_MAX`, raise
+  this with it.
+- **`GET /api/request-logs/stats` sink counters are per worker**, labelled as such in the response.
+  With `UVICORN_WORKERS > 1` each process keeps its own queue and its own drop counter.
+- **Clearing the exclude-path list does not mean "log everything."** An empty list falls back to the
+  shipped defaults, so the log viewer and the raw-body agent heartbeat stay excluded; the UI now
+  says so and shows what was actually applied.
+- **To disable entirely:** set `REQUEST_LOG_ENABLED=false` in the backend environment and restart —
+  the middleware is then not registered at all and costs nothing, not even a settings lookup. The
+  `enabled` toggle in Settings is the no-restart equivalent (it takes effect immediately).
+- **Default admin password is not reset** by this bump; user seeding is guarded by an existence
+  check, not an upsert.
+- **Rollback:** downgrade the backend image freely. `request_logs` is purely additive and is simply
+  ignored by v1.10.x. Drop the table manually if you want the space back:
+  `DROP TABLE IF EXISTS request_logs;`
+
+---
+
+# Upgrade Notes — v1.10.14 (a converged node keeps acknowledging)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** a VIP shows `SYNCING (0/n)` with an empty *Last ack* even though every member node
+  has the rendered `keepalived.conf` on disk, keepalived is running and the VIP is held.
+- **Cause:** the deploy report was sent only when the agent actually wrote the config. Once the
+  node matched, it took the idempotency early return every cycle and never reported again, so any
+  report lost in transit was never retried and the server's view stayed stale permanently.
+- **Fix:** the agent re-asserts its state on the idempotent path as well. One request per node
+  per poll cycle (~2.5 min); nothing is written and keepalived is not reloaded.
+- **Recovery is automatic.** A VIP stuck at SYNCING converges on the first poll after the agents
+  pick up the new script. No action on the nodes, no re-apply, no edit to force a rewrite.
+- **This is not new in 1.10.12.** The gap dates from the original HA/VIP work; it only became
+  visible when acknowledgements were dropped for an unrelated reason.
+
+**Rollback:** safe. Reverting restores the previous behaviour, in which a lost acknowledgement is
+never recovered.
+
+---
+
+# Upgrade Notes — v1.10.13 (deploy acknowledgements were dropped)
+
+**Backend only, no schema change.** No `SCHEMA_VERSION` bump, no agent change. If you deployed
+v1.10.12, deploy this one too.
+
+- **Regression in v1.10.12.** The takeover-retirement clause added to the keepalived status
+  endpoint reused a query placeholder for both an assignment and a comparison. PostgreSQL types a
+  placeholder per use, so it was deduced as `text` in one place and `character varying` in the
+  other, and asyncpg refused the statement outright.
+- **Symptom:** a VIP stayed at `SYNCING (0/n)` with an empty *Last ack* even though the agent log
+  showed `applied config for VIP <id>` on every member. Teardown acknowledgements were lost the
+  same way, so a deletion never showed as complete.
+- **Nothing was damaged.** The failure was on the write of the acknowledgement, not on the node.
+  Configs were deployed correctly throughout; only the reporting was lost. Existing VIPs converge
+  on the next poll once this is deployed, with no action on the nodes.
+- **Verified against a real PostgreSQL**, not by inspection: both statements execute, a matching
+  hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash`
+  leave it in place, and the acknowledgement is recorded in every case.
+
+**Rollback:** do not roll back to v1.10.12; roll back to v1.10.11 instead, which predates the
+clause entirely.
+
+---
+
+# Upgrade Notes — v1.10.12 (valid config rejected by its own warning)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** applying a VIP left it stuck at `SYNCING`, the node kept its previous config and
+  the agent logged only `config validation failed (keepalived -t)`.
+- **Cause:** the agent treated any non-zero exit from `keepalived -t` as invalid. keepalived's
+  config-test exit code does not separate fatal from benign: on 2.2.8 a clean config exits 0,
+  while `Truncating auth_pass to 8 characters` exits 5 and so do a missing `}` and an unknown
+  keyword. A VRRP password longer than eight characters was enough to block every apply, even on
+  a node whose own running config emits the same warning.
+- **Fix:** the gate judges the output instead. Known-benign messages are dropped and anything
+  left still fails, so it fails closed. Verified against real keepalived: the truncation warning
+  passes; a missing brace, an unknown keyword and a `SECURITY VIOLATION` are refused.
+- **Also:** the agent now reports what keepalived actually said, in its log and in the status the
+  HA/VIP page shows. The refusal was correct but unactionable without reproducing it by hand.
+- **The fail-safe itself is unchanged:** a config that genuinely fails validation is never
+  written and keepalived is never restarted.
+
+**Rollback:** safe. Reverting restores the stricter gate, which rejects valid configs whose
+password exceeds eight characters.
+
+---
+
+# Upgrade Notes — v1.10.11 (Adoptable tag names the real blocker)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The *Adoptable* tag and the disabled *Adopt* button were derived separately, so they could
+  name different problems. A pair blocked by a peer whose config could not be parsed showed
+  **MASTER missing**, because the unreadable node's `state MASTER` had not been counted — true,
+  but it sent the operator to the wrong node. Both now come from one ordered decision.
+- New label **blocked by peer** for a group held up by a node that references the same address
+  but cannot be taken over with it. Two MASTERs is now distinct from none.
+- Display only. The endpoint's checks and refusals are unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.10 (Adoption blockers listed once per instance)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The adoption panel merged every member's blocker list, so a two-node pair showed each shared
+  problem twice. The two files report different line numbers for the same directive, so exact
+  de-duplication did not collapse them. Blockers are now merged on the message with the leading
+  `line N:` ignored.
+- Display only. The endpoint already evaluated the combined set across all nodes, and what it
+  accepts or refuses is unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.9 (Adoption refuses to strand a node)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent change.
+
+- **Adoption will not leave a node behind.** v1.10.8 resolved the whole VRRP instance, but only
+  from nodes it could parse, that were enabled and that were in the same pool. Anything else fell
+  out of the set silently while its peers were rewritten. Adoption now refuses if any reported
+  `keepalived.conf` mentions the virtual address and is not among the nodes being taken over, and
+  says which node and why.
+- **The nodes must agree on the shared fields.** `prefix_length`, unicast/multicast mode, HAProxy
+  tracking and the VRRP password live on the VIP and are re-rendered onto every member, so one
+  node's value used to be imposed on the rest. A disagreement is now refused with both values
+  shown.
+- **The takeover authorisation is now retired on acknowledgement.** It is the permission to
+  overwrite a `keepalived.conf` that does not carry our marker. It was never cleared, so it stayed
+  valid for that exact file content indefinitely; restoring the pre-adoption file would have been
+  overwritten again without fresh approval. It is now dropped once the member acks our rendered
+  config, gated on the acked hash matching `applied_config_hash` so a failed deploy cannot strand
+  the VIP.
+- **Nothing to do on upgrade.** Existing adopted VIPs keep working; their authorisation is retired
+  on the next successful acknowledgement.
+
+**Rollback:** safe. No schema or data migration; reverting restores the previous (more permissive)
+adoption checks.
+
+---
+
+# Upgrade Notes — v1.10.8 (VIP adoption takes the whole VRRP instance)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact: nothing about what the agent reports or how it takes a
+config over changes.
+
+- **Adoption is now per VRRP instance, not per node.** Every node in the pool reporting the same
+  `virtual_router_id` and virtual address becomes a member of one VIP, each with the role,
+  priority and interface its own `keepalived.conf` declares, and each with its own one-shot
+  takeover hash. The panel lists one row per instance.
+- **Why this mattered:** single-node adoption could not produce a working pair. The BACKUP alone
+  failed apply, the MASTER alone left the peer unmanaged and the peer could not then be adopted
+  (VRID collision). On a **unicast** instance it was worse than inconvenient: the render drops
+  the unicast block when there are no peers, so the adopted node fell back to multicast while its
+  peer stayed unicast and both could hold the address.
+- **New refusals, each with the reason in the message:** the group does not have exactly one
+  MASTER; the nodes disagree on `advert_int`; a declared unicast peer is not among the nodes being
+  adopted; a node is already a member of a live VIP.
+- **Apply Management "View Change" now renders the adopt diff correctly.** It did not recognise
+  the `adopt` action and fell through to the generic HAProxy diff, which compared the staged
+  `keepalived.conf` against the cluster's previous `haproxy.cfg` and showed the whole HAProxy
+  config as removed. Alarming, but display-only — nothing was ever applied from that view.
+- **Rejecting an adoption is recoverable again.** It used to hide the node from the panel
+  permanently. Nothing clears `vip_discoveries.adopted_vip_id`, a VIP is only soft-deleted so the
+  column's `ON DELETE SET NULL` never fires, and the agent does not re-report a file whose hash
+  has not changed. Adoptability is now derived from whether the linked VIP is still active.
+- **If you adopted a VIP on 1.10.4-1.10.7**, check it before applying: it may have only one
+  member. Add the peer from the VIP's edit form, or reject the pending adoption and adopt again —
+  the node reappears in the panel under this release.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous single-node
+adoption behaviour.
+
+---
+
+# Upgrade Notes — v1.10.7 (HA / VIP follows the selected cluster)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact.
+
+- **The HA / VIP page ignored the cluster picker.** Both the VIP table and the *Unmanaged
+  keepalived detected* panel queried the whole fleet, so on an install with more than one
+  cluster the lists never changed when the selection did. Both now pass `cluster_id`, mapped to
+  the cluster's pool the same way `GET /api/vip?cluster_id=` already worked for Apply
+  Management.
+- **Behaviour change worth knowing:** the VIP table is now scoped to the selected cluster. It
+  used to show every VIP in the fleet. If you relied on the fleet-wide view, the API still
+  supports it — `GET /api/vip` and `GET /api/vip/discoveries` without `cluster_id` return
+  everything, unchanged.
+- **API compatibility:** `cluster_id` is optional on both endpoints. Existing integrations that
+  do not send it behave exactly as before.
+
+**Rollback:** safe. The change is a query parameter plus the page that sends it; reverting
+restores the fleet-wide lists and touches no data.
+
+---
+
+# Upgrade Notes — v1.10.6 (VIP adoption panel was unreachable)
+
+**One backend fix, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No API-shape change, no frontend change and zero agent impact.
+
+- **v1.10.4's adoption panel never appeared.** `GET /discoveries` was declared after
+  `GET /{vip_id}` in `routers/vip.py`. FastAPI matches routes in declaration order, so the
+  discovery list was routed into the get-one-VIP handler, which declares `vip_id: int` and
+  answered **422** before the real handler ran. The HA/VIP page treats any non-OK response as
+  "nothing to show", so the feature was invisible with no error in any log.
+- **Nothing was lost.** The agent side always worked: discoveries were reported and stored in
+  `vip_discoveries`. Deploy this backend and the rows appear immediately — no agent upgrade, no
+  re-sync of the agent script, no re-report needed.
+- **If you are upgrading straight from 1.10.3 or earlier**, follow the v1.10.4 notes below as
+  well: that release does bump `SCHEMA_VERSION` (10 → 11), which re-seeds the four built-in
+  roles, and its agent script has to reach the nodes before discovery starts.
+- **Regression guard.** A static source scan now fails the build if any literal API path in any
+  router is declared after a parameterised route that would swallow it. The whole router tree is
+  clean as of this release.
+
+**Rollback:** safe and immediate. The change is a route declaration order plus a test; reverting
+to 1.10.5 restores the previous (broken-panel) behaviour and touches no data.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.5 (HTTP-01 challenge backend on split deployments)
+
+**Bug fixes, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are **not**
+re-seeded. No API-shape change and zero agent impact.
+
+- **HTTP-01 could fail silently when HAProxy runs on different hosts than the management stack.**
+  The rendered config wrote `server _acme_mgmt <mgmt>:8080` from a value that defaults to
+  loopback — and HAProxy resolves that address **on the HAProxy node**, so it pointed at the wrong
+  box. Every check still reported success. The per-cluster `acme_backend_url` now has a UI field
+  (Cluster Management), changing it actually mints a config version, and the value is validated at
+  the write boundary.
+- **A config-generation failure could be pushed to agents as the cluster's whole `haproxy.cfg`.**
+  The generator reported failure by *returning* `# Error ...` instead of raising, and the apply
+  path hashed that comment and stored it as an APPLIED version. Both persisting call sites now
+  refuse with 422 and leave the running config in force. **This is worth knowing even if you never
+  touch ACME**, since any exception in the generator could trigger it.
+- **`frontends.mode` is nullable and was interpolated raw**, emitting a literal `mode None` that
+  HAProxy rejects — which fails the whole cluster config, not just that frontend. Normalised now.
+- **Cluster creation ignored the ACME fields**: a cluster created with ACME switched on came back
+  switched off, with no error.
+- **`docker-compose.yml` hardcoded `PUBLIC_URL` / `MANAGEMENT_BASE_URL`**, so a value in your
+  `.env` or host environment was silently ignored. They are interpolated now, with the previous
+  literals as defaults, so behaviour is unchanged unless you actually set them.
+- **Diagnostics stop over-reporting health.** The port-80 check now reads the body, so a reverse
+  proxy answering 200 with a web page is no longer counted as a working challenge endpoint. Every
+  new condition is a **warning, never a failure** — the Site Wizard blocks submit on a failing
+  check, so a new failing condition would have locked installs on upgrade day.
+- **Rollback:** downgrade freely. No schema or data change.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.4 (Adopt an existing keepalived VIP)
+
+**Additive, but this release DOES bump the schema — read the role warning below.** Nothing on any
+node changes until you adopt a VIP and apply it.
+
+- **Schema:** `SCHEMA_VERSION` bumps to `11`, so on first start the (idempotent) migration
+  sequence re-runs once and adds **one new table** (`vip_discoveries`) plus two additive columns
+  (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`). **No existing table is
+  altered**, no existing row changes, and the admin password is not reset.
+- **⚠️ Built-in roles are re-seeded to their defaults** — the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump. If you customised `super_admin` / `operator` / `security_admin` /
+  `viewer`, **re-apply those changes after upgrading**. (The three previous releases did not bump
+  the version, so this is the first re-seed since v1.9.0.) No new permission strings are
+  introduced: discovery and adoption are governed by the existing `vip.read` / `vip.create`.
+- **⚠️ The Linux agent script changed, and discovery does not start until nodes run it.** The
+  fallback latest Linux agent version moves `2.0.0` → `2.1.0`, so nodes will pull the new script
+  through the normal agent-upgrade path. The addition is **read-only**: the agent reads the
+  `keepalived.conf` it does not own and reports it, rate-limited to once per content change. It
+  writes nothing new to the node. Until a node has upgraded, it simply never appears under
+  *Unmanaged keepalived detected*.
+- **Nothing is taken over implicitly.** The agent still refuses to overwrite a `keepalived.conf`
+  that lacks OpenManager's ownership marker. Adoption authorises exactly **one** takeover of
+  exactly the file that was analysed, pinned to its md5: if the file changes between adoption and
+  Apply, the agent refuses again and reports `externally_managed` rather than clobbering your
+  edit. Re-adopt to pick up the current file.
+- **Adoption can refuse, on purpose.** It replaces the file with OpenManager's render, so anything
+  the renderer cannot reproduce would be destroyed. Those directives are listed as blockers —
+  `notify_*` failover hooks, `vrrp_sync_group`, LVS `virtual_server` sections, a second address in
+  one instance, a custom `track_script`, extra `global_defs`. You can accept that loss explicitly
+  with a tick, but a value that is *unknown* rather than lost (an absent `virtual_router_id`, or
+  an address with no prefix length) cannot be waived — the VRID is fatal to guess and the prefix
+  has to be supplied, because picking a netmask for a live VIP would change its routing.
+- **Multi-node VIPs need every node.** Adoption covers the node that reported. Its unicast peers
+  hold their own `keepalived.conf`, so adopt or add them as members before applying — otherwise
+  the render has no peers. The UI says so after a successful adopt.
+- **Secrets:** the reported config may contain the VRRP `auth_pass`. It is split at ingest — the
+  password is Fernet-encrypted into its own column (same key path as `vip_instances`,
+  `VIP_ENCRYPTION_KEY` falling back to a key derived from `SECRET_KEY`) and the stored copy of the
+  file has it masked, so nothing readable through the API, the UI preview or a DB dump carries it
+  in cleartext.
+- **Rollback:** downgrading to 1.10.3 leaves `vip_discoveries` as an unused table and the two new
+  columns unread; managed VIPs keep working. One caveat: a VIP adopted on 1.10.4 but **not yet
+  applied** loses its takeover authorisation on downgrade, so the node's original config stays in
+  place and the VIP sits PENDING — harmless, but re-adopt after upgrading again. Agents already on
+  script 2.1.0 keep reporting discoveries to an endpoint that no longer exists; the report fails
+  quietly and nothing on the node is affected.
+
+---
+
+# Upgrade Notes — v1.10.3 (Multi-account ACME wizard fix)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change, no
+environment variable, zero agent impact. Installations with a single ACME account behave exactly as
+before.
+
+- **What was broken:** with **more than one** ACME account registered, the *Request ACME
+  Certificate* wizard did not honour the account you selected. Choosing an HTTP-01 account still
+  submitted a DNS-01 request, which the API rejected with
+  `The selected ACME account has no DNS provider configured for DNS-01.` The *Review* step also
+  named the default account rather than the chosen one, so the mismatch was invisible before
+  submitting.
+- **Default account:** the wizard previously previewed the **oldest** valid account while the
+  backend uses the **newest** (`ORDER BY created_at DESC`). If you never picked an account
+  explicitly and have several, requests were already going to the newest one — only the preview was
+  wrong. The wizard now previews that same account, marks it `(default)`, and sends `account_id`
+  explicitly so the two can no longer diverge.
+- **Wildcard guard:** the client-side "wildcard requires a DNS-01 account" block silently stopped
+  applying on the *Review* step. Requests were still rejected by the backend, so nothing incorrect
+  was ever issued — you now get the warning before submitting instead of an error after.
+- **No action needed on existing certificates or orders.** Nothing about issuance, renewal or the
+  stored accounts changes; only how the wizard resolves which account a new request uses.
+- **Rollback:** downgrade freely. This release changes frontend behaviour only.
+
+---
+
+# Upgrade Notes — v1.10.2 (Dark mode fixes on Apply Management)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change,
+no environment variable, zero agent impact. Light mode is byte-identical: every colour swapped in
+this release resolves, under the default algorithm, to exactly the literal it replaced
+(`colorWarningBg` → `#fffbe6`, `colorSuccessBg` → `#f6ffed`, `colorErrorBg` → `#fff2f0`, …), so
+only dark mode changes.
+
+- **Apply Management panels** were painted with light-mode colour literals, so in dark mode the
+  "Pending Changes" box rendered as a cream panel with light text on it. Measured contrast was
+  **1.03:1** — effectively invisible. It is now **11.50:1**. The same class of bug affected the
+  diff rows in *View Change* (2.21:1 and 2.99:1, now 5.49:1 and 4.01:1), the ACME/pending version
+  panels, the VIP pending-delete row and the agent-error recommendation box.
+- **Static confirm dialogs came up white in dark mode.** In Ant Design 5 the static
+  `Modal.confirm` / `message` / `notification` APIs render into their own detached root and never
+  see the app's `ConfigProvider`, so they always used the light algorithm. This release registers
+  `ConfigProvider.config({ holderRender })` once at the app root, which fixes **every** static
+  dialog in the app (12 components use them), not just Apply Management.
+- **Rollback:** downgrade freely. This release changes rendering only.
+
+---
+
+# Upgrade Notes — v1.10.1 (CSR private key encrypted at rest)
+
+**Backward compatible.** Nothing to do on upgrade, and nothing changes for existing clusters,
+agents or certificates:
+
+- **Schema:** **no `SCHEMA_VERSION` bump and no migration.** The Fernet token replaces the PEM
+  inside the *existing* `ssl_csrs.private_key_pem` TEXT column. As in v1.10.0, this means the
+  four built-in roles are **not** re-seeded, so any customization of `super_admin` / `operator` /
+  `security_admin` / `viewer` survives.
+- **Existing pending CSRs keep working.** Rows written before this release hold a raw PEM and are
+  still read transparently, so a CSR that is already out for signature can be imported normally
+  after the upgrade. There is no data migration and no downtime step. Those rows stay plaintext
+  until they are imported (which NULLs the key) — if you want everything encrypted immediately,
+  delete and re-create any long-pending CSRs.
+- **Scope:** this covers the PENDING CSR key only. It is the one key in the system that sits idle
+  for the whole signing window and is never transmitted. `ssl_certificates.private_key_content`
+  and the ACME order keys are unchanged, because agents must receive those in plaintext on every
+  poll.
+- **Optional env:** `CSR_ENCRYPTION_KEY` (see `.env.template`). If unset, the key is derived from
+  `SECRET_KEY` via HKDF with its own info string, so it is independent of the VIP, MFA and DNS
+  provider keys.
+  - **⚠️ Rotating `SECRET_KEY` while `CSR_ENCRYPTION_KEY` is unset makes pending CSR keys
+    unrecoverable.** Import then fails with an explicit "delete this CSR and create a new one"
+    error rather than a misleading key-mismatch. Set an explicit `CSR_ENCRYPTION_KEY` if you
+    rotate `SECRET_KEY`. Certificates already imported are unaffected — their key lives on the
+    certificate row.
+- **API / UI / agents:** unchanged. No CSR endpoint ever returned the private key before or now,
+  and nothing about the CSR tab changes.
+- **Rollback:** the application downgrades cleanly — 1.10.0 starts normally against the same
+  database and every other feature is unaffected. The one casualty is a CSR **created on 1.10.1
+  and still pending**: 1.10.0 has no decrypt step, so it hands the Fernet token straight to the
+  key-pairing check. Measured on a real downgrade, the import then fails with
+  `HTTP 500 — Could not verify the certificate/key pair: key parse failed (encrypted?)`; it does
+  **not** silently pair the wrong key, and it does not corrupt anything. Import or delete CSRs
+  created on 1.10.1 before downgrading. Certificates already imported are unaffected, since their
+  key lives on the certificate row, and CSRs created before 1.10.1 are plaintext and still work.
+
+---
+
+# Upgrade Notes — v1.10.0 (GoDaddy DNS-01 provider)
+
+**Backward compatible & additive.** Nothing changes unless you select **GoDaddy** as an ACME
+account's DNS provider:
+
+- **Schema:** **no `SCHEMA_VERSION` bump.** The GoDaddy credentials (API Key + Secret) are stored
+  as two keys inside the *existing* encrypted
+  `letsencrypt_account_dns_credentials.credentials_encrypted` blob — no new table, no new column,
+  no migration.
+- **✅ Built-in roles are NOT re-seeded.** The re-seed warning in the v1.9.0 notes below is
+  triggered by a `SCHEMA_VERSION` bump. This release does not bump it, so any customization you
+  made to `super_admin` / `operator` / `security_admin` / `viewer` survives untouched.
+- **Permissions / API shape:** unchanged. `GET /api/letsencrypt/dns-providers` simply returns one
+  extra entry in its `providers` array; every request and response shape is identical, and the
+  credential form is rendered from that schema, so there is no frontend behaviour change either.
+- **Environment:** no new variable. GoDaddy credentials use the same Fernet-at-rest path as
+  Cloudflare (`DNS_PROVIDER_ENCRYPTION_KEY`, falling back to a key derived from `SECRET_KEY`).
+- **Agents:** zero agent changes. DNS-01 is invisible to agents; an issued certificate follows the
+  normal PENDING → Apply Management → agent pull pipeline exactly as before.
+- **Using it:** the API Key must be a **Production** key from `developer.godaddy.com/keys` (the
+  first key that dashboard issues is an OTE/test key and is rejected), the zone must be in the same
+  GoDaddy account, and that account needs at least one registered domain before GoDaddy permits DNS
+  API access. A Personal Access Token also works — paste it as the Key and leave the Secret blank.
+  Credentials are checked against the GoDaddy API before they are stored, so an invalid, OTE or
+  ineligible key fails at save time. Note the check is a **read**: a Personal Access Token that has
+  `domains.domain:read` but not `domains.dns:update` saves successfully and only fails at the first
+  publish, with a 403 in the order timeline.
+- **Rollback:** don't select GoDaddy. Existing Manual and Cloudflare accounts and all HTTP-01
+  issuance are untouched. **Downgrading after adopting GoDaddy is not a no-op**: on 1.9.0
+  `godaddy` is not a known provider, so any account still set to it degrades to the manual-confirm
+  path (in-flight DNS-01 orders wait for a confirmation nobody can give and expire after 48h, and
+  renewals stop), and the cleanup sweep marks published TXT records cleaned without removing them.
+  Before downgrading, switch affected accounts back to Manual or Cloudflare and let the reconcile
+  sweep remove outstanding `_acme-challenge` records first. The stored credential row itself is
+  inert — an encrypted blob for an unknown provider.
+
+---
+
+# Upgrade Notes — v1.9.0 (CSR creation)
+
+**Backward compatible & additive.** Upgrading to v1.9.0 changes nothing for existing
+clusters/agents until you create a CSR:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `10`, so on first start the (idempotent)
+  migration sequence re-runs once and adds **one new table** (`ssl_csrs`) plus its
+  indexes. **No existing table is altered**, existing rows are untouched, and the
+  **admin password is not reset** (the default-user seeding is guarded by an
+  existence check, not an upsert). No new permission strings are introduced — all
+  CSR endpoints are governed by the existing `ssl.create` / `ssl.read` /
+  `ssl.delete` permissions.
+- **⚠️ Built-in roles are re-seeded to their defaults (pre-existing behaviour of
+  every `SCHEMA_VERSION` bump — verified in a v1.8.10 → v1.9.0 upgrade drill).**
+  Because the version gate re-runs the whole sequence, `update_system_roles_to_enterprise_rbac()`
+  issues an unconditional `UPDATE roles SET … permissions = <defaults> WHERE name = …`
+  for the four **built-in** roles (`super_admin`, `operator`, `security_admin`,
+  `viewer`). **Any customization you made to a built-in role is reverted.** In the
+  drill, an `operator` role that had been narrowed by removing `apply.execute` and
+  `config.bulk_import` came back with both restored (57 → 59 permissions).
+  - **Roles you created yourself are NOT affected** — the re-seed matches on the four
+    built-in names only.
+  - This is not new in v1.9.0: it happens on every release that bumps
+    `SCHEMA_VERSION` (v1.7.0, v1.8.0, v1.8.8 …). It is documented as intentional at
+    `backend/database/migrations.py` (the "BUMP THIS … OR seeded/role data" note) —
+    the migration is treated as the authority on built-in-role contents.
+  - **If you have hardened a built-in role, do this:** export it before upgrading
+    (`GET /api/roles`), then re-apply your changes after the first start
+    (`PUT /api/roles/{id}`) — or, preferably, move your customization into a
+    purpose-made custom role, which survives every upgrade.
+- **Key storage:** CSR private keys are stored in the database like every other key
+  in the system (`ssl_certificates.private_key_content` and the ACME order keys).
+  The key is never returned by any CSR API endpoint, and after a successful import
+  the CSR row's key copy is set to NULL (the key then lives only on the certificate
+  row).
+- **Agents:** zero agent changes. Agents never read the new table; a CSR becomes
+  visible to agents only after its signed certificate is imported **and** applied via
+  Apply Management (the standard PENDING pipeline).
+- **Rollback:** simply don't use the CSR tab. The `ssl_csrs` table is inert when
+  empty; downgrading the application leaves it as an ignored extra table.
+
+---
+
 # Upgrade Notes — v1.7.0 (HA / VIP Keepalived management, Issue #27)
 
 **Backward compatible & opt-in.** Upgrading to v1.7.0 changes nothing for existing

backend/Dockerfile

@@ -37,5 +37,13 @@ USER appuser
 # Expose port
 EXPOSE 8000
 
-# Run the application in production mode (without --reload)
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+# Run the application in production mode (without --reload).
+# UVICORN_WORKERS (default 1) opts into multiple worker processes on multi-core
+# hosts; with 1 worker uvicorn runs in-process, identical to the flagless CMD
+# this replaces. Falls back to WEB_CONCURRENCY when UVICORN_WORKERS is unset
+# because flagless uvicorn honored WEB_CONCURRENCY (uvicorn config.py) — this
+# keeps any deployment that relied on it byte-for-byte compatible. Background
+# tasks are multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as
+# already exercised by the k8s HPA deployment. `exec` keeps uvicorn as PID 1
+# so signal handling is unchanged.
+CMD ["sh", "-c", "exec uvicorn main:app --host 0.0.0.0 --port 8000 --workers ${UVICORN_WORKERS:-${WEB_CONCURRENCY:-1}}"]

backend/agent_notifications.py

@@ -25,31 +25,42 @@ async def notify_agents_config_change(cluster_id: int, version_name: str) -> Lis
         for agent in agents:
             try:
                 agent_url = f"http://{agent['ip_address']}:8081"  # Agent default port
-                
+
                 async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
                     payload = {
                         "cluster_id": cluster_id,
                         "version_name": version_name,
                         "action": "config_update"
                     }
-                    
-                    async with session.post(f"{agent_url}/api/config/update", json=payload) as response:
-                        if response.status == 200:
-                            results.append({
-                                'node': agent['name'],
-                                'success': True,
-                                'message': f'Configuration updated successfully',
-                                'version': version_name
-                            })
-                            logger.info(f"✅ Agent {agent['name']} notified successfully")
-                        else:
-                            error_text = await response.text()
-                            results.append({
-                                'node': agent['name'],
-                                'success': False,
-                                'error': f'HTTP {response.status}: {error_text}'
-                            })
-                            logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
+
+                    # v1.11.0: instrumented so the code stays correct if the push
+                    # architecture is ever reverted. Unreachable today — see the
+                    # unconditional early return above.
+                    from utils.http_instrumentation import outbound_span, TARGET_AGENT
+
+                    push_url = f"{agent_url}/api/config/update"
+                    async with outbound_span(
+                        target=TARGET_AGENT, method="POST", url=push_url, request_body=payload
+                    ) as span:
+                        async with session.post(push_url, json=payload) as response:
+                            if response.status == 200:
+                                span.set_response(response.status, getattr(response, "headers", None))
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': True,
+                                    'message': f'Configuration updated successfully',
+                                    'version': version_name
+                                })
+                                logger.info(f"✅ Agent {agent['name']} notified successfully")
+                            else:
+                                error_text = await response.text()
+                                span.set_response(response.status, getattr(response, "headers", None), error_text)
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': False,
+                                    'error': f'HTTP {response.status}: {error_text}'
+                                })
+                                logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
                             
             except asyncio.TimeoutError:
                 results.append({

backend/auth_middleware.py

@@ -1,4 +1,4 @@
-from fastapi import HTTPException, status
+from fastapi import HTTPException, status, Header
 from typing import Optional, Dict, Any
 from jose import jwt
 import logging
@@ -92,6 +92,18 @@ async def get_current_user_from_token(authorization: Optional[str] = None) -> Op
             detail="Authentication failed"
         )
 
+async def require_authenticated_user(authorization: Optional[str] = Header(None)) -> Dict[str, Any]:
+    """FastAPI dependency: require a valid operator JWT, else 401.
+
+    Reads the Authorization header itself, so it can be attached at router or
+    route level to gate operator/UI endpoints that must not be public:
+        APIRouter(..., dependencies=[Depends(require_authenticated_user)])
+        @router.get(..., dependencies=[Depends(require_authenticated_user)])
+    Any authenticated user passes (no fine-grained RBAC here) — this restores the
+    pre-existing "logged-in users only" expectation without changing role access.
+    """
+    return await get_current_user_from_token(authorization)
+
 async def get_current_user_from_token_no_exception(authorization: Optional[str] = None) -> Optional[Dict[str, Any]]:
     """
     Get current user from JWT token without raising HTTPException.

backend/config.py

@@ -37,4 +37,65 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS = 30
 
 # Entity snapshot enabled by default (rollback functionality)
 # Set to "false" only if you need to disable snapshot temporarily
-ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true" 
+ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true"
+
+
+# ---------------------------------------------------------------------------
+# v1.11.0 — unified request/response log
+# ---------------------------------------------------------------------------
+# These four are deliberately ENV-only (not database settings): they decide
+# whether the middleware is even registered and how much memory the writer
+# queue may hold, so they must be resolvable before the DB pool exists.
+# Everything the operator tunes at runtime (retention, body capture, sampling,
+# excluded paths) lives in `system_settings` under the `requestlog.` category
+# and is editable from Settings → Request Log.
+
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() not in ("0", "false", "no", "off", "")
+
+
+def _int_env(name: str, default: int, minimum: int, maximum: int) -> int:
+    """Read an int env var, clamped. A malformed value falls back to the
+    default rather than crashing the process at import time."""
+    raw = os.getenv(name)
+    if raw is None or not raw.strip():
+        return default
+    try:
+        value = int(raw.strip())
+    except (TypeError, ValueError):
+        return default
+    return max(minimum, min(maximum, value))
+
+
+# Hard kill-switch. When false the logging middleware is never added to the
+# ASGI stack and neither the writer nor the prune task is started — literally
+# zero overhead, not even a settings lookup.
+REQUEST_LOG_ENABLED = _bool_env("REQUEST_LOG_ENABLED", True)
+# Per-worker in-process queue depth. When full, rows are DROPPED (counted, and
+# reported through GET /api/request-logs/stats) — the request path never blocks
+# on the database.
+REQUEST_LOG_QUEUE_MAX = _int_env("REQUEST_LOG_QUEUE_MAX", 2000, 100, 100000)
+# HARD memory ceiling for the same queue, per worker. The row count above does
+# NOT bound memory on its own, because how much a row weighs is an operator
+# setting: `requestlog.max_body_bytes` is editable from Settings and its stated
+# ceiling is 256 KB, which a row can carry twice (request + response). Measured
+# on the real dataclass, the two limits multiply out to:
+#
+#     defaults (2 000 rows x 8 KB)          33.9 MiB   3.3% of the 1 GiB pod limit
+#     max_body_bytes at its 256 KB ceiling   1003 MiB   at the pod limit
+#     REQUEST_LOG_QUEUE_MAX at its ceiling   1695 MiB   over the pod limit
+#
+# Both of those are reachable from documented, in-range values, and the drop
+# warning used to advise raising the queue - so following the tool's own advice
+# could OOM the worker. Whichever limit is hit FIRST now stops the queue, so
+# memory stays bounded no matter what the other is set to.
+REQUEST_LOG_QUEUE_MAX_BYTES = _int_env(
+    "REQUEST_LOG_QUEUE_MAX_BYTES", 64 * 1024 * 1024, 1024 * 1024, 1024 * 1024 * 1024
+)
+# Rows per batched INSERT: one pool acquire per batch, not per request.
+REQUEST_LOG_BATCH_SIZE = _int_env("REQUEST_LOG_BATCH_SIZE", 100, 1, 1000)
+# Max wait before a partial batch is flushed (milliseconds).
+REQUEST_LOG_FLUSH_MS = _int_env("REQUEST_LOG_FLUSH_MS", 500, 50, 10000)

backend/database/migrations.py

@@ -194,7 +194,14 @@ async def ensure_agents_table():
             'use_backend_rules': "ALTER TABLE frontends ADD COLUMN use_backend_rules JSONB DEFAULT '[]'::jsonb;",
             'request_headers': "ALTER TABLE frontends ADD COLUMN request_headers TEXT;",
             'response_headers': "ALTER TABLE frontends ADD COLUMN response_headers TEXT;",
-            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;"
+            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;",
+            # Issue #38: SPOE filter directives (e.g. Coraza WAF) and frontend
+            # log-format were silently dropped on bulk-import / manual edit
+            # because the parser recognised only a fixed set of directives.
+            # These nullable TEXT columns persist them verbatim (multi-line for
+            # `filters`), mirroring the request_headers/options passthrough.
+            'log_format': "ALTER TABLE frontends ADD COLUMN log_format TEXT;",
+            'filters': "ALTER TABLE frontends ADD COLUMN filters TEXT;"
         }
         
         for col, query in frontend_columns.items():
@@ -1358,6 +1365,10 @@ async def update_system_roles_to_enterprise_rbac():
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health', 'statistics.export',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — request/response log. `read` browses the log,
+                    # `manage` edits retention/capture settings and triggers a
+                    # manual purge.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.update', 'settings.system', 'settings.security',
                     'system.restart', 'system.logs', 'system.database', 'system.services', 'system.emergency'
                 ]
@@ -1377,7 +1388,11 @@ async def update_system_roles_to_enterprise_rbac():
                     'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
-                    'activity.read'
+                    'activity.read',
+                    # v1.11.0 — operators debug failing applies and ACME orders,
+                    # so they get read access to the request log; retention and
+                    # purge stay with the admins.
+                    'requestlog.read'
                 ]
             },
             'security_admin': {
@@ -1396,9 +1411,16 @@ async def update_system_roles_to_enterprise_rbac():
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — the request log is a security-forensics surface,
+                    # so the security admin gets both read and retention control.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.security'
                 ]
             },
+            # NOTE (v1.11.0): `viewer` deliberately gets NEITHER requestlog
+            # permission. Even redacted, captured request/response bodies are a
+            # far broader disclosure surface than the read-only configuration
+            # views a viewer is meant to have.
             'viewer': {
                 'display_name': 'Viewer',
                 'description': 'Read-only access to view configurations, statistics, and monitor system status',
@@ -1741,7 +1763,40 @@ async def ensure_agent_activity_logs_table():
 # columns on letsencrypt_accounts/letsencrypt_orders/acme_challenges and the brand-new
 # letsencrypt_account_dns_credentials table (ensure_letsencrypt_dns_credentials step).
 # All additive + idempotent; default challenge_type 'http-01' keeps existing flows byte-identical.
-SCHEMA_VERSION = 8
+# v1.8.8 (Issue #38 — SPOE filter + frontend log-format): bumped 8 -> 9 for the additive
+# `log_format` + `filters` TEXT columns on `frontends` (frontend_columns loop). Without this
+# bump, already-deployed databases (version >= 8) skip the whole migration run and never gain
+# the columns, so the frontends SELECT/INSERT would fail. Additive + idempotent + nullable;
+# existing rows stay NULL and render byte-identical.
+# v1.9.0 (CSR creation): bumped 9 -> 10 for the brand-new `ssl_csrs` table
+# (ensure_ssl_csrs_table step). Holds a locally generated private key + CSR PEM
+# until the operator imports the CA-signed certificate; the import creates a
+# normal ssl_certificates row and NULLs the key copy here. Additive + idempotent;
+# no existing table is altered, agents never read this table.
+# v1.10.4 (VIP adoption): bumped 10 -> 11 for the new `vip_discoveries` table plus two
+# additive columns (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`).
+# Holds the keepalived.conf an agent found already on a node so an existing VIP can be
+# adopted instead of retyped. Additive + idempotent; no existing table is altered and no
+# existing row changes. NOTE for the upgrade notes: a SCHEMA_VERSION bump re-seeds the four
+# built-in roles to their defaults, so role customizations are lost on this upgrade.
+# v1.11.0 (unified request/response log): bumped 11 -> 12 for the brand-new
+# `request_logs` table (ensure_request_logs_table), its retention-settings seed
+# (ensure_request_log_settings), and the new `requestlog.read` /
+# `requestlog.manage` permissions added to the built-in roles in
+# update_system_roles_to_enterprise_rbac().
+#
+# 12, NOT 11. The feature branch was cut when this constant was still 10 and
+# proposed 11, but 11 was taken in the meantime by v1.10.4 (vip_discoveries)
+# above. Landing it as 11 would be silently inert: run_all_migrations() returns
+# early on `applied_version >= SCHEMA_VERSION`, so every database already at 11
+# would skip the whole sequence and get neither the table nor the permissions,
+# while a fresh install would get both. Same class of bug the bump exists to
+# prevent, one number later.
+#
+# Additive + idempotent; no existing table is altered, agents never read this
+# table. Same caveat as v1.10.4: this bump re-seeds the four built-in roles to
+# their defaults, so export role customizations before upgrading.
+SCHEMA_VERSION = 12
 
 
 async def run_all_migrations():
@@ -1878,12 +1933,246 @@ async def _run_all_migrations_inner():
     await ensure_mfa_columns()
 
     # Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
-    # MUST stay last: FK-references haproxy_cluster_pools/agents/users, all created above.
+    # MUST run after its FK targets (haproxy_cluster_pools/agents/users), all created above.
     await ensure_vip_tables()
 
+    # v1.9.0 — CSR creation: brand-new ssl_csrs table. FK-references
+    # ssl_certificates/users, both created above.
+    await ensure_ssl_csrs_table()
+
+    # v1.11.0 — unified request/response log: brand-new request_logs table
+    # (no FK targets) plus the seed for its operator-tunable retention
+    # settings. Both are additive and idempotent.
+    await ensure_request_logs_table()
+    await ensure_request_log_settings()
+
     logger.info("Database migrations completed successfully.")
 
 
+async def ensure_ssl_csrs_table():
+    """v1.9.0 — CSR (Certificate Signing Request) creation. Additive only:
+    one brand-new table (ssl_csrs) + indexes. No ALTER of any existing table,
+    so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (ssl_certificates, users)
+    are created earlier in the sequence.
+
+    A CSR row holds a locally generated private key + CSR PEM until the
+    operator imports the CA-signed certificate. The import creates a normal
+    ssl_certificates row (source='csr', last_config_status='PENDING') and
+    NULLs the private_key_pem copy here — the key then lives only on the
+    certificate row, like every other key in the system. Agents never read
+    this table: the agent SSL delivery endpoint selects from
+    ssl_certificates only, so a pending CSR can never leak to an agent.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS ssl_csrs (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(100) NOT NULL,
+                common_name VARCHAR(253) NOT NULL,
+                subject JSONB NOT NULL DEFAULT '{}'::jsonb,
+                sans JSONB NOT NULL DEFAULT '[]'::jsonb,
+                key_algorithm VARCHAR(20) NOT NULL DEFAULT 'rsa-2048',
+                csr_pem TEXT NOT NULL,
+                private_key_pem TEXT,
+                status VARCHAR(20) NOT NULL DEFAULT 'pending',
+                ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL,
+                completed_at TIMESTAMP,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT ssl_csrs_status_check CHECK (status IN ('pending', 'completed'))
+            );
+        """)
+
+        # Only PENDING CSRs reserve their name: the name becomes the
+        # ssl_certificates.name (and thus /etc/ssl/haproxy/{name}.pem on every
+        # agent) at import time, so two open CSRs must not target the same
+        # cert name. Completed CSRs are history and may share a name across
+        # reissues — mirrors the uq_vip_name_active partial-index rationale.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_ssl_csrs_name_pending ON ssl_csrs(name) WHERE status = 'pending';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_status ON ssl_csrs(status);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_cert ON ssl_csrs(ssl_certificate_id);"
+        )
+
+        logger.info("ssl_csrs table ensured (v1.9.0 CSR creation)")
+    except Exception as e:
+        logger.error(f"Error ensuring ssl_csrs table: {e}")
+        # Re-raise (ensure_ssl_cluster_junction_table precedent): this step is
+        # part of the SCHEMA_VERSION=10 bump, and run_all_migrations() records
+        # the marker only after the inner sequence completes cleanly. Swallowing
+        # a failure here would stamp version 10 with no ssl_csrs table, and the
+        # version gate would then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_request_logs_table():
+    """v1.11.0 — unified inbound/outbound request/response log.
+
+    Additive only: one brand-new table (request_logs) + indexes. No ALTER of
+    any existing table; agents never read this table.
+
+    Deliberately has NO foreign key on user_id. This is the highest-volume
+    table in the system — one row per API call — and per-insert FK validation
+    is not worth it here; `username` is a denormalized snapshot so a row stays
+    readable after the user who made the request is deleted. That is also the
+    correct audit semantics: the record should outlive the account.
+
+    Fully idempotent (CREATE TABLE/INDEX IF NOT EXISTS). Uses only PostgreSQL
+    9.5+ features (BIGSERIAL, JSONB, partial indexes, varchar_pattern_ops) so
+    there is no server-version floor beyond what the rest of the schema needs.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS request_logs (
+                id                  BIGSERIAL PRIMARY KEY,
+                request_id          VARCHAR(64) NOT NULL,
+                direction           VARCHAR(8)  NOT NULL,
+                target              VARCHAR(32),
+                method              VARCHAR(10) NOT NULL,
+                url                 TEXT        NOT NULL,
+                path                VARCHAR(512),
+                query_params        JSONB,
+                status_code         INTEGER,
+                status_class        SMALLINT    NOT NULL DEFAULT 0,
+                duration_ms         INTEGER     NOT NULL DEFAULT 0,
+                user_id             INTEGER,
+                username            VARCHAR(50),
+                client_ip           INET,
+                user_agent          TEXT,
+                request_headers     JSONB,
+                request_body        JSONB,
+                request_body_bytes  INTEGER     NOT NULL DEFAULT 0,
+                response_headers    JSONB,
+                response_body       JSONB,
+                response_body_bytes INTEGER     NOT NULL DEFAULT 0,
+                error               TEXT,
+                truncated           BOOLEAN     NOT NULL DEFAULT FALSE,
+                created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                CONSTRAINT request_logs_direction_check
+                    CHECK (direction IN ('inbound', 'outbound'))
+            );
+        """)
+
+        # Indexes run UNCONDITIONALLY on every startup, not only on first
+        # creation (the R16-2 rule established for acme_order_events): an older
+        # deploy that raced ahead of an index would otherwise be stuck doing
+        # sequential scans forever. All are IF NOT EXISTS, so re-running is free.
+
+        # --- read paths: the filters the log viewer actually issues ---
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_created_at "
+            "ON request_logs(created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_dir_created "
+            "ON request_logs(direction, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_status_created "
+            "ON request_logs(status_class, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_user_created "
+            "ON request_logs(user_id, created_at DESC) WHERE user_id IS NOT NULL;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_target_created "
+            "ON request_logs(target, created_at DESC) WHERE target IS NOT NULL;"
+        )
+        # Correlates one inbound row with the outbound calls it caused — this is
+        # what makes "which request went where" readable as a single trace.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_request_id "
+            "ON request_logs(request_id);"
+        )
+        # Prefix search on path (LIKE 'x%') needs pattern_ops to be usable under
+        # a non-C collation.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_path_prefix "
+            "ON request_logs(path varchar_pattern_ops);"
+        )
+
+        # --- prune paths: the TTL delete is split by outcome, so a plain
+        # (status_class, created_at) index would still range-scan the half it
+        # is not interested in.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_ok "
+            "ON request_logs(created_at) WHERE status_class BETWEEN 1 AND 3;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_err "
+            "ON request_logs(created_at) WHERE status_class = 0 OR status_class >= 4;"
+        )
+
+        logger.info("request_logs table ensured (v1.11.0 request/response log)")
+    except Exception as e:
+        logger.error(f"Error ensuring request_logs table: {e}")
+        # Re-raise (ssl_csrs precedent): this step is part of the
+        # SCHEMA_VERSION=11 bump and the version marker is written only after
+        # the inner sequence completes cleanly. Swallowing here would stamp
+        # version 11 with no request_logs table, and the version gate would
+        # then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_request_log_settings():
+    """v1.11.0 — seed the request/response-log retention defaults.
+
+    Runs UNCONDITIONALLY rather than inside an `if not table_exists:` branch,
+    so an install that already has `system_settings` picks the rows up too.
+    ON CONFLICT DO NOTHING means an operator's tuning is never overwritten by a
+    later upgrade.
+
+    Defaults are mirrored in utils/request_log_settings.py; the pair is pinned
+    by backend/tests/test_request_log_settings.py so they cannot drift apart.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            INSERT INTO system_settings (key, value, category, description) VALUES
+            ('requestlog.enabled', 'true', 'requestlog', 'Master switch for the request/response log'),
+            ('requestlog.capture_inbound', 'true', 'requestlog', 'Log inbound API calls'),
+            ('requestlog.capture_outbound', 'true', 'requestlog', 'Log outbound HTTP calls made by the backend'),
+            ('requestlog.capture_bodies', 'true', 'requestlog', 'Capture redacted, size-capped request/response bodies'),
+            ('requestlog.capture_get', 'true', 'requestlog', 'Log inbound GET requests'),
+            ('requestlog.capture_agent_success', 'false', 'requestlog', 'Log SUCCESSFUL agent polls too (failures are always logged); off by default because the row rate scales with fleet size, not operator activity'),
+            ('requestlog.max_body_bytes', '8192', 'requestlog', 'Per-body capture cap in bytes'),
+            ('requestlog.sample_rate', '1.0', 'requestlog', 'Sampling rate for successful inbound requests (errors always 1.0)'),
+            ('requestlog.exclude_paths', '["/api/request-logs","/api/health","/api/docs","/api/redoc","/api/openapi.json","/.well-known/acme-challenge","/api/agents/heartbeat","/static","/favicon.ico"]', 'requestlog', 'Path prefixes that are never logged'),
+            ('requestlog.success_retention_days', '7', 'requestlog', 'Retention for 1xx/2xx/3xx rows, in days'),
+            ('requestlog.error_retention_days', '30', 'requestlog', 'Retention for 4xx/5xx/transport-error rows, in days'),
+            ('requestlog.max_rows', '500000', 'requestlog', 'Hard row cap; oldest rows are pruned beyond this'),
+            ('requestlog.prune_interval_minutes', '60', 'requestlog', 'Minimum interval between retention prune passes')
+            ON CONFLICT (key) DO NOTHING
+        """)
+        logger.info("request_log retention settings seeded (v1.11.0)")
+    except Exception as e:
+        logger.error(f"Error seeding request_log settings: {e}")
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
 async def ensure_mfa_columns():
     """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
 
@@ -2072,6 +2361,45 @@ async def ensure_vip_tables():
             "CREATE INDEX IF NOT EXISTS idx_vip_members_agent ON vip_members(agent_id);"
         )
 
+        # ── v1.10.4 — VIP adoption: what the agent found already on the node ──────────
+        # A node with a hand-maintained keepalived.conf reports it here so an existing VIP can
+        # be adopted instead of retyped. One row per agent (the file is per-node); the agent
+        # only reports a config it does NOT own, and only when the content changed.
+        #
+        # SECRETS: `raw_config` is stored MASKED (auth_pass replaced) because it is served to
+        # the UI. The real VRRP password is Fernet-encrypted in auth_pass_encrypted, mirroring
+        # vip_instances, so adoption can carry it into the managed VIP without it ever being
+        # readable through the API or a DB dump. `analysis` is the parser output with auth_pass
+        # stripped out.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_discoveries (
+                id SERIAL PRIMARY KEY,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                config_path VARCHAR(500) NOT NULL,
+                config_hash VARCHAR(64) NOT NULL,
+                is_managed BOOLEAN NOT NULL DEFAULT FALSE,
+                raw_config_masked TEXT,
+                auth_pass_encrypted TEXT,
+                analysis JSONB,
+                parse_error TEXT,
+                adopted_vip_id INTEGER REFERENCES vip_instances(id) ON DELETE SET NULL,
+                reported_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_discovery_agent_unique UNIQUE (agent_id)
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_discoveries_agent ON vip_discoveries(agent_id);"
+        )
+        # Adoption provenance + the one-shot takeover authorisation. The agent refuses to
+        # overwrite a keepalived.conf that lacks our ownership marker, which is exactly the
+        # guard adoption has to pass. Rather than weaken it, an adopted VIP carries the hash of
+        # the file we analysed: the agent takes over ONLY if the file on disk still hashes to
+        # that value, so a config that changed after adoption is never clobbered.
+        await conn.execute(
+            "ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS adopted_at TIMESTAMP;")
+        await conn.execute(
+            "ALTER TABLE vip_members ADD COLUMN IF NOT EXISTS takeover_expected_hash VARCHAR(64);")
+
         logger.info("✅ VIP tables ensured (Issue #27 — HA/VIP Keepalived management)")
     except Exception as e:
         logger.error(f"Failed to ensure VIP tables: {e}")
@@ -2214,19 +2542,19 @@ async def create_initial_system_data(conn):
                 'name': 'super_admin',
                 'display_name': 'Super Administrator', 
                 'description': 'Full system access with all permissions',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'operator',
                 'display_name': 'Operator',
                 'description': 'Daily operational access for managing HAProxy configurations',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request","requestlog.read"]
             },
             {
                 'name': 'security_admin',
                 'display_name': 'Security Administrator',
                 'description': 'Security-focused access for WAF rules and SSL certificates',
-                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'viewer',

backend/haproxy_client.py

@@ -61,14 +61,27 @@ class HAProxyClient:
             if self.stats_username and self.stats_password:
                 auth = aiohttp.BasicAuth(self.stats_username, self.stats_password)
             
+            # v1.11.0: instrumented for completeness. NOTE the CSV body is
+            # deliberately NOT handed to the span — a full stats dump is large,
+            # changes every poll, and has no diagnostic value in an audit row;
+            # status + duration is what matters. `auth` is likewise never logged:
+            # aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+            # password.
+            from utils.http_instrumentation import outbound_span, TARGET_HAPROXY_STATS
+
             async with aiohttp.ClientSession() as session:
-                async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
-                    if response.status == 200:
-                        csv_data = await response.text()
-                        return self._parse_csv_stats(csv_data)
-                    else:
-                        logger.warning(f"HTTP stats request failed with status {response.status}")
-                        return self._get_fallback_stats()
+                async with outbound_span(
+                    target=TARGET_HAPROXY_STATS, method="GET", url=url,
+                    capture_body=False, capture_response_body=False,
+                ) as span:
+                    async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
+                        span.set_response(response.status, getattr(response, "headers", None))
+                        if response.status == 200:
+                            csv_data = await response.text()
+                            return self._parse_csv_stats(csv_data)
+                        else:
+                            logger.warning(f"HTTP stats request failed with status {response.status}")
+                            return self._get_fallback_stats()
         except Exception as e:
             logger.error(f"HTTP stats request failed: {e}")
             return self._get_fallback_stats()

backend/main.py

@@ -8,9 +8,13 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-# Build/deploy marker for the v1.8.x (Issue #35, DNS-01) rollout — ensures the pipeline ships this commit's image.
-_version_info = {"version": "1.8.1", "releaseName": "ACME DNS-01 fixes (Cloudflare token, EAB, Apply Management)", "releaseDate": "2026-06-24"}
-for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
+# Single source of truth: backend/version.json, which sits next to this module and is baked into
+# every image by `COPY . .` (build context ./backend) — no pipeline staging needed. The literal
+# below is only a last-resort "file missing" marker; it is deliberately NOT a real version so it can
+# never silently drift out of sync (this exact drift showed a stale version after v1.8.5/v1.8.6).
+# Keep the canonical version ONLY in backend/version.json — test_version_consistency.py enforces it.
+_version_info = {"version": "unknown", "releaseName": "unknown", "releaseDate": ""}
+for _vpath in [os.path.join(os.path.dirname(__file__), "version.json"), "/app/version.json"]:
     try:
         with open(_vpath) as _vf:
             _version_info = json.load(_vf)
@@ -25,7 +29,7 @@ for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), ".."
 
 # Import configurations and database
 
-from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL
+from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL, REQUEST_LOG_ENABLED
 from database.connection import redis_client, get_database_connection, close_database_connection, init_database_pool, close_database_pool
 from database.migrations import run_all_migrations
 
@@ -43,6 +47,8 @@ from routers.acme_diagnostics import router as acme_diagnostics_router
 from routers.site_wizard import router as site_wizard_router
 from routers.mfa import router as mfa_router
 from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
+from routers.csr import router as csr_router  # v1.9.0 — CSR creation (in-app key+CSR generation, signed-cert import)
+from routers.request_logs import router as request_logs_router  # v1.11.0 — unified request/response log
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -51,6 +57,10 @@ from middleware.error_handler import (
     GlobalExceptionHandler, get_error_statistics
 )
 from middleware.activity_logger import log_activity_middleware
+from middleware.request_logger import RequestResponseLogMiddleware  # v1.11.0
+from utils.http_instrumentation import begin_background_trace  # v1.11.0
+from utils.request_log_settings import refresh_config as refresh_request_log_config
+from utils.request_log_sink import request_log_sink
 
 # Setup structured logging
 logger = setup_production_logging(LOG_LEVEL)
@@ -227,6 +237,8 @@ Most operations are **cluster-scoped**:
 async def monitor_agent_status():
     """Background task to monitor agent status and mark offline agents"""
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("agent_status_monitor")
         try:
             conn = await get_database_connection()
             
@@ -271,6 +283,10 @@ async def complete_pending_acme_orders():
     """
     await asyncio.sleep(60)
     while True:
+        # v1.11.0: one correlation id per TICK, so the outbound rows for this
+        # pass group together and do not merge with every other pass this
+        # process has ever run.
+        begin_background_trace("acme_complete_orders")
         try:
             conn_check = await get_database_connection()
             try:
@@ -318,7 +334,6 @@ async def complete_pending_acme_orders():
                                     OR dns01_last_attempt_at < NOW() - (
                                         (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
                                         || ' minutes')::INTERVAL
-                                    )
                                 )
                             )
                         )
@@ -641,6 +656,8 @@ async def check_letsencrypt_renewals():
     """
     await asyncio.sleep(120)
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("acme_renewals")
         conn = None
         try:
             conn = await get_database_connection()
@@ -838,6 +855,41 @@ async def cleanup_stuck_agent_upgrades():
         # Wait 120 seconds (2 minutes) before next check
         await asyncio.sleep(120)
 
+async def prune_request_logs_loop():
+    """v1.11.0 — retention prune for `request_logs`.
+
+    Kept independent of the ACME prune loop on purpose: that one is gated on
+    the `letsencrypt_orders` table existing, which would silently disable this
+    prune on an install that never uses ACME.
+
+    The 5-minute tick is only a heartbeat — the real gate is the DB watermark
+    plus `requestlog.prune_interval_minutes`, so N replicas ticking every 5
+    minutes still produce one pass per configured interval.
+    """
+    # Stagger past startup so migrations and the first request burst are done.
+    await asyncio.sleep(180)
+
+    while True:
+        try:
+            conn = await get_database_connection()
+            try:
+                table_exists = await conn.fetchval("""
+                    SELECT EXISTS (
+                        SELECT 1 FROM information_schema.tables
+                        WHERE table_name = 'request_logs'
+                    )
+                """)
+            finally:
+                await close_database_connection(conn)
+
+            if table_exists:
+                from utils.request_log_prune import prune_request_logs_if_due
+                await prune_request_logs_if_due()
+        except Exception as e:
+            logger.error(f"Error in request_logs prune loop: {e}")
+
+        await asyncio.sleep(300)
+
 # Production middleware stack (order matters!)
 app.add_middleware(PerformanceMonitoringMiddleware, slow_request_threshold_ms=1000)
 app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/docs", "/redoc"])
@@ -845,15 +897,35 @@ app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/do
 # Activity logging middleware - must be before CORS
 app.middleware("http")(log_activity_middleware)
 
-# CORS middleware  
+# CORS middleware
 app.add_middleware(
     CORSMiddleware,
     allow_origins=CORS_ORIGINS,
     allow_credentials=True,
     allow_methods=["*"],
     allow_headers=["*"],
+    # v1.11.0: without an explicit expose list, browser JS on a cross-origin
+    # deployment cannot read ANY of these — so an operator could see the
+    # X-Request-ID in devtools but the app could never quote it back. Same-origin
+    # (nginx) deployments were already fine; this fixes the split-origin case.
+    expose_headers=["X-Correlation-ID", "X-Response-Time", "X-Request-ID"],
 )
 
+# v1.11.0 — unified request/response log.
+#
+# MUST be the LAST add_middleware call: Starlette inserts each new middleware at
+# index 0, so the last registration ends up OUTERMOST. Outermost is what we want:
+#   (a) we see the exact status/headers/body the client receives, including the
+#       JSONResponse that RequestLoggingMiddleware fabricates from an exception
+#       it swallowed, and
+#   (b) we seed correlation_id_context BEFORE RequestLoggingMiddleware calls
+#       get_correlation_id(), so X-Correlation-ID matches request_logs.request_id.
+#
+# REQUEST_LOG_ENABLED=false keeps it out of the ASGI stack entirely — not a
+# runtime branch, genuinely zero overhead.
+if REQUEST_LOG_ENABLED:
+    app.add_middleware(RequestResponseLogMiddleware)
+
 # Global exception handlers
 from fastapi.exceptions import RequestValidationError
 from starlette.exceptions import HTTPException as StarletteHTTPException
@@ -889,6 +961,8 @@ app.include_router(dashboard_stats_router)  # HAProxy stats dashboard
 app.include_router(agent_router)
 app.include_router(waf_router)
 app.include_router(ssl_router)
+app.include_router(csr_router)  # v1.9.0: CSR creation (in-app key+CSR generation, signed-cert import)
+app.include_router(request_logs_router)  # v1.11.0: unified request/response log
 app.include_router(security_router)
 app.include_router(configuration_router)
 app.include_router(settings_router)
@@ -1026,6 +1100,17 @@ async def startup_event():
         # Decoupled from auto_renew_enabled flag so user-initiated orders also complete.
         asyncio.create_task(complete_pending_acme_orders())
         logger.info("ACME order auto-completion task started (60s checks, replica-safe)")
+
+        # v1.11.0 — request/response log: load the operator's capture/retention
+        # policy, then start the batching writer and the retention prune.
+        # Guarded by the env kill-switch so a deployment that turned the log off
+        # pays for neither task.
+        if REQUEST_LOG_ENABLED:
+            await refresh_request_log_config()
+            asyncio.create_task(request_log_sink.run())
+            logger.info("Request/response log sink started (batching writer)")
+            asyncio.create_task(prune_request_logs_loop())
+            logger.info("Request/response log retention prune task started")
         
         # Create test activity log entry to verify system is working
         try:
@@ -1054,6 +1139,16 @@ async def shutdown_event():
     """Cleanup on shutdown"""
     logger.info("HAProxy OpenManager API shutting down...")
 
+    # v1.11.0: flush queued request-log rows FIRST. The sink's writer is a
+    # `while True` loop, so it can never satisfy the asyncio.wait below — the
+    # rows still sitting in its queue would be lost when the pool closes.
+    try:
+        flushed = await request_log_sink.flush(timeout=3.0)
+        if flushed:
+            logger.info(f"Flushed {flushed} queued request-log row(s)")
+    except Exception as flush_err:
+        logger.warning(f"request-log flush skipped: {flush_err}")
+
     # R18c audit fix (round 3 #5): drain pending fire-and-forget
     # background tasks BEFORE closing the DB pool. The audit
     # logger middleware (`activity_logger.py`) and the wizard
@@ -1106,9 +1201,19 @@ async def get_version():
     return _version_info
 
 @app.get("/.well-known/acme-challenge/{token}")
-async def serve_acme_challenge(token: str):
+async def serve_acme_challenge(token: str, request: Request):
     """Serve ACME HTTP-01 challenge token. Public endpoint, no auth required."""
-    logger.info(f"ACME-CHALLENGE: Incoming request for token={token[:32]}...")
+    # Log who reached us. When HTTP-01 fails, the first question is always "did the
+    # request get here at all?" — and the answer separates a broken challenge-backend
+    # address (nothing arrives) from a wrong response (arrives, wrong body). The peer
+    # is normally the HAProxy node; X-Forwarded-For carries the CA when the frontend
+    # sets `option forwardfor`.
+    _peer = request.client.host if request.client else 'unknown'
+    _xff = request.headers.get('x-forwarded-for') or '-'
+    logger.info(
+        f"ACME-CHALLENGE: Incoming request for token={token[:32]}... "
+        f"peer={_peer} xff={_xff} host={request.headers.get('host') or '-'}"
+    )
     conn = None
     try:
         conn = await get_database_connection()

backend/middleware/request_logger.py

@@ -0,0 +1,321 @@
+"""v1.11.0 — inbound half of the unified request/response log.
+
+Pure ASGI on purpose, NOT BaseHTTPMiddleware:
+
+  * `BaseHTTPMiddleware` hands the response back as a
+    `starlette.middleware.base._StreamingResponse`, which has no `.body` to
+    read, and
+  * `await request.body()` inside a `dispatch()` DRAINS the receive channel.
+    `POST /api/agents/heartbeat` (routers/agent.py) reads the raw stream
+    itself, as does the validation-error body preview in
+    middleware/error_handler.py — draining it here would break both.
+
+So we never consume anything: we TEE. `receive` and `send` are wrapped, every
+message is forwarded verbatim, and a size-capped copy is kept for the log row.
+Cost per in-flight request is therefore bounded at ~2 × max_body_bytes (8 KB
+by default), not the size of the upload.
+
+Registration: this MUST be the LAST `app.add_middleware(...)` call, because
+Starlette inserts at index 0 — the last registration is the OUTERMOST
+middleware. Outermost is what we want: we then see the exact status and body
+the client receives (including the JSONResponse that RequestLoggingMiddleware
+fabricates out of a swallowed exception), and we can seed
+`correlation_id_context` before anything downstream reads it.
+"""
+import logging
+import time
+import uuid
+from typing import Any, Dict, List, Optional, Tuple
+
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from utils.logging_config import correlation_id_context
+from utils.request_log_redaction import is_capturable_content_type, scrub_query_string
+from utils.request_log_settings import get_config
+from utils.request_log_sink import (
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    request_id_context,
+    request_log_sink,
+)
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Hard floor, NOT settable away through `requestlog.exclude_paths`. Without it
+# an operator who clears the exclude list turns the log viewer into a machine
+# that logs itself reading its own logs.
+_ALWAYS_EXCLUDED: Tuple[str, ...] = ("/api/request-logs",)
+
+
+def _header(scope: Scope, name: bytes) -> Optional[str]:
+    for key, value in scope.get("headers") or ():
+        if key == name:
+            try:
+                return value.decode("latin-1")
+            except Exception:
+                return None
+    return None
+
+
+def _identify(scope: Scope) -> Tuple[Optional[int], Optional[str]]:
+    """Resolve the caller from the JWT locally — NO database round-trip.
+
+    `log_activity_middleware` already pays a `SELECT ... FROM users` per
+    non-GET request; this middleware runs on every request including GETs, so a
+    second lookup per call is not acceptable. The token issued at
+    routers/auth.py carries both `user_id` and `username`, which is everything
+    the log row needs.
+
+    A token that fails to decode simply yields (None, None): this is a logging
+    path, not an authorization path — the real auth check still runs
+    downstream.
+    """
+    raw = _header(scope, b"authorization")
+    if not raw:
+        return None, None
+    token = raw[7:].strip() if raw.lower().startswith("bearer ") else raw.strip()
+    if not token or token in ("null", "undefined") or token.count(".") != 2:
+        return None, None
+    try:
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None, None
+
+    raw_uid = payload.get("user_id") or payload.get("sub")
+    try:
+        user_id = int(raw_uid) if raw_uid is not None else None
+    except (TypeError, ValueError):
+        user_id = None
+    username = payload.get("username")
+    return user_id, (str(username) if username else None)
+
+
+def _is_agent_call(scope: Scope) -> bool:
+    """True when the caller authenticated as an AGENT rather than as a user.
+
+    Every call the installed agent makes carries `X-API-Key` and never an
+    `Authorization` header (linux_install.sh / macos_install.sh: heartbeat,
+    config, pending-requests, upgrade-status, keepalived-*, config-response are
+    all `-H "X-API-Key: $AGENT_TOKEN"`), while the UI carries a JWT and never an
+    agent key. The one endpoint that accepts either -
+    `POST /api/agents/generate-install-script`, used by agent self-upgrade -
+    is correctly classified by the same rule: an operator generating a script
+    sends Authorization, the self-upgrading agent sends only the key.
+
+    Header-only, so it costs two scope reads and no database round-trip.
+    """
+    if _header(scope, b"authorization"):
+        return False
+    return bool(_header(scope, b"x-api-key"))
+
+
+def _client_ip(scope: Scope) -> Optional[str]:
+    """The peer address only.
+
+    `request_logs.client_ip` is an INET column, so a comma-joined
+    X-Forwarded-For string would raise on INSERT (the same trap as
+    `user_activity_logs.ip_address`). The XFF header is still captured — it is
+    on the header allowlist — so the original client is not lost behind a proxy.
+    """
+    client = scope.get("client")
+    if not client:
+        return None
+    try:
+        return str(client[0])
+    except Exception:
+        return None
+
+
+class RequestResponseLogMiddleware:
+    def __init__(self, app: ASGIApp):
+        self.app = app
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope.get("type") != "http":
+            await self.app(scope, receive, send)
+            return
+
+        cfg = get_config()
+        path = scope.get("path", "") or ""
+        method = scope.get("method", "") or ""
+
+        if (
+            not cfg.enabled
+            or not cfg.capture_inbound
+            # OPTIONS never reaches a handler — CORSMiddleware short-circuits
+            # it below us — and a preflight carries no information worth a row.
+            or method == "OPTIONS"
+            or (method == "GET" and not cfg.capture_get)
+            or any(path.startswith(prefix) for prefix in _ALWAYS_EXCLUDED)
+            or any(path.startswith(prefix) for prefix in cfg.exclude_paths)
+        ):
+            await self.app(scope, receive, send)
+            return
+
+        request_id = uuid.uuid4().hex
+        # Seed the id BEFORE the downstream app runs so error_handler's
+        # get_correlation_id() adopts ours instead of minting a second one; the
+        # X-Correlation-ID header then matches request_logs.request_id.
+        cid_token = correlation_id_context.set(request_id[:8])
+        rid_token = request_id_context.set(request_id)
+
+        cap = cfg.max_body_bytes if cfg.capture_bodies else 0
+        req_ctype = _header(scope, b"content-type")
+        req_capturable = is_capturable_content_type(req_ctype)
+
+        req_buf = bytearray()
+        res_buf = bytearray()
+        state = {
+            "req_bytes": 0,
+            "res_bytes": 0,
+            "status": None,
+            "res_headers": {},
+            "res_ctype": None,
+            "res_capturable": True,
+        }
+
+        async def tee_receive() -> Dict[str, Any]:
+            message = await receive()
+            try:
+                if message.get("type") == "http.request":
+                    chunk = message.get("body", b"") or b""
+                    state["req_bytes"] += len(chunk)
+                    if cap and req_capturable and len(req_buf) < cap:
+                        req_buf.extend(chunk[: cap - len(req_buf)])
+            except Exception:
+                pass
+            return message  # forwarded verbatim, always
+
+        async def tee_send(message: Dict[str, Any]) -> None:
+            try:
+                mtype = message.get("type")
+                if mtype == "http.response.start":
+                    state["status"] = message.get("status")
+                    raw_headers: List[Tuple[bytes, bytes]] = message.get("headers") or []
+                    headers = {}
+                    for key, value in raw_headers:
+                        try:
+                            headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+                        except Exception:
+                            continue
+                    state["res_headers"] = headers
+                    state["res_ctype"] = headers.get("content-type")
+                    state["res_capturable"] = is_capturable_content_type(state["res_ctype"])
+                    # Hand the id to the client so a user reporting a problem can
+                    # quote it and an operator can find the exact row.
+                    if isinstance(raw_headers, list):
+                        raw_headers.append((b"x-request-id", request_id.encode("latin-1")))
+                elif mtype == "http.response.body":
+                    chunk = message.get("body", b"") or b""
+                    state["res_bytes"] += len(chunk)
+                    if cap and state["res_capturable"] and len(res_buf) < cap:
+                        res_buf.extend(chunk[: cap - len(res_buf)])
+            except Exception:
+                pass
+            await send(message)  # forwarded verbatim, always
+
+        started = time.perf_counter()
+        error_text: Optional[str] = None
+        try:
+            await self.app(scope, tee_receive, tee_send)
+        except Exception as exc:
+            # Almost never taken: RequestLoggingMiddleware sits below us and
+            # converts exceptions into a JSONResponse first. It IS taken for
+            # paths on that middleware's own exclude list, so the row still has
+            # to be recorded before the exception continues upward.
+            error_text = f"{type(exc).__name__}: {exc}"[:2000]
+            raise
+        finally:
+            duration_ms = int((time.perf_counter() - started) * 1000)
+            try:
+                self._record(
+                    scope=scope,
+                    request_id=request_id,
+                    method=method,
+                    path=path,
+                    duration_ms=duration_ms,
+                    status=state["status"],
+                    req_buf=bytes(req_buf),
+                    req_bytes=state["req_bytes"],
+                    req_ctype=req_ctype,
+                    res_buf=bytes(res_buf),
+                    res_bytes=state["res_bytes"],
+                    res_ctype=state["res_ctype"],
+                    res_headers=state["res_headers"],
+                    error_text=error_text,
+                )
+            except Exception as exc:  # pragma: no cover - defensive
+                logger.debug(f"request_log: failed to record inbound row: {exc}")
+            try:
+                correlation_id_context.reset(cid_token)
+                request_id_context.reset(rid_token)
+            except Exception:
+                pass
+
+    @staticmethod
+    def _record(
+        *,
+        scope: Scope,
+        request_id: str,
+        method: str,
+        path: str,
+        duration_ms: int,
+        status: Optional[int],
+        req_buf: bytes,
+        req_bytes: int,
+        req_ctype: Optional[str],
+        res_buf: bytes,
+        res_bytes: int,
+        res_ctype: Optional[str],
+        res_headers: Dict[str, str],
+        error_text: Optional[str],
+    ) -> None:
+        raw_query = scope.get("query_string") or b""
+        try:
+            query = raw_query.decode("latin-1")
+        except Exception:
+            query = ""
+        scrubbed_query, query_params = scrub_query_string(query)
+
+        user_id, username = _identify(scope)
+
+        req_headers = {}
+        for key, value in scope.get("headers") or ():
+            try:
+                req_headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+            except Exception:
+                continue
+
+        request_log_sink.offer(
+            RequestLogRow(
+                request_id=request_id,
+                direction="inbound",
+                # Who was on the other end. `offer()` uses this to drop
+                # SUCCESSFUL agent polls, which are ~9 800 rows/day per node and
+                # would otherwise make the table's size a function of fleet size.
+                target=TARGET_INBOUND_AGENT if _is_agent_call(scope) else None,
+                method=method,
+                url=path + (("?" + scrubbed_query) if scrubbed_query else ""),
+                path=path,
+                query_string=scrubbed_query or None,
+                query_params=query_params,
+                status_code=status,
+                duration_ms=duration_ms,
+                user_id=user_id,
+                username=username,
+                client_ip=_client_ip(scope),
+                user_agent=req_headers.get("user-agent"),
+                request_headers=req_headers or None,
+                response_headers=res_headers or None,
+                request_body_raw=req_buf or None,
+                request_body_bytes=req_bytes,
+                request_content_type=req_ctype,
+                response_body_raw=res_buf or None,
+                response_body_bytes=res_bytes,
+                response_content_type=res_ctype,
+                error=error_text,
+            )
+        )

backend/models/cluster.py

@@ -1,6 +1,22 @@
-from pydantic import BaseModel
+from pydantic import BaseModel, field_validator
 from typing import Optional, List
 
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
+
+
+def _validated_acme_backend_url(value: Optional[str]) -> Optional[str]:
+    """Shared field validator body for `acme_backend_url`.
+
+    Pydantic turns the raised ValueError into a 422 with this message attached, so
+    the operator sees why the value was refused instead of discovering months later
+    that HTTP-01 never worked. Returns the normalised value — callers persist THIS,
+    not the raw input, so surrounding whitespace never reaches haproxy.cfg.
+    """
+    try:
+        return validate_acme_backend_url(value)
+    except AcmeBackendUrlError as exc:
+        raise ValueError(str(exc)) from None
+
 class HAProxyClusterCreate(BaseModel):
     name: str
     description: Optional[str] = None
@@ -10,6 +26,16 @@ class HAProxyClusterCreate(BaseModel):
     haproxy_bin_path: str = "/usr/sbin/haproxy"  # HAProxy binary path
     keepalived_config_path: str = "/etc/keepalived/keepalived.conf"  # HA/VIP: keepalived.conf path (Issue #27)
     pool_id: Optional[int] = None  # Which pool this cluster belongs to
+    # The create form submits both of these. Until they were declared here pydantic
+    # dropped them and the INSERT never carried them, so a cluster created with ACME
+    # switched on came back switched off with no error shown — the same silent-success
+    # failure this work exists to remove.
+    acme_enabled: Optional[bool] = None
+    acme_backend_url: Optional[str] = None
+
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
 
 class HAProxyClusterUpdate(BaseModel):
     name: Optional[str] = None
@@ -24,6 +50,10 @@ class HAProxyClusterUpdate(BaseModel):
     acme_enabled: Optional[bool] = None
     acme_backend_url: Optional[str] = None
 
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
+
 class HAProxyClusterResponse(BaseModel):
     id: int
     name: str

backend/models/csr.py

@@ -0,0 +1,251 @@
+"""
+Pydantic models for the CSR (Certificate Signing Request) feature (v1.9.0).
+
+A CSR row is the precursor of an ssl_certificates row: the backend generates
+the private key + CSR locally, the operator has the CSR signed by an external
+CA and then imports the signed certificate. The CSR `name` therefore obeys the
+exact same path-traversal contract as the SSL certificate name (Bulgu #21) —
+at import time it becomes /etc/ssl/haproxy/{name}.pem on every agent and is
+shell-processed by the agent script as root.
+
+The import model deliberately has NO private key field: the key never leaves
+the server. It is stored on the ssl_csrs row at generation time and paired
+with the signed certificate server-side.
+"""
+
+import re
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+KEY_ALGORITHMS = ('rsa-2048', 'rsa-4096', 'ecdsa-p256', 'ecdsa-p384')
+
+# RFC 1035 LDH hostname, lowercase, optional single leftmost wildcard label.
+# Single-label names are allowed (internal CAs routinely sign bare hostnames).
+_DNS_NAME_PATTERN = re.compile(
+    r'^(\*\.)?[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?'
+    r'(\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)*$'
+)
+
+# Reject control characters in free-text subject fields: they would be
+# persisted, echoed into the UI / issuer column, and printed into agent logs
+# via `openssl -subject` output.
+_CONTROL_CHARS_PATTERN = re.compile(r'[\x00-\x1f\x7f]')
+
+_MAX_SANS = 100
+_MAX_CERT_PEM_BYTES = 64 * 1024       # a leaf certificate is ~2 KB; 64 KB is generous
+_MAX_CHAIN_PEM_BYTES = 256 * 1024     # agents re-download all cert content every poll
+
+
+def _validate_dns_name(value: str, field_label: str) -> str:
+    v = (value or '').strip().lower()
+    if not v:
+        raise ValueError(f'{field_label} must not be empty')
+    if len(v) > 253:
+        raise ValueError(f'{field_label} must be 253 characters or fewer')
+    if not _DNS_NAME_PATTERN.match(v):
+        raise ValueError(
+            f'{field_label} {value!r} is not a valid DNS name — lowercase '
+            'letters, digits, hyphens and dots only; a wildcard is allowed '
+            'only as the leftmost label (e.g. *.example.com).'
+        )
+    return v
+
+
+def _validate_subject_text(value: Optional[str], field_label: str, max_len: int = 64) -> Optional[str]:
+    if value is None:
+        return None
+    v = value.strip()
+    if not v:
+        return None
+    if len(v) > max_len:
+        raise ValueError(f'{field_label} must be {max_len} characters or fewer')
+    if _CONTROL_CHARS_PATTERN.search(v):
+        raise ValueError(f'{field_label} must not contain control characters')
+    return v
+
+
+def _validate_csr_name(v: str) -> str:
+    """Mirror of SSLCertificateCreate.validate_name_no_path_traversal (Bulgu #21)
+    with one deliberate tightening: max length 100, matching the
+    ssl_certificates.name VARCHAR(100) column (the historical 200-char limit
+    overflows the column and 500s — not replicated here)."""
+    if v is None:
+        raise ValueError('CSR name is required')
+    stripped = v.strip()
+    if not stripped:
+        raise ValueError('CSR name must not be empty')
+    if stripped != v:
+        raise ValueError('CSR name must not contain leading/trailing whitespace')
+    if len(stripped) > 100:
+        raise ValueError('CSR name must be 100 characters or fewer')
+    if not re.match(r'^[A-Za-z0-9_.-]+$', stripped):
+        raise ValueError(
+            f'CSR name={v!r} contains forbidden characters — only letters, '
+            'digits, underscore, hyphen, and dot are allowed (the name becomes '
+            'a filename component under /etc/ssl/haproxy/ at import).'
+        )
+    if '..' in stripped:
+        raise ValueError(f'CSR name={v!r} must not contain ".." (path traversal)')
+    if stripped.startswith('.'):
+        raise ValueError(f'CSR name={v!r} must not start with "." (hidden filename)')
+    if stripped.startswith('-'):
+        raise ValueError(f'CSR name={v!r} must not start with "-" (CLI flag confusion)')
+    return stripped
+
+
+class SSLCSRCreate(BaseModel):
+    name: str                                   # becomes the certificate name at import
+    common_name: str
+    organization: Optional[str] = None          # O
+    organizational_unit: Optional[str] = None   # OU
+    locality: Optional[str] = None              # L
+    state: Optional[str] = None                 # ST
+    country: Optional[str] = None               # C — exactly 2 letters
+    email: Optional[str] = None                 # emailAddress
+    sans: List[str] = []                        # DNS names; CN is auto-added server-side
+    key_algorithm: str = 'rsa-2048'
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        return _validate_csr_name(v)
+
+    @field_validator('common_name')
+    @classmethod
+    def validate_common_name(cls, v):
+        v = _validate_dns_name(v, 'Common Name')
+        # RFC 5280 ub-common-name — many CAs reject CNs longer than 64 chars.
+        if len(v) > 64:
+            raise ValueError(
+                'Common Name must be 64 characters or fewer (RFC 5280 upper '
+                'bound) — put longer names in the SAN list instead.'
+            )
+        return v
+
+    @field_validator('sans')
+    @classmethod
+    def validate_sans(cls, v):
+        if not v:
+            return []
+        if len(v) > _MAX_SANS:
+            raise ValueError(f'At most {_MAX_SANS} SAN entries are allowed')
+        seen = set()
+        result = []
+        for entry in v:
+            normalised = _validate_dns_name(entry, 'SAN entry')
+            if normalised not in seen:
+                seen.add(normalised)
+                result.append(normalised)
+        return result
+
+    @field_validator('organization')
+    @classmethod
+    def validate_organization(cls, v):
+        return _validate_subject_text(v, 'Organization (O)')
+
+    @field_validator('organizational_unit')
+    @classmethod
+    def validate_organizational_unit(cls, v):
+        return _validate_subject_text(v, 'Organizational Unit (OU)')
+
+    @field_validator('locality')
+    @classmethod
+    def validate_locality(cls, v):
+        return _validate_subject_text(v, 'Locality (L)')
+
+    @field_validator('state')
+    @classmethod
+    def validate_state(cls, v):
+        return _validate_subject_text(v, 'State/Province (ST)')
+
+    @field_validator('country')
+    @classmethod
+    def validate_country(cls, v):
+        # cryptography raises a bare ValueError for a non-2-char COUNTRY_NAME;
+        # pre-validate so the operator gets a friendly 422 instead of a 500.
+        if v is None:
+            return None
+        v = v.strip()
+        if not v:
+            return None
+        if not re.match(r'^[A-Za-z]{2}$', v):
+            raise ValueError('Country (C) must be exactly 2 letters (ISO 3166-1 alpha-2, e.g. TR, US)')
+        return v.upper()
+
+    @field_validator('email')
+    @classmethod
+    def validate_email(cls, v):
+        v = _validate_subject_text(v, 'Email', max_len=254)
+        if v is not None and ('@' not in v or v.startswith('@') or v.endswith('@')):
+            raise ValueError('Email must be a valid address (missing or misplaced "@")')
+        return v
+
+    @field_validator('key_algorithm')
+    @classmethod
+    def validate_key_algorithm(cls, v):
+        if v not in KEY_ALGORITHMS:
+            raise ValueError(
+                f'key_algorithm must be one of: {", ".join(KEY_ALGORITHMS)}'
+            )
+        return v
+
+
+class SSLCSRImport(BaseModel):
+    """Import the CA-signed certificate for a pending CSR. The private key is
+    NOT part of the request — it is already stored on the CSR row."""
+    certificate_content: str                    # PEM
+    chain_content: Optional[str] = None         # PEM, optional
+    usage_type: str = 'frontend'                # "frontend" or "server"
+    is_global: bool = False
+    cluster_ids: Optional[List[int]] = None
+    # Escape hatch for name collisions that appeared AFTER the CSR was
+    # created: overrides the CSR's reserved name for the certificate row.
+    name: Optional[str] = None
+
+    @field_validator('certificate_content')
+    @classmethod
+    def validate_certificate(cls, v):
+        if not v or not v.strip():
+            raise ValueError('Certificate content is required')
+        v = v.strip()
+        if len(v.encode('utf-8', errors='ignore')) > _MAX_CERT_PEM_BYTES:
+            raise ValueError('Certificate content exceeds the 64 KB limit')
+        if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+            raise ValueError('Certificate must be in PEM format')
+        return v
+
+    @field_validator('chain_content')
+    @classmethod
+    def validate_chain(cls, v):
+        if v and v.strip():
+            v = v.strip()
+            if len(v.encode('utf-8', errors='ignore')) > _MAX_CHAIN_PEM_BYTES:
+                raise ValueError('Certificate chain exceeds the 256 KB limit')
+            if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+                raise ValueError('Certificate chain must be in PEM format')
+            return v
+        return None
+
+    @field_validator('usage_type')
+    @classmethod
+    def validate_usage_type(cls, v):
+        if v not in ['frontend', 'server']:
+            raise ValueError('usage_type must be either "frontend" or "server"')
+        return v
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        if v is None or not str(v).strip():
+            return None
+        return _validate_csr_name(v)
+
+    @model_validator(mode='after')
+    def validate_cluster_selection(self):
+        if not self.is_global and not self.cluster_ids:
+            raise ValueError(
+                'cluster_ids is required when is_global is false — pick at '
+                'least one cluster or import the certificate as global.'
+            )
+        return self

backend/models/frontend.py

@@ -85,6 +85,11 @@ class FrontendConfig(BaseModel):
     response_headers: Optional[str] = None
     options: Optional[str] = None
     tcp_request_rules: Optional[str] = None
+    # Issue #38: SPOE filter directives (Coraza WAF etc.) + frontend log-format.
+    # Passthrough TEXT (no validator) — SPOE `filter ... config <path>` legitimately
+    # references an operator-managed file, so the ACL `-f` guard must NOT apply here.
+    log_format: Optional[str] = None
+    filters: Optional[str] = None
     timeout_client: Optional[int] = None
     timeout_http_request: Optional[int] = None
     rate_limit: Optional[int] = None
@@ -451,26 +456,17 @@ class FrontendConfig(BaseModel):
             if any(dangerous in rule.lower() for dangerous in ['$(', '`']):
                 raise ValueError(f'ACL rule contains potentially dangerous content: "{rule}"')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) — reject
-            # the HAProxy `-f <file>` pattern-file flag here too so the
-            # manual Frontend API mirrors the wizard's parity rule.
-            # HAProxy OpenManager does not provision pattern files
-            # onto the HAProxy node filesystem, so any `-f /path/...`
-            # reference will fail HAProxy's `-c` parse at apply time
-            # with "failed to open pattern file". Reject up-front so
-            # operators get the same actionable error from both the
-            # manual page and the wizard.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'ACL rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not "
-                    "provision pattern files onto the HAProxy node "
-                    "filesystem, so the reference would fail at "
-                    "reload time. Use inline values instead "
-                    "(e.g. `src 10.0.0.0/24` rather than "
-                    "`src -f /etc/haproxy/admins.lst`)."
-                )
+            # Issue #38 follow-up — the `-f <file>` pattern-file flag
+            # is ACCEPTED here (the Bulgu #12 hard reject was removed).
+            # Pattern files are operator-managed host files, exactly
+            # like the SPOE `filter ... config <path>` reference this
+            # release started preserving: bulk import always accepted
+            # `-f`, the free-form fields (request_headers,
+            # tcp_request_rules) always accepted it, and the agent
+            # runs `haproxy -c` before every reload so a missing file
+            # fails safely (previous config keeps running). The route
+            # handlers surface a non-blocking warning listing the
+            # referenced pattern files instead.
 
             validated_rules.append(rule)
 
@@ -510,20 +506,12 @@ class FrontendConfig(BaseModel):
             if not any(rule.startswith(redirect_type) for redirect_type in valid_redirects):
                 raise ValueError(f'Invalid redirect rule: "{rule}". Must start with: location, prefix, or scheme.')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) —
-            # mirror the wizard's `-f <file>` guard here. The
-            # `X !X` contradiction check used to live alongside
-            # this guard, but Bulgu #62 (round-22 audit) moved
-            # it into the route handler so updates can grandfather
-            # legacy rules created before the contradiction guard
-            # landed. See `routers/frontend.py::_collect_routing_rule_contradictions`.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'Redirect rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not provision "
-                    "pattern files onto the HAProxy node filesystem."
-                )
+            # Issue #38 follow-up — `-f <file>` pattern-file references
+            # are ACCEPTED (Bulgu #12 hard reject removed; see
+            # validate_acl_rules for the full rationale). The `X !X`
+            # contradiction check lives in the route handler
+            # (`routers/frontend.py::_collect_routing_rule_contradictions`,
+            # Bulgu #62) and is unchanged.
 
             validated_rules.append(rule)
 
@@ -531,9 +519,12 @@ class FrontendConfig(BaseModel):
 
     @validator('use_backend_rules')
     def validate_use_backend_rules_syntax(cls, v):
-        """Phase K Phase D follow-up (Bulgu #12 round 3) — manual
-        Frontend API parity guard: reject `-f <file>` references
-        and dangerous shell patterns.
+        """Manual Frontend API guard for dangerous shell patterns.
+
+        Issue #38 follow-up — the Bulgu #12 `-f <file>` hard reject
+        was removed (see validate_acl_rules for the rationale);
+        pattern-file references are operator-managed host files and
+        are surfaced as non-blocking warnings by the route handlers.
 
         Bulgu #62 (round-22 audit) — the `X !X` contradiction check
         previously lived here but moved into the route handler so
@@ -563,13 +554,5 @@ class FrontendConfig(BaseModel):
                     f'use_backend rule contains potentially dangerous '
                     f'content: "{rule}"'
                 )
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'use_backend rule "{rule}" uses the HAProxy '
-                    "`-f <file>` pattern-file flag, which is not "
-                    "supported in HAProxy OpenManager: the product "
-                    "does not provision pattern files onto the HAProxy "
-                    "node filesystem."
-                )
             validated_rules.append(rule)
         return validated_rules

backend/models/site_wizard.py

@@ -179,35 +179,17 @@ _MAX_RULE_STRING_LEN = 4096
 # attempts.
 _DANGEROUS_RULE_PATTERNS = ("$(", "`")
 
-# Phase K Phase D follow-up (Bulgu #12 round 3) — the HAProxy `-f
-# <file>` ACL/condition flag instructs HAProxy to load match patterns
-# from a server-side file at parse time. HAProxy OpenManager is a
-# fully-managed product: we do NOT provision pattern files onto the
-# HAProxy node's filesystem, and operators have no UI to upload one.
-# A `-f /some/path` reference therefore ALWAYS resolves to
-# "file not found" when HAProxy's real `-c` parse runs at apply
-# time, producing exactly the operator-reported failure mode:
-#   [ALERT] parsing ACL 'acl1' : failed to open pattern file </path>.
-#   [ALERT] parsing switching rule : no such ACL : 'acl1'.
-#
-# Surface this BEFORE persist by rejecting `-f` in any rule string
-# that comes through the wizard / manual frontend API. Reject ALL
-# variants (` -f `, leading `-f `, trailing `... -f`) defensively so
-# operators cannot slip the flag through with creative spacing.
-# The check is anchored to ACL/condition rule strings only; raw
-# HAProxy snippet fields (tcp_request_rules, request_headers, ...)
-# are NOT touched because those are inherently free-form and
-# advanced operators may legitimately reference pre-provisioned
-# pattern files there.
-_ACL_FILE_FLAG_PATTERN = re.compile(r"(^|\s)-f(\s|$)")
-_ACL_FILE_FLAG_MESSAGE = (
-    "pattern-file references with '-f <file>' are not supported in ACL / "
-    "use_backend / redirect rules: HAProxy OpenManager does not provision "
-    "pattern files onto the HAProxy node's filesystem, so the reference "
-    "would always fail at HAProxy reload time. Use inline values "
-    "instead (e.g. `acl is_admin src 10.0.0.0/24` rather than "
-    "`acl is_admin src -f /etc/haproxy/admins.lst`)."
-)
+# Issue #38 follow-up — the HAProxy `-f <file>` ACL/condition flag
+# loads match patterns from a file on the HAProxy host. The Bulgu #12
+# hard reject (`_ACL_FILE_FLAG_PATTERN`/`_ACL_FILE_FLAG_MESSAGE`) was
+# removed: pattern files are operator-managed host files (exactly like
+# the SPOE `filter ... config <path>` reference preserved since
+# v1.8.8), bulk import and the free-form fields (tcp_request_rules,
+# request_headers) always accepted them, and the agent runs
+# `haproxy -c` before every reload so a missing file fails safely
+# (the previous config keeps running). The manual frontend route
+# handlers emit a non-blocking warning listing referenced pattern
+# files (`routers/frontend.py::_pattern_file_warnings`).
 
 # Phase K Phase D follow-up (Bulgu #13) — detect a routing /
 # redirect rule whose condition references the SAME ACL in both
@@ -305,13 +287,10 @@ def _validate_haproxy_directive_string(
                 f"{field_label} entry contains potentially dangerous content: "
                 f"{pattern!r}"
             )
-    # Phase K Phase D follow-up (Bulgu #12 round 3) — reject the
-    # HAProxy `-f <file>` pattern-file flag because OpenManager does
-    # not manage the HAProxy node filesystem. See the module-level
-    # `_ACL_FILE_FLAG_PATTERN` docstring for the full operator-
-    # reported failure mode this guards against.
-    if _ACL_FILE_FLAG_PATTERN.search(stripped):
-        raise ValueError(f"{field_label}: {_ACL_FILE_FLAG_MESSAGE}")
+    # Issue #38 follow-up — `-f <file>` pattern-file references are
+    # ACCEPTED (Bulgu #12 hard reject removed; see the module-level
+    # `_ACL_FILE_FLAG_PATTERN` comment). The route handlers surface
+    # a non-blocking pattern-file warning instead.
     # Phase K Phase D follow-up (Bulgu #13) — for routing /
     # redirect rules (not ACL definitions themselves), reject a
     # condition that contains the same ACL in both positive and
@@ -1083,21 +1062,12 @@ class FrontendStep(BaseModel):
         normalised: List[Union[str, dict]] = []
         for el in v:
             if isinstance(el, dict):
-                # Phase K Phase D follow-up (Bulgu #12 round 3
-                # extension) — dict-shaped redirect rules emit their
-                # `condition` / `target` fields VERBATIM into the
-                # rendered HAProxy directive. A dict with
-                # `condition: "if { src -f /etc/haproxy/x.lst }"`
-                # would slip past the string-only validator above
-                # and trigger the same operator-reported "failed to
-                # open pattern file" rejection at apply time. Reject
-                # `-f` in any string-shaped value the dict carries.
-                for field_name in ("condition", "target", "type"):
-                    val = el.get(field_name)
-                    if isinstance(val, str) and _ACL_FILE_FLAG_PATTERN.search(val):
-                        raise ValueError(
-                            f"redirect_rules.{field_name}: {_ACL_FILE_FLAG_MESSAGE}"
-                        )
+                # Issue #38 follow-up — dict-shaped redirect rules may
+                # carry `-f <file>` pattern-file references in their
+                # `condition`/`target` values; these are ACCEPTED now
+                # (Bulgu #12 hard reject removed — operator-managed
+                # host files, fail-safe apply; see module-level
+                # `_ACL_FILE_FLAG_PATTERN` comment).
                 # Bulgu #13 extension — same contradiction guard
                 # for dict-shaped redirect conditions.
                 cond_val = el.get("condition")

backend/routers/agent.py

@@ -9,8 +9,15 @@ import os
 import json
 import ipaddress
 import hashlib
+import re
 # Pipeline trigger - force backend redeploy v2
 
+# v1.10.4 — a discovered keepalived.conf is stored and served to the UI, so the VRRP password is
+# masked out of the stored copy (the real value lives Fernet-encrypted in its own column). Mask
+# the WHOLE remainder of the line, mirroring vip.py's version-diff masking, so a password
+# containing whitespace cannot partially leak.
+_AUTH_PASS_MASK_RE = re.compile(r"(auth_pass\s+).*")
+
 from models import AgentCreate
 from models.agent import AgentToggle, AgentHeartbeat, AgentScriptRequest, AgentUpgradeRequest
 from database.connection import get_database_connection, close_database_connection
@@ -26,9 +33,43 @@ logger = logging.getLogger(__name__)
 # Global version storage (acts as in-memory database)
 AGENT_VERSIONS = {
     "macos": "2.1.0",  # Updated via endpoint
-    "linux": "2.0.0"
+    "linux": "2.1.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -217,7 +258,7 @@ def calculate_agent_health(status, last_seen):
         return "offline"
 
 @router.get("", summary="Get All Agents", response_description="List of all agents")
-async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None)):
+async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All Agents
     
@@ -269,9 +310,22 @@ async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(
     - **haproxy_status**: Status of HAProxy service on agent's server
     - **last_seen**: Last heartbeat timestamp
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): the agent inventory (names, hostnames, IPs,
+    # pools, OS) is operator data and was previously served unauthenticated — it is
+    # also the read-back channel used in the RCE exfil PoC. Require EITHER a valid
+    # operator JWT OR a valid agent X-API-Key: deployed agents poll this endpoint
+    # (with their key, not a JWT) to read their own applied_config_version and avoid
+    # re-applying config on restart, so a JWT-only gate would break them. Checked
+    # before the try so the 401 is not swallowed by the generic handler.
+    if authorization:
+        current_user = await get_current_user_from_token(authorization)  # raises 401 on invalid JWT
+    else:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         try:
             if pool_id:
                 agents = await conn.fetch("""
@@ -729,6 +783,14 @@ async def generate_uninstall_script(platform: str, authorization: str = Header(N
     sudo ./uninstall-agent.sh
     ```
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): require authentication (operator JWT or agent
+    # key), consistent with generate-install-script. The uninstall script itself is
+    # generic (no secrets/topology), but an agent-management endpoint should not be
+    # anonymously reachable. Checked before the try so the 401 is not swallowed.
+    if authorization:
+        await get_current_user_from_token(authorization)
+    elif not await validate_agent_api_key(x_api_key):
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         # Normalize platform to a canonical key (always 'linux' or 'macos').
         # macOS agents register with platform 'darwin' (from `uname -s`), so the
@@ -924,14 +986,24 @@ def _extract_agent_ip(heartbeat_data: AgentHeartbeat) -> Optional[str]:
     return None
 
 @router.post("/{agent_id}/heartbeat")
-async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat):
+async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat, x_api_key: Optional[str] = Header(None)):
     """Receive agent heartbeat and update status."""
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). This legacy by-ID
+    # heartbeat previously had NO auth, allowing unauthenticated state spoofing of
+    # any agent row. Deployed agents use the by-name heartbeat; a valid global
+    # agent token is now required here too. NOTE: raised BEFORE the try below so
+    # the 401 is not swallowed by the generic `except Exception` handler.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on by-id heartbeat for agent ID {agent_id}")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         await conn.execute("""
-            UPDATE agents 
-            SET status = 'online', 
+            UPDATE agents
+            SET status = 'online',
                 last_seen = CURRENT_TIMESTAMP,
                 hostname = COALESCE($2, hostname),
                 haproxy_status = COALESCE($3, haproxy_status),
@@ -1054,6 +1126,8 @@ async def agent_config_applied_notification(agent_name: str, notification_data:
         await close_database_connection(conn)
         return {"status": "ok", "message": "Config applied notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config applied notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1149,6 +1223,8 @@ async def agent_config_validation_failed(agent_name: str, notification_data: dic
         
         return {"status": "ok", "message": "Validation error notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process validation error notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1438,6 +1514,8 @@ async def agent_config_sync(agent_name: str, sync_data: dict, x_api_key: Optiona
         logger.info(f"CONFIG SYNC: Agent '{agent_name}' synced {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers with database")
         return {"status": "ok", "message": f"Config synced - {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers processed"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config sync from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1455,47 +1533,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1512,21 +1576,25 @@ async def agent_heartbeat_by_name(
         logger.error(f"Unexpected error processing heartbeat: {e}")
         raise HTTPException(status_code=500, detail="Internal server error")
     
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). A valid global agent
+    # token is required to heartbeat OR auto-register. Deployed agents always send
+    # X-API-Key; an absent/invalid key is an unauthenticated caller. This is done
+    # OUTSIDE the processing try below (whose generic `except Exception` would
+    # otherwise convert the 401 into a 500), and before opening a DB connection
+    # (validate_agent_api_key(None) needs no DB). Closes keyless heartbeat spoofing
+    # and keyless rogue-agent auto-registration (the `elif not agent` keyless path
+    # below is now unreachable, since agent_auth is guaranteed truthy past here).
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on heartbeat for agent '{heartbeat_data.name}'")
+        raise HTTPException(status_code=401, detail="Authentication required")
+
     # Continue with normal heartbeat processing
     try:
-        # Validate agent API key for security
-        from auth_middleware import validate_agent_api_key
-        agent_auth = await validate_agent_api_key(x_api_key)
-        
         conn = await get_database_connection()
         agent_name = heartbeat_data.name
 
-        # If API key provided, validate it exists but allow placeholder agent updates
-        if x_api_key and not agent_auth:
-            await close_database_connection(conn)
-            logger.warning(f"Invalid API key provided by agent '{agent_name}'")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-
         agent = await conn.fetchrow("SELECT id, pool_id, api_key FROM agents WHERE name = $1", agent_name)
         
         # If agent exists and is using a different API key, update the token association
@@ -1675,9 +1743,13 @@ async def agent_heartbeat_by_name(
             """, x_api_key)
 
         # Check if agent was in upgrading status and version has changed
-        current_agent_status = await conn.fetchval("SELECT status FROM agents WHERE id = $1", agent_id)
-        current_agent_version = await conn.fetchval("SELECT version FROM agents WHERE id = $1", agent_id)
-        current_upgrade_status = await conn.fetchval("SELECT upgrade_status FROM agents WHERE id = $1", agent_id)
+        # (v1.8.6: one round-trip instead of three; row is None exactly when the
+        # per-column fetchvals would each have returned None)
+        current_agent_row = await conn.fetchrow(
+            "SELECT status, version, upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_status = current_agent_row['status'] if current_agent_row else None
+        current_agent_version = current_agent_row['version'] if current_agent_row else None
+        current_upgrade_status = current_agent_row['upgrade_status'] if current_agent_row else None
         
         # Determine new status - preserve upgrading status unless version actually changed
         new_status = current_agent_status or 'online'
@@ -1931,9 +2003,19 @@ async def agent_heartbeat_by_name(
 @router.get("/{agent_name}/config")
 async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get HAProxy configuration for specific agent"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked BEFORE any
+    # DB work and before the existence check, so an unauthenticated caller learns
+    # neither the full haproxy.cfg nor whether the agent exists. Raised before the
+    # try so it is not swallowed by the generic handler; validate_agent_api_key(None)
+    # returns None without touching the DB.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' config fetch")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent info first to check pool
         # CRITICAL: Include cluster's haproxy_bin_path, haproxy_config_path, stats_socket_path
         # These are needed for dynamic validation - cluster admin can change paths without reinstalling agent
@@ -1945,30 +2027,19 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' config fetch")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching config using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Config fetch authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Config fetch authorized for agent '{agent_name}'")
+
         if not agent_info['enabled']:
             await close_database_connection(conn)
             return {
@@ -2059,9 +2130,18 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
 @router.get("/{agent_name}/ssl-certificates")
 async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = None, x_api_key: Optional[str] = Header(None)):
     """Get SSL certificates for specific agent's cluster"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). This response
+    # returns SSL private_key_content, so authentication is checked BEFORE any DB
+    # work and before the existence check. Raised before the try so the 401 is not
+    # swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' SSL certificates")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent and cluster info first
         agent_info = await conn.fetchrow("""
             SELECT a.id, a.name, a.pool_id, hc.id as cluster_id, hc.name as cluster_name,
@@ -2070,29 +2150,18 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' SSL certificates")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
         
         if not agent_info['enabled']:
             await close_database_connection(conn)
@@ -2256,21 +2325,46 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
         # to write/own-marker-check even on not_configured/teardown.
         agent = await conn.fetchrow("""
             SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
-                   hc.keepalived_config_path
+                   hc.keepalived_config_path,
+                   -- v1.11.1: does the server already hold a discovery for this node? The agent
+                   -- caches the hash of its last discovery report next to the config and skips
+                   -- re-posting while it matches. That cache used to be written even when the
+                   -- POST was REJECTED, so a node could be hidden from the adoption panel for
+                   -- good: the file never changes, so the agent never speaks again. Telling it
+                   -- what we actually hold lets it recover on its own, with no extra request and
+                   -- no one having to touch the node.
+                   EXISTS (SELECT 1 FROM vip_discoveries vd WHERE vd.agent_id = a.id)
+                       AS discovery_known
             FROM agents a
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
+            -- A pool may hold more than one cluster, and the join then multiplies this row. With
+            -- no ordering the fetch took an arbitrary one, so the keepalived.conf PATH handed to
+            -- the agent was non-deterministic whenever two clusters in a pool disagreed on it:
+            -- the agent would look at the wrong file, find nothing there, and the node would
+            -- never appear for adoption — intermittently, which is the worst way to fail.
+            --
+            -- A CUSTOMISED path wins over the shipped default, then the lowest cluster id. The
+            -- column defaults to '/etc/keepalived/keepalived.conf' rather than NULL, so ordering
+            -- by id alone could have picked a default-valued row over one the operator had
+            -- deliberately set — turning "undefined" into "reliably wrong" for that install.
+            -- When every cluster in the pool carries the default the string is identical, so the
+            -- ordering cannot change what any working deployment already receives.
+            ORDER BY (hc.keepalived_config_path IS NULL
+                      OR hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'),
+                     hc.id
+            LIMIT 1
         """, agent_name)
         if not agent:
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
         config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
         if not agent['enabled']:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
 
         row = await conn.fetchrow("""
             SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
                    v.purge_on_teardown,
-                   m.applied_config_content, m.applied_config_hash
+                   m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
             FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
             WHERE m.agent_id = $1
             -- Active VIP first (an agent has at most one). With NO active VIP, pick the most
@@ -2281,22 +2375,22 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
         """, agent['id'])
 
         if not row:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
         if not row['is_active']:
             # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
             # the agent still only purges on nodes where IT installed keepalived (install marker).
             return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
-                    "config_path": config_path, "keepalived": None,
+                    "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None,
                     "purge": bool(row['purge_on_teardown'])}
         if not row['applied_config_content']:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
 
         from services.keepalived_config import build_haproxy_check_script
         check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
         return {
             "agent_name": agent_name,
             "status": "available",
-            "config_path": config_path,
+            "config_path": config_path, "discovery_known": bool(agent["discovery_known"]),
             "keepalived": {
                 "desired_state": "enabled",
                 "install_if_missing": True,
@@ -2305,6 +2399,14 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
                 "config_content": row['applied_config_content'],
                 "config_hash": row['applied_config_hash'],
                 "check_script": check_script,
+                # v1.10.4 adoption handoff. The agent refuses to overwrite a keepalived.conf
+                # without our ownership marker — the guard that protects a hand-maintained
+                # setup. Adoption does not weaken it: it authorises exactly ONE takeover, of
+                # exactly the file we analysed, by pinning its hash. If the file changed since
+                # adoption the hashes differ and the agent keeps refusing, so an edit made
+                # between adoption and Apply can never be silently overwritten.
+                "allow_takeover": bool(row['takeover_expected_hash']),
+                "takeover_expected_hash": row['takeover_expected_hash'],
             },
         }
     except HTTPException:
@@ -2346,19 +2448,40 @@ async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key:
         state = (status_data.get("state") or "").strip()[:24]
         config_hash = (status_data.get("config_hash") or "")[:64]
         message = status_data.get("message")
+        # v1.10.9 — retire the adoption takeover authorisation once the node CONFIRMS it is
+        # running our rendered config. `takeover_expected_hash` is the permission to overwrite a
+        # keepalived.conf that lacks our ownership marker; it was written at adoption and never
+        # cleared, so it stayed valid indefinitely and "one-shot" was only true in the sense of
+        # "for exactly that file content". Clearing it the moment the member acks OUR hash makes
+        # the claim real: if the file is replaced by hand afterwards the agent refuses and reports
+        # "externally managed", which is the visible behaviour an operator should get.
+        #
+        # Gated on the acked hash MATCHING applied_config_hash, so a partial or failed deploy
+        # never drops the authorisation and leaves the VIP unable to converge.
+        # The hash is passed TWICE on purpose. Reusing one placeholder for both the assignment
+        # (`last_deploy_hash=$n`, a VARCHAR column) and the comparison inside the CASE made
+        # PostgreSQL deduce two different types for it and asyncpg refused the whole statement
+        # with AmbiguousParameterError ("text versus character varying"). Because the failure is
+        # in the UPDATE itself, not in one column, EVERY status ack was lost and every VIP sat at
+        # SYNCING forever — including teardown acks. A separate placeholder is only ever compared
+        # against the column, so its type is unambiguous.
+        _retire_takeover = ("takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL "
+                            "AND applied_config_hash = {p} THEN NULL ELSE takeover_expected_hash END")
         if vip_id is None:
             # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
-            await conn.execute("""
+            await conn.execute(f"""
                 UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
-                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$5")}
                 WHERE agent_id=$1
-            """, agent['id'], state, message, config_hash)
+            """, agent['id'], state, message, config_hash, config_hash)
         else:
-            await conn.execute("""
+            await conn.execute(f"""
                 UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
-                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$6")}
                 WHERE agent_id=$1 AND vip_id=$2
-            """, agent['id'], int(vip_id), state, message, config_hash)
+            """, agent['id'], int(vip_id), state, message, config_hash, config_hash)
         return {"status": "ok"}
     except HTTPException:
         raise
@@ -2369,6 +2492,100 @@ async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key:
         if conn:
             await close_database_connection(conn)
 
+
+@router.post("/{agent_name}/keepalived-discovery")
+async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key: Optional[str] = Header(None)):
+    """v1.10.4 — the agent reports a keepalived.conf it found on the node but does NOT own.
+
+    This is what makes adopting a hand-maintained VIP possible: the heartbeat only carries the
+    VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+    fields, so the file itself has to be read. Read-only on the agent side — reporting never
+    changes anything on the node.
+
+    Auth mirrors /keepalived-status: a MISSING key is rejected outright, and because the token
+    is a shared install token a name mismatch is an advisory audit log rather than a 403.
+
+    SECRETS: the reported content may contain the VRRP `auth_pass`. It is split immediately —
+    the password is Fernet-encrypted into its own column and the stored copy of the file has it
+    masked, so nothing readable through the API or a DB dump carries it in cleartext. The
+    parse result is never logged.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived discovery using API key "
+                        f"from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        config_path = (payload.get("config_path") or "/etc/keepalived/keepalived.conf")[:500]
+        exists = bool(payload.get("exists"))
+        if not exists:
+            # The file is gone (keepalived removed, or we adopted and now own it) — drop the row
+            # so the UI stops offering a stale candidate.
+            await conn.execute("DELETE FROM vip_discoveries WHERE agent_id = $1", agent['id'])
+            return {"status": "cleared"}
+
+        content = payload.get("config_content") or ""
+        if len(content) > 256_000:
+            raise HTTPException(status_code=413, detail="keepalived.conf too large to analyse")
+        is_managed = bool(payload.get("is_managed"))
+        config_hash = hashlib.md5(content.encode("utf-8", "replace")).hexdigest()
+
+        from services.keepalived_parser import analyse_keepalived_conf, KeepalivedParseError
+        from services.keepalived_config import encrypt_vrrp_secret
+
+        parse_error = None
+        analysis = None
+        auth_enc = None
+        try:
+            analysis = analyse_keepalived_conf(content)
+            # Split the secret out of everything we persist or serve.
+            for cand in analysis.get("candidates", []):
+                secret = (cand.get("vip") or {}).pop("auth_pass", None)
+                cand["vip"]["has_auth_pass"] = bool(secret)
+                if secret and auth_enc is None:
+                    auth_enc = encrypt_vrrp_secret(secret)
+        except KeepalivedParseError as exc:
+            parse_error = str(exc)[:500]
+        except Exception as exc:  # noqa: BLE001 — a malformed file must not 500 the agent loop
+            parse_error = f"could not analyse the config ({type(exc).__name__})"
+
+        masked = _AUTH_PASS_MASK_RE.sub(r"\1********", content)
+        await conn.execute("""
+            INSERT INTO vip_discoveries
+                (agent_id, config_path, config_hash, is_managed, raw_config_masked,
+                 auth_pass_encrypted, analysis, parse_error, reported_at)
+            VALUES ($1,$2,$3,$4,$5,$6,$7,$8,CURRENT_TIMESTAMP)
+            ON CONFLICT (agent_id) DO UPDATE SET
+                config_path = EXCLUDED.config_path,
+                config_hash = EXCLUDED.config_hash,
+                is_managed = EXCLUDED.is_managed,
+                raw_config_masked = EXCLUDED.raw_config_masked,
+                auth_pass_encrypted = EXCLUDED.auth_pass_encrypted,
+                analysis = EXCLUDED.analysis,
+                parse_error = EXCLUDED.parse_error,
+                reported_at = CURRENT_TIMESTAMP
+        """, agent['id'], config_path, config_hash, is_managed, masked, auth_enc,
+            json.dumps(analysis) if analysis is not None else None, parse_error)
+        return {"status": "recorded", "config_hash": config_hash}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-discovery failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-discovery failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
 @router.get("/script-version")
 async def get_latest_script_version(platform: str = "macos"):
     """Get the latest available agent script version for specified platform"""
@@ -2416,16 +2633,24 @@ async def get_latest_script_version(platform: str = "macos"):
 @router.get("/{agent_name}/upgrade-status")
 async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get agent upgrade status - used by agents to check if they should upgrade"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked before any
+    # DB work; deployed agents always send X-API-Key. Raised before the try so the
+    # 401 is not swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' upgrade status")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Check if agent has upgrade pending (include platform and pool for validation)
         agent = await conn.fetchrow("""
             SELECT status, version as current_version, platform, pool_id
-            FROM agents 
+            FROM agents
             WHERE name = $1
         """, agent_name)
-        
+
         if not agent:
             await close_database_connection(conn)
             return {
@@ -2433,26 +2658,15 @@ async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = H
                 "target_version": "",
                 "message": "Agent not found"
             }
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' upgrade status")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
+
         await close_database_connection(conn)
         
         # Agent should upgrade if status is 'upgrading'
@@ -2886,7 +3100,17 @@ async def get_agent_script_template(platform: str, authorization: str = Header(N
     """Get the latest script template for specified platform from database"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): the raw install/upgrade script is a
+        # version-management surface. Gate reads with agents.version too, matching
+        # the write path above (operator/security_admin/super_admin retain access).
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         conn = await get_database_connection()
         
         # Get latest script template for platform
@@ -2937,7 +3161,19 @@ async def save_agent_script_template(platform: str, template_data: dict, authori
     """Save updated script template to database using shared helper function"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): agent script templates become the
+        # install/self-upgrade script executed as root on HAProxy nodes. A poisoned
+        # template is RCE. Authentication alone is NOT enough — require the same
+        # agents.version permission as POST /versions; otherwise any JWT holder
+        # (including viewer) could overwrite the active script.
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         script_content = template_data.get('script_content', '')
         version = template_data.get('version', '')
         

backend/routers/cluster.py

@@ -111,6 +111,14 @@ router = APIRouter(prefix="/api/clusters", tags=["clusters"])
 logger = logging.getLogger(__name__)
 
 
+class _AcmeNoConfigChange(Exception):
+    """Internal signal: an ACME edit renders the same config, so mint nothing.
+
+    Control flow, not an error — it unwinds out of the version-minting block without
+    tripping the generic `except Exception` handler that would log it as a failure.
+    """
+
+
 class _ConcurrentlyDrained(Exception):
     """Sentinel raised inside ``apply_pending_changes`` when the
     advisory-lock-protected re-fetch shows that another caller
@@ -294,12 +302,14 @@ async def create_cluster(cluster: HAProxyClusterCreate, authorization: str = Hea
         cluster_id = await conn.fetchval("""
             INSERT INTO haproxy_clusters (name, description, connection_type, is_active,
                                         stats_socket_path, haproxy_config_path, haproxy_bin_path,
-                                        keepalived_config_path, pool_id)
-            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8)
+                                        keepalived_config_path, pool_id,
+                                        acme_enabled, acme_backend_url)
+            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8, COALESCE($9, FALSE), $10)
             RETURNING id
         """, cluster.name, cluster.description, cluster.connection_type,
             cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path,
-            cluster.keepalived_config_path, cluster.pool_id)
+            cluster.keepalived_config_path, cluster.pool_id,
+            cluster.acme_enabled, cluster.acme_backend_url)
         
         await close_database_connection(conn)
         
@@ -389,7 +399,8 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
         # Check if cluster exists and get current values
         existing_cluster = await conn.fetchrow("""
             SELECT name, description, connection_type, is_active, stats_socket_path,
-                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled
+                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled,
+                   acme_backend_url
             FROM haproxy_clusters WHERE id = $1
         """, cluster_id)
         if not existing_cluster:
@@ -453,7 +464,13 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"acme_enabled = ${param_counter}")
             update_values.append(cluster.acme_enabled)
             param_counter += 1
-        if cluster.acme_backend_url is not None:
+        # Keyed on "was the field submitted?", not "is it non-None". With a plain
+        # `is not None` test there is no way to CLEAR the value: the validator maps an
+        # empty box to None, which is indistinguishable from "not supplied", so once an
+        # operator set a per-cluster URL they could never revert to the global setting —
+        # the field would accept the edit and silently keep the old value.
+        _acme_url_submitted = 'acme_backend_url' in cluster.model_fields_set
+        if _acme_url_submitted:
             update_fields.append(f"acme_backend_url = ${param_counter}")
             update_values.append(cluster.acme_backend_url)
             param_counter += 1
@@ -463,14 +480,75 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_query = f"UPDATE haproxy_clusters SET {', '.join(update_fields)} WHERE id = $1"
             await conn.execute(update_query, *update_values)
 
-        # If acme_enabled actually changed, create a PENDING config version with entity snapshot
-        if cluster.acme_enabled is not None and cluster.acme_enabled != existing_cluster.get('acme_enabled', False):
+        # Create a PENDING config version when an ACME edit would change what the
+        # HAProxy nodes actually run.
+        #
+        # This used to trigger only on `acme_enabled` flipping. `acme_backend_url` is
+        # written to the DB a few lines above but minted nothing, so correcting a wrong
+        # challenge backend from the panel was a silent no-op: the value changed, no
+        # pending version existed, Apply answered "No pending changes to apply", and the
+        # nodes kept the old address indefinitely. That made the one field an operator
+        # needs to fix HTTP-01 impossible to actually apply.
+        _acme_toggled = (
+            cluster.acme_enabled is not None
+            and cluster.acme_enabled != existing_cluster.get('acme_enabled', False)
+        )
+        if _acme_toggled or _acme_url_submitted:
             try:
-                from services.haproxy_config import generate_haproxy_config_for_cluster
+                from services.haproxy_config import (
+                    generate_haproxy_config_for_cluster,
+                    is_config_generation_error,
+                    extract_acme_backend_target,
+                )
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
+                if is_config_generation_error(config_content):
+                    # Same sentinel-instead-of-exception contract as the apply path. A
+                    # PENDING version holding the sentinel is a landmine: the operator
+                    # sees a pending change and applies it, replacing the whole config.
+                    logger.error(
+                        f"ACME TOGGLE: config generation for cluster {cluster_id} returned an "
+                        f"error sentinel; no PENDING version created: {config_content!r}"
+                    )
+                    raise HTTPException(
+                        status_code=422,
+                        detail=(
+                            "ACME setting was saved, but a configuration could not be generated "
+                            "for this cluster, so no pending change was created. "
+                            f"Generator reported: {config_content.strip()[:300]}"
+                        ),
+                    )
+                # A URL edit that renders the same `server _acme_mgmt` line changes
+                # nothing on the nodes, so minting a version would put a no-op pending
+                # change in front of the operator. Compare that one line rather than the
+                # whole config: the generator also renders unrelated PENDING entities,
+                # so a full-text diff reports a change on every edit anyone has queued.
+                _active_config = await conn.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                _new_target = extract_acme_backend_target(config_content)
+                _old_target = extract_acme_backend_target(_active_config)
+                if not _acme_toggled and _new_target == _old_target:
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} URL updated but the rendered "
+                        f"challenge backend is unchanged ({_new_target!r}); no config "
+                        f"version created."
+                    )
+                    raise _AcmeNoConfigChange()
+
                 import time as _time
                 import json as _json
-                version_name = f"cluster-{cluster_id}-acme-{'enable' if cluster.acme_enabled else 'disable'}-{int(_time.time())}"
+                if _acme_toggled:
+                    _acme_kind = 'enable' if cluster.acme_enabled else 'disable'
+                else:
+                    _acme_kind = 'backend'
+                version_name = f"cluster-{cluster_id}-acme-{_acme_kind}-{int(_time.time())}"
+                logger.info(
+                    f"ACME-BACKEND: cluster {cluster_id} pending config version "
+                    f"'{version_name}' created — challenge backend {_old_target!r} -> "
+                    f"{_new_target!r}. Apply the cluster for the nodes to pick it up."
+                )
 
                 from utils.entity_snapshot import save_entity_snapshot
                 snapshot_metadata = await save_entity_snapshot(
@@ -482,7 +560,14 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                         "acme_backend_url": existing_cluster.get('acme_backend_url'),
                     },
                     new_values={
-                        "acme_enabled": cluster.acme_enabled,
+                        # `acme_enabled` is None when only the URL was submitted; the
+                        # snapshot must record the value that is actually in force, or a
+                        # rollback would write NULL over a working flag.
+                        "acme_enabled": (
+                            cluster.acme_enabled
+                            if cluster.acme_enabled is not None
+                            else existing_cluster.get('acme_enabled', False)
+                        ),
                         "acme_backend_url": getattr(cluster, 'acme_backend_url', None) or existing_cluster.get('acme_backend_url'),
                     },
                     operation="UPDATE"
@@ -497,9 +582,20 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                     """, cluster_id, version_name, config_content, current_user.get('id', 1), metadata_json)
                 finally:
                     await close_database_connection(conn2)
+            except _AcmeNoConfigChange:
+                # Not an error: the edit was accepted and simply renders the same
+                # address, so there is nothing for the operator to apply.
+                pass
+            except HTTPException:
+                # The sentinel guard above deliberately fails the request. Without this
+                # clause the generic handler below would swallow it and report success
+                # while no PENDING version exists — the exact silent-success failure
+                # mode this change exists to remove.
+                await close_database_connection(conn)
+                raise
             except Exception as acme_err:
                 logger.error(f"Failed to create ACME config version for cluster {cluster_id}: {acme_err}")
-        
+
         await close_database_connection(conn)
         
         # Log activity
@@ -1914,6 +2010,28 @@ defaults
                 logger.info(f"🧩 APPLY: Generating fresh configuration from database for cluster {cluster_id}")
                 fresh_config_content = await generate_haproxy_config_for_cluster(cluster_id, conn)
             
+            # `generate_haproxy_config_for_cluster` reports failure by RETURNING a
+            # one-line comment instead of raising (haproxy_config.py outer `except`).
+            # Without this guard that sentinel is hashed, stored as an APPLIED version
+            # and shipped to every agent — silently replacing the cluster's entire
+            # configuration with a comment. Any generator exception (an out-of-range
+            # port in acme_backend_url is enough) triggers it. Refuse the apply instead;
+            # the previous APPLIED version stays in force.
+            from services.haproxy_config import is_config_generation_error
+            if is_config_generation_error(fresh_config_content):
+                logger.error(
+                    f"APPLY ABORTED: config generation for cluster {cluster_id} returned an "
+                    f"error sentinel instead of a configuration: {fresh_config_content!r}"
+                )
+                raise HTTPException(
+                    status_code=422,
+                    detail=(
+                        "Configuration could not be generated for this cluster, so nothing "
+                        "was applied and the running configuration is unchanged. "
+                        f"Generator reported: {fresh_config_content.strip()[:300]}"
+                    ),
+                )
+
             # Create a new consolidated config version with fresh content
             import hashlib
             import time
@@ -2586,7 +2704,11 @@ async def get_config_version_diff(cluster_id: int, version_id: int, authorizatio
         # HA/VIP (Issue #27): vip-{id}-{action} versions show the generated keepalived.conf
         # each member node will deploy as the change content (VRRP secret masked). Mirrors
         # the ssl-* special case above so VIP uses the STANDARD View Change diff modal.
-        vip_match = re.search(r'vip-(\d+)-(create|update|delete)', current_version['version_name'])
+        # `adopt` MUST stay in this alternation. A vip-* action missing here does not degrade
+        # gracefully: the version falls through to the generic HAProxy diff, which compares this
+        # row's keepalived.conf against the cluster's previous haproxy.cfg and shows the whole
+        # HAProxy config as removed. v1.10.4 added `adopt` without it (fixed in v1.10.8).
+        vip_match = re.search(r'vip-(\d+)-(create|update|delete|adopt)', current_version['version_name'])
         if vip_match:
             vip_id = int(vip_match.group(1))
             vip_action = vip_match.group(2)
@@ -3696,17 +3818,19 @@ async def confirm_restore_config_version(
                     # UPDATE existing frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        UPDATE frontends 
-                        SET bind_address = $1, bind_port = $2, default_backend = $3, 
+                        UPDATE frontends
+                        SET bind_address = $1, bind_port = $2, default_backend = $3,
                             mode = $4, ssl_enabled = $5, ssl_port = $6,
                             maxconn = $7, timeout_client = $8,
+                            log_format = $11, filters = $12,
                             updated_at = CURRENT_TIMESTAMP, last_config_status = 'PENDING'
                         WHERE id = $9 AND cluster_id = $10
-                    """, 
+                    """,
                         parsed_fe.bind_address, parsed_fe.bind_port, parsed_fe.default_backend,
                         parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        fe_id, cluster_id
+                        fe_id, cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_updated"] += 1
                     logger.info(f"RESTORE: Updated frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")
@@ -3714,16 +3838,17 @@ async def confirm_restore_config_version(
                     # CREATE new frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        INSERT INTO frontends 
+                        INSERT INTO frontends
                         (name, bind_address, bind_port, default_backend, mode, ssl_enabled, ssl_port,
                          maxconn, timeout_client,
-                         cluster_id, is_active, last_config_status, created_at, updated_at)
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
-                    """, 
+                         cluster_id, log_format, filters, is_active, last_config_status, created_at, updated_at)
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
+                    """,
                         parsed_fe.name, parsed_fe.bind_address, parsed_fe.bind_port,
                         parsed_fe.default_backend, parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        cluster_id
+                        cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_created"] += 1
                     logger.info(f"RESTORE: Created frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")

backend/routers/config.py

@@ -3,7 +3,7 @@ Configuration Management and Validation API
 Provides endpoints for HAProxy configuration validation, templates, and optimization
 """
 
-from fastapi import APIRouter, HTTPException, Header, Request
+from fastapi import APIRouter, HTTPException, Header, Request, Depends
 from pydantic import BaseModel
 from typing import Dict, List, Any, Optional
 import logging
@@ -18,7 +18,7 @@ from utils.config_templates import (
 )
 from utils.haproxy_config_parser import parse_haproxy_config
 from utils.logging_config import log_with_correlation, PerformanceLogger
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from database.connection import get_database_connection, close_database_connection
 
 router = APIRouter(prefix="/api/config", tags=["Configuration Management"])
@@ -62,7 +62,7 @@ class ConfigOptimizationRequest(BaseModel):
     optimization_level: str = "balanced"  # conservative, balanced, aggressive
     target_environment: str = "production"  # development, staging, production
 
-@router.post("/validate")
+@router.post("/validate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth; runs HAProxy validator on caller input
 async def validate_configuration(
     request: ConfigValidationRequest,
     current_user: dict = None,
@@ -203,7 +203,7 @@ async def get_template_details(template_id: str):
         )
         raise HTTPException(status_code=500, detail=f"Failed to get template: {str(e)}")
 
-@router.post("/templates/{template_id}/generate")
+@router.post("/templates/{template_id}/generate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def generate_configuration(
     template_id: str,
     request: TemplateGenerationRequest,
@@ -271,7 +271,7 @@ async def generate_configuration(
                 detail=f"Configuration generation failed: {str(e)}"
             )
 
-@router.post("/optimize")
+@router.post("/optimize", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def optimize_configuration(
     request: ConfigOptimizationRequest,
     current_user: dict = None,
@@ -855,6 +855,9 @@ async def parse_bulk_config(
                 "response_headers": frontend.response_headers,
                 "options": frontend.options,
                 "tcp_request_rules": frontend.tcp_request_rules,
+                # Issue #38: SPOE filters + frontend log-format
+                "log_format": frontend.log_format,
+                "filters": frontend.filters,
                 # CRITICAL: SSL Advanced Options (parsed from bind directive)
                 "ssl_alpn": frontend.ssl_alpn,
                 "ssl_npn": frontend.ssl_npn,
@@ -1089,7 +1092,69 @@ async def parse_bulk_config(
         
         # Add auto-assignment info at the beginning
         enhanced_warnings = ssl_auto_assign_info + enhanced_warnings
-        
+
+        # ─────────────────────────────────────────────────────────────────
+        # Issue #38: SPOE pre-flight advisories. Surface, at preview time, the
+        # SPOE configurations that would FAIL HAProxy's `haproxy -c` at apply so
+        # the operator sees them BEFORE importing. Cluster-aware: the referenced
+        # SPOE engine config (e.g. coraza.cfg) is a sibling of the cluster's
+        # haproxy_config_path, which HAProxy OpenManager does not provision.
+        # ─────────────────────────────────────────────────────────────────
+        try:
+            _cfg_path = await conn.fetchval(
+                "SELECT haproxy_config_path FROM haproxy_clusters WHERE id = $1",
+                request.cluster_id,
+            ) or "/etc/haproxy/haproxy.cfg"
+            _cfg_dir = _cfg_path.rsplit("/", 1)[0] or "/etc/haproxy"
+            for _fe in frontends_data:
+                _rh = _fe.get("request_headers") or ""
+                _filters = _fe.get("filters") or ""
+                # engines declared by `filter spoe engine <name> config <path>`
+                _declared_engines = set(re.findall(
+                    r"filter\s+spoe\s+engine\s+(\S+)", _filters, re.IGNORECASE))
+                # engines referenced by `... send-spoe-group <name> <group>`
+                _used_engines = set(re.findall(
+                    r"send-spoe-group\s+(\S+)", _rh, re.IGNORECASE))
+                _missing = _used_engines - _declared_engines
+                if _missing:
+                    enhanced_warnings.append(
+                        f"⚠️ Frontend '{_fe['name']}': 'send-spoe-group' references SPOE "
+                        f"engine(s) {', '.join(sorted(_missing))} but no matching "
+                        f"'filter spoe engine <name> ...' line was found. HAProxy will "
+                        f"reject this at apply with \"unable to find SPOE engine\". Add the "
+                        f"filter line to this frontend."
+                    )
+                for _path in re.findall(
+                        r"filter\s+spoe\s+engine\s+\S+\s+config\s+(\S+)",
+                        _filters, re.IGNORECASE):
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': SPOE engine config '{_path}' and its "
+                        f"SPOA backend must exist on the HAProxy host (cluster config dir: "
+                        f"{_cfg_dir}). HAProxy OpenManager preserves the filter directive but "
+                        f"does not provision these files; otherwise 'haproxy -c' fails at apply."
+                    )
+                # Issue #38 follow-up: ACL `-f <file>` pattern-file advisory.
+                # Scan only the structured rule fields (acl/use_backend) —
+                # request_headers/tcp_request_rules were always free-form and
+                # warning on them now would add new noise for existing users.
+                _pattern_paths = []
+                for _rule in (_fe.get("acl_rules") or []) + (_fe.get("use_backend_rules") or []):
+                    if isinstance(_rule, str):
+                        _pattern_paths.extend(
+                            re.findall(r"(?:^|\s)-f\s+(\S+)", _rule))
+                if _pattern_paths:
+                    _uniq = sorted(set(_pattern_paths))
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': ACL/routing rules reference pattern "
+                        f"file(s) {', '.join(_uniq)}. Each file must exist at that exact path "
+                        f"on every HAProxy host in the cluster (cluster config dir: {_cfg_dir}) "
+                        f"— HAProxy OpenManager does not create or distribute pattern files. "
+                        f"A missing file fails safely at 'haproxy -c' (previous config keeps "
+                        f"running)."
+                    )
+        except Exception as _spoe_adv_err:
+            logger.warning(f"SPOE advisory generation skipped: {_spoe_adv_err}")
+
         # BULK IMPORT MVP: Check existing entities for UPSERT detection
         # Mark each entity as new or update for UI display
         # CRITICAL: Only mark as UPDATE if there are actual field changes
@@ -1150,7 +1215,17 @@ async def parse_bulk_config(
                 if frontend.get("tcp_request_rules") and frontend["tcp_request_rules"] != existing["tcp_request_rules"]:
                     has_changes = True
                     changes["tcp_request_rules"] = {"old": existing["tcp_request_rules"], "new": frontend["tcp_request_rules"]}
-                
+                # Issue #38: SPOE filters + log-format change detection. REQUIRED for
+                # persistence (not just display): without it, an import that only adds
+                # a `filter`/`log-format` to an existing frontend would be flagged
+                # "no change" and the directive would never be written to the DB.
+                if frontend.get("log_format") and frontend["log_format"] != existing.get("log_format"):
+                    has_changes = True
+                    changes["log_format"] = {"old": existing.get("log_format"), "new": frontend["log_format"]}
+                if frontend.get("filters") and frontend["filters"] != existing.get("filters"):
+                    has_changes = True
+                    changes["filters"] = {"old": existing.get("filters"), "new": frontend["filters"]}
+
                 # CRITICAL: SSL Advanced Options change detection
                 if frontend.get("ssl_alpn") is not None and frontend.get("ssl_alpn") != existing.get("ssl_alpn"):
                     has_changes = True
@@ -2094,7 +2169,18 @@ async def bulk_create_entities(
                         update_fields.append(f"options = ${param_index}")
                         update_values.append(frontend_data["options"])
                         param_index += 1
-                    
+
+                    # Issue #38: SPOE filters + frontend log-format (merge strategy)
+                    if frontend_data.get("log_format") and frontend_data["log_format"] != existing_full.get("log_format"):
+                        update_fields.append(f"log_format = ${param_index}")
+                        update_values.append(frontend_data["log_format"])
+                        param_index += 1
+
+                    if frontend_data.get("filters") and frontend_data["filters"] != existing_full.get("filters"):
+                        update_fields.append(f"filters = ${param_index}")
+                        update_values.append(frontend_data["filters"])
+                        param_index += 1
+
                     # CRITICAL FIX: Update SSL advanced options (alpn, npn, ciphers, etc.)
                     # These are parsed from bind directive and should be preserved in database
                     if "ssl_alpn" in frontend_data and frontend_data.get("ssl_alpn") != existing_full.get("ssl_alpn"):
@@ -2214,9 +2300,10 @@ async def bulk_create_entities(
                             timeout_client, timeout_http_request, maxconn,
                             request_headers, response_headers, tcp_request_rules, options,
                             rate_limit, compression, log_separate, monitor_uri,
-                            cluster_id, acl_rules, use_backend_rules, redirect_rules, updated_at
-                        ) 
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                            cluster_id, acl_rules, use_backend_rules, redirect_rules,
+                            log_format, filters, updated_at
+                        )
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
                         RETURNING id
                     """, 
                         frontend_data["name"],
@@ -2257,7 +2344,9 @@ async def bulk_create_entities(
                         request.cluster_id,
                         json.dumps(frontend_data.get("acl_rules", [])),  # acl_rules
                         json.dumps(frontend_data.get("use_backend_rules", [])),  # use_backend_rules
-                        json.dumps([])   # redirect_rules
+                        json.dumps([]),   # redirect_rules
+                        frontend_data.get("log_format"),  # Issue #38
+                        frontend_data.get("filters")      # Issue #38
                     )
                     
                     created_entities["frontends"].append({

backend/routers/configuration.py

@@ -201,13 +201,15 @@ async def get_pending_config_requests(agent_name: str, x_api_key: Optional[str]
     Called during heartbeat.
     """
     try:
-        # Validate agent API key
+        # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Deployed agents
+        # always send X-API-Key; an absent/invalid key is unauthenticated. This
+        # endpoint also mutates state (marks requests 'processing'), so a keyless
+        # caller could otherwise starve the real agent.
         agent_auth = await validate_agent_api_key(x_api_key)
-        
-        if x_api_key and not agent_auth:
-            logger.warning(f"Invalid API key provided by agent '{agent_name}' for pending requests")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-        
+        if not agent_auth:
+            logger.warning(f"Missing/invalid API key from '{agent_name}' for pending requests")
+            raise HTTPException(status_code=401, detail="Authentication required")
+
         conn = await get_database_connection()
         
         # Get pending requests

backend/routers/csr.py

@@ -0,0 +1,375 @@
+"""
+CSR (Certificate Signing Request) endpoints (v1.9.0).
+
+Generate a private key + CSR in-app, download the CSR PEM, have it signed by
+an external CA, then import the signed certificate — which creates a normal
+ssl_certificates row that flows through the existing pipeline
+(config version → Apply Management → agent pull).
+
+Security posture:
+- All endpoints enforce ssl.* permissions explicitly (including the read
+  endpoints — deliberately stricter than the legacy cert detail route).
+- The private key is NEVER returned by any endpoint here; after import it is
+  reachable only via the existing certificate detail route.
+- Key generation is offloaded to a thread (RSA-4096 takes seconds; the
+  backend runs a single-worker event loop by default) and rate-limited
+  per user via the user_activity_logs COUNT pattern (acme_diagnostics
+  precedent — slowapi is not registered on the app).
+"""
+
+import asyncio
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, HTTPException, Request, Header
+
+from database.connection import get_database_connection, close_database_connection
+from auth_middleware import get_current_user_from_token, check_user_permission
+from models.csr import SSLCSRCreate, SSLCSRImport
+from services import csr_service, ssl_service
+from routers.ssl import _assert_safe_cert_name, validate_user_cluster_access
+from utils.activity_log import log_user_activity
+
+router = APIRouter(prefix="/api/ssl/csrs", tags=["SSL CSRs"])
+logger = logging.getLogger(__name__)
+
+_RATE_LIMIT_CREATE_PER_MIN = 10
+
+# Columns exposed to the API — private_key_pem is deliberately absent so a
+# future `SELECT *` refactor cannot silently start leaking it.
+_CSR_LIST_COLUMNS = """
+    c.id, c.name, c.common_name, c.subject, c.sans, c.key_algorithm,
+    c.status, c.ssl_certificate_id, c.completed_at, c.created_at, c.updated_at,
+    s.name AS certificate_name, u.username AS created_by_username
+"""
+
+_INT32_MAX = 2_147_483_647
+
+
+def _client_ip(request: Optional[Request]) -> Optional[str]:
+    try:
+        return str(request.client.host) if request and request.client else None
+    except Exception:
+        return None
+
+
+def _user_agent(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.headers.get("user-agent") if request else None
+    except Exception:
+        return None
+
+
+async def _require(authorization: Optional[str], action: str):
+    """Authenticate + enforce ssl.<action>; returns current_user or raises 401/403."""
+    current_user = await get_current_user_from_token(authorization)
+    ok = await check_user_permission(current_user["id"], "ssl", action, current_user=current_user)
+    if not ok:
+        raise HTTPException(status_code=403, detail=f"Insufficient permissions: ssl.{action} required")
+    return current_user
+
+
+def _assert_int32_id(csr_id: int) -> None:
+    """ssl_csrs.id is int4 — an out-of-range path param would surface as an
+    asyncpg DataError 500 (Bulgu #96 precedent); return a clean 404 instead."""
+    if csr_id < 1 or csr_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="CSR not found")
+
+
+def _assert_valid_cluster_id(cluster_id: int) -> None:
+    """Same int4 guard for body-supplied cluster ids: haproxy_clusters.id is
+    SERIAL/int4, so an out-of-range value would raise asyncpg DataError inside
+    validate_user_cluster_access and surface as a 500 with the raw driver
+    error. Fail with the same clean 404 the cluster lookup itself produces."""
+    if not isinstance(cluster_id, int) or cluster_id < 1 or cluster_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="Cluster not found")
+
+
+async def _enforce_create_rate_limit(conn, user_id: int) -> None:
+    """Per-user per-minute limit on key generation, counted against the
+    csr_create audit-log action (acme_diagnostics _enforce_rate_limit pattern,
+    backed by the (user_id, action, created_at DESC) composite index)."""
+    cnt = await conn.fetchval(
+        """
+        SELECT COUNT(*)
+        FROM user_activity_logs
+        WHERE user_id = $1
+          AND action = 'csr_create'
+          AND created_at >= NOW() - INTERVAL '60 seconds'
+        """,
+        user_id,
+    )
+    if cnt is not None and cnt >= _RATE_LIMIT_CREATE_PER_MIN:
+        raise HTTPException(
+            status_code=429,
+            detail=(
+                f"Rate limit exceeded: at most {_RATE_LIMIT_CREATE_PER_MIN} "
+                "CSRs may be created per minute"
+            ),
+        )
+
+
+@router.post("")
+async def create_csr(payload: SSLCSRCreate, request: Request, authorization: Optional[str] = Header(None)):
+    """Generate a private key + CSR. Returns the CSR PEM immediately (so the
+    UI can show copy/download in one round trip) — never the private key."""
+    current_user = await _require(authorization, "create")
+    conn = None
+    try:
+        # Belt and braces on top of the model validator — same duplication
+        # convention as the certificate create route.
+        _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+        await _enforce_create_rate_limit(conn, current_user["id"])
+
+        # Fail fast on a taken name BEFORE burning CPU on key generation;
+        # insert_csr_row re-checks and the partial unique index closes the race.
+        await csr_service.assert_csr_name_available(conn, payload.name)
+
+        bundle = await asyncio.to_thread(csr_service.generate_csr_bundle, payload)
+        csr_id = await csr_service.insert_csr_row(conn, payload, bundle, current_user["id"])
+
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_create',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'csr_name': payload.name,
+                'common_name': payload.common_name,
+                'sans': bundle['sans'],
+                'key_algorithm': payload.key_algorithm,
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": f"CSR '{payload.name}' created successfully",
+            "csr": csr_service.csr_row_to_dict(row, include_pem=True),
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error creating CSR: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("")
+async def list_csrs(authorization: Optional[str] = Header(None)):
+    """List CSRs (no PEM payloads — fetch the detail route for the CSR PEM).
+    Cluster-agnostic: a CSR binds to clusters only at import time."""
+    await _require(authorization, "read")
+    conn = None
+    try:
+        conn = await get_database_connection()
+        rows = await conn.fetch(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            ORDER BY c.created_at DESC
+            """
+        )
+        return [csr_service.csr_row_to_dict(r) for r in rows]
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing CSRs: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("/{csr_id}")
+async def get_csr(csr_id: int, authorization: Optional[str] = Header(None)):
+    """CSR detail including the CSR PEM. The private key is never included."""
+    await _require(authorization, "read")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        return csr_service.csr_row_to_dict(row, include_pem=True)
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{csr_id}/import")
+async def import_csr_certificate(
+    csr_id: int,
+    payload: SSLCSRImport,
+    request: Request,
+    authorization: Optional[str] = Header(None),
+):
+    """Import the CA-signed certificate for a pending CSR. Creates an
+    ssl_certificates row (source='csr', PENDING) and stages one config
+    version per affected cluster — the operator applies manually."""
+    current_user = await _require(authorization, "create")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        if payload.name:
+            _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+
+        if not payload.is_global:
+            for cluster_id in payload.cluster_ids or []:
+                _assert_valid_cluster_id(cluster_id)
+                await validate_user_cluster_access(current_user["id"], cluster_id, conn)
+
+        result = await csr_service.import_signed_certificate(
+            conn, csr_id, payload, current_user["id"]
+        )
+        cert_id = result["certificate_id"]
+
+        if payload.is_global:
+            cluster_rows = await conn.fetch(
+                "SELECT id FROM haproxy_clusters WHERE is_active = TRUE"
+            )
+            affected_clusters = [r['id'] for r in cluster_rows]
+        else:
+            affected_clusters = payload.cluster_ids or []
+
+        # Post-commit staging — a config-generation failure never rolls back
+        # the certificate (same semantics as the manual create flow).
+        sync_results = await ssl_service.stage_ssl_config_versions(
+            conn, cert_id, affected_clusters, action='create',
+            created_by=current_user["id"],
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='create',
+            resource_type='ssl_certificate',
+            resource_id=str(cert_id),
+            details={
+                'certificate_name': result['certificate_name'],
+                'domain': result.get('primary_domain', 'unknown'),
+                'via': 'csr',
+                'csr_id': csr_id,
+                'usage_type': payload.usage_type,
+                'is_global': payload.is_global,
+                'cluster_ids': payload.cluster_ids,
+                'warnings': result['warnings'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_import',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'certificate_id': cert_id,
+                'certificate_name': result['certificate_name'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": (
+                f"Certificate '{result['certificate_name']}' imported "
+                "successfully. Go to Apply Management to deploy."
+            ),
+            "certificate_id": cert_id,
+            "warnings": result["warnings"],
+            "sync_results": sync_results,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error importing signed certificate for CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.delete("/{csr_id}")
+async def delete_csr(csr_id: int, request: Request, authorization: Optional[str] = Header(None)):
+    """Hard delete. For a pending CSR this permanently destroys the private
+    key (any certificate later signed from that CSR becomes unusable); for a
+    completed CSR it only removes history — the imported certificate is not
+    affected (the FK points csr → cert)."""
+    current_user = await _require(authorization, "delete")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        async with conn.transaction():
+            # FOR UPDATE serialises against an in-flight import of the same CSR.
+            row = await conn.fetchrow(
+                "SELECT id, name, status FROM ssl_csrs WHERE id = $1 FOR UPDATE",
+                csr_id,
+            )
+            if not row:
+                raise HTTPException(status_code=404, detail="CSR not found")
+            await conn.execute("DELETE FROM ssl_csrs WHERE id = $1", csr_id)
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='delete',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={'csr_name': row['name'], 'status': row['status']},
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        if row['status'] == 'pending':
+            message = (
+                f"CSR '{row['name']}' deleted — its private key has been "
+                "permanently destroyed."
+            )
+        else:
+            message = (
+                f"CSR '{row['name']}' deleted (history only) — the imported "
+                "certificate is not affected."
+            )
+        return {"message": message}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error deleting CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)

backend/routers/dashboard.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Header
+from fastapi import APIRouter, HTTPException, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 from datetime import datetime
 import logging
@@ -11,7 +12,7 @@ from agent_notifications import get_cluster_agents_status
 router = APIRouter(prefix="/api", tags=["dashboard", "pools"])
 logger = logging.getLogger(__name__)
 
-@router.get("/dashboard/overview")
+@router.get("/dashboard/overview", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaked cluster/pool/agent stats, names, health & alerts anonymously (auth was optional)
 async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization: str = Header(None)):
     """Get dashboard overview with comprehensive statistics, optionally filtered by cluster"""
     try:
@@ -238,7 +239,7 @@ async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization
         logger.error(f"Error fetching dashboard overview: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/dashboard/stats")
+@router.get("/dashboard/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): aggregate cluster/agent counts
 async def get_dashboard_stats():
     """Get dashboard statistics"""
     try:
@@ -279,7 +280,7 @@ async def get_dashboard_stats():
     except Exception as e:
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/pools")
+@router.get("/pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): pool names/env/counts
 async def get_pools():
     """Get all HAProxy cluster pools"""
     try:
@@ -350,7 +351,7 @@ async def get_pools():
         logger.error(f"Error fetching pools: {e}")
         return {"pools": []}
 
-@router.get("/haproxy-cluster-pools")
+@router.get("/haproxy-cluster-pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_cluster_pools():
     """Get all HAProxy cluster pools (legacy endpoint)"""
     # Just call the main pools endpoint
@@ -474,7 +475,7 @@ async def update_pool(pool_id: int, pool: PoolUpdate, authorization: str = Heade
         logger.error(f"Failed to update pool: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to update pool: {str(e)}")
 
-@router.get("/haproxy-cluster-pools/{pool_id}/agents")
+@router.get("/haproxy-cluster-pools/{pool_id}/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): full agent inventory — same class as GET /api/agents
 async def get_pool_agents(pool_id: int):
     """Get all agents for a specific pool"""
     try:
@@ -561,7 +562,7 @@ async def get_pool_agents(pool_id: int):
         logger.error(f"Error fetching pool agents: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to fetch pool agents: {str(e)}")
 
-@router.get("/haproxy/stats")
+@router.get("/haproxy/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_stats(cluster_id: Optional[int] = None):
     """Get HAProxy statistics"""
     try:

backend/routers/dashboard_stats.py

@@ -3,13 +3,22 @@ Dashboard Stats Router
 API endpoints for HAProxy statistics dashboard
 """
 
-from fastapi import APIRouter, HTTPException, Query
+from fastapi import APIRouter, HTTPException, Query, Depends
 from typing import Optional, List
 import logging
 
 from services.dashboard_stats_service import dashboard_stats_service
+from auth_middleware import require_authenticated_user
 
-router = APIRouter(prefix="/api/dashboard-stats", tags=["dashboard-stats"])
+# SECURITY (GHSA-3p5c-m5m4-mjpx): this entire router (traffic metrics, backend
+# health, cluster topology, agent status) was mounted without authentication.
+# Require a valid JWT on every route. The frontend Dashboard already sends the
+# operator JWT on these calls, so this is transparent to the UI.
+router = APIRouter(
+    prefix="/api/dashboard-stats",
+    tags=["dashboard-stats"],
+    dependencies=[Depends(require_authenticated_user)],
+)
 logger = logging.getLogger(__name__)
 
 

backend/routers/frontend.py

@@ -117,6 +117,41 @@ def _rule_contradiction_text(rule: Any) -> Optional[str]:
     return None
 
 
+def _pattern_file_warnings(
+    acl_rules: Optional[List[Any]] = None,
+    use_backend_rules: Optional[List[Any]] = None,
+    redirect_rules: Optional[List[Any]] = None,
+) -> List[str]:
+    """Issue #38 follow-up — non-blocking `-f <file>` pattern-file
+    advisory for the manual frontend API.
+
+    The Bulgu #12 hard reject was removed from the Pydantic models:
+    pattern files are operator-managed host files (same policy as the
+    SPOE `filter ... config <path>` reference preserved since v1.8.8)
+    and the agent's pre-reload `haproxy -c` makes a missing file fail
+    safely. This helper returns one warning listing the unique file
+    paths referenced across the rule fields, or [] when no rule uses
+    `-f` — operators who don't use pattern files see no change.
+    """
+    paths: List[str] = []
+    for rules in (acl_rules, use_backend_rules, redirect_rules):
+        for rule in rules or []:
+            text = rule if isinstance(rule, str) else (
+                rule.get("condition") if isinstance(rule, dict) else None)
+            if isinstance(text, str):
+                paths.extend(re.findall(r"(?:^|\s)-f\s+(\S+)", text))
+    if not paths:
+        return []
+    uniq = sorted(set(paths))
+    return [
+        f"ACL/routing rules reference pattern file(s) {', '.join(uniq)}. "
+        f"Each file must exist at that exact path on every HAProxy host "
+        f"in the cluster — HAProxy OpenManager does not create or "
+        f"distribute pattern files. A missing file fails safely at "
+        f"'haproxy -c' (the previous config keeps running)."
+    ]
+
+
 def _collect_routing_rule_contradictions(
     rules: List[Any], origin_label: str,
 ) -> List[Tuple[str, Any]]:
@@ -408,6 +443,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -424,6 +460,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -453,6 +490,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -465,6 +503,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -480,6 +519,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -492,6 +532,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -601,6 +642,8 @@ async def get_frontends(
                     "response_headers": f.get("response_headers"),
                     "options": f.get("options"),
                     "tcp_request_rules": f.get("tcp_request_rules"),
+                    "log_format": f.get("log_format"),  # Issue #38
+                    "filters": f.get("filters"),        # Issue #38
                     "timeout_client": f.get("timeout_client"),
                     "timeout_http_request": f.get("timeout_http_request"),
                     "rate_limit": f.get("rate_limit"),
@@ -735,18 +778,18 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 acl_rules, redirect_rules, use_backend_rules,
                 request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
                 rate_limit, compression, log_separate, monitor_uri,
-                cluster_id, maxconn, updated_at
-            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                cluster_id, maxconn, log_format, filters, updated_at
+            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
             RETURNING id
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, frontend.ssl_enabled,
             frontend.ssl_certificate_id, ssl_cert_ids_json, frontend.ssl_port, frontend.ssl_cert_path, frontend.ssl_cert, frontend.ssl_verify,
-            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites, 
+            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
             frontend.ssl_min_ver, frontend.ssl_max_ver, frontend.ssl_strict_sni,
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters)
         
         # If cluster_id provided, create new config version for agents
         sync_results = []
@@ -825,12 +868,19 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 user_agent=request.headers.get('user-agent')
             )
         
-        return {
+        response: dict = {
             "message": f"Frontend '{frontend.name}' created successfully",
             "id": frontend_id,
             "frontend": frontend.dict(),
             "sync_results": sync_results
         }
+        # Issue #38 follow-up — non-blocking pattern-file advisory
+        # (additive field; absent when no rule references `-f`).
+        pattern_warnings = _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if pattern_warnings:
+            response["warnings"] = pattern_warnings
+        return response
     except HTTPException:
         raise
     except Exception as e:
@@ -1060,9 +1110,10 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                 acl_rules = $20, redirect_rules = $21, use_backend_rules = $22,
                 request_headers = $23, response_headers = $24, options = $25, tcp_request_rules = $26, timeout_client = $27, timeout_http_request = $28,
                 rate_limit = $29, compression = $30, log_separate = $31, monitor_uri = $32,
-                cluster_id = $33, maxconn = $34, updated_at = CURRENT_TIMESTAMP
-            WHERE id = $35
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+                cluster_id = $33, maxconn = $34, log_format = $35, filters = $36,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $37
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, ssl_enabled,
             ssl_certificate_id, ssl_cert_ids_json, ssl_port, ssl_cert_path, ssl_cert, ssl_verify,
             frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
@@ -1070,7 +1121,7 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn, frontend_id)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters, frontend_id)
             
         # Debug: Check what was actually saved
         updated_frontend = await conn.fetchrow("""
@@ -1124,6 +1175,8 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                     "response_headers": frontend.response_headers,
                     "options": filtered_options,
                     "tcp_request_rules": frontend.tcp_request_rules,
+                    "log_format": frontend.log_format,  # Issue #38
+                    "filters": frontend.filters,        # Issue #38
                     "timeout_client": frontend.timeout_client,
                     "timeout_http_request": frontend.timeout_http_request,
                     "rate_limit": frontend.rate_limit,
@@ -1235,8 +1288,12 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
         # yellow toast on the next refresh. The save SUCCEEDED; the
         # warnings only flag latent legacy data the operator may
         # want to clean up at their convenience.
-        if contradiction_warnings:
-            response["warnings"] = contradiction_warnings
+        # Issue #38 follow-up — append the pattern-file advisory to
+        # the same list (additive; empty when no rule uses `-f`).
+        all_warnings = list(contradiction_warnings or []) + _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if all_warnings:
+            response["warnings"] = all_warnings
         return response
     except HTTPException:
         raise

backend/routers/health.py

@@ -3,8 +3,9 @@ Production-Ready Health Check and Monitoring Endpoints
 Provides comprehensive system health monitoring for Kubernetes and production environments
 """
 
-from fastapi import APIRouter, HTTPException
+from fastapi import APIRouter, HTTPException, Depends
 from fastapi.responses import JSONResponse
+from auth_middleware import require_authenticated_user
 import logging
 import asyncio
 import time
@@ -74,7 +75,7 @@ async def readiness_probe():
         logger.error(f"Readiness probe failed: {e}")
         raise HTTPException(status_code=503, detail=f"Not ready: {str(e)}")
 
-@router.get("/deep")
+@router.get("/deep", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks host CPU/mem/disk, DB/redis/python versions, PID
 async def deep_health_check():
     """Comprehensive health check with detailed system information"""
     global _health_cache
@@ -197,7 +198,7 @@ async def deep_health_check():
         
         raise HTTPException(status_code=503, detail=error_response)
 
-@router.get("/agents")
+@router.get("/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks agent names/hostnames
 async def agents_health():
     """Monitor agent connectivity and health status"""
     try:
@@ -261,7 +262,7 @@ async def agents_health():
         logger.error(f"Agent health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Agent health check failed: {str(e)}")
 
-@router.get("/clusters")
+@router.get("/clusters", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks cluster names, HAProxy versions, pending counts
 async def clusters_health():
     """Monitor HAProxy cluster health and configuration status"""
     try:
@@ -329,7 +330,7 @@ async def clusters_health():
         logger.error(f"Cluster health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Cluster health check failed: {str(e)}")
 
-@router.get("/errors")
+@router.get("/errors", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): app error metrics, sibling of /deep,/agents,/clusters
 async def error_statistics():
     """Get application error statistics and metrics"""
     try:

backend/routers/letsencrypt.py

@@ -87,6 +87,32 @@ class AccountCreate(BaseModel):
             raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
         return v
 
+    @field_validator('directory_url')
+    @classmethod
+    def _validate_directory_url(cls, v):
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): reject non-https URLs and literal
+        # non-public IP hosts at the API boundary. The full DNS-based SSRF check
+        # runs at fetch time (acme_service.get_directory -> ssrf_guard).
+        if not v:
+            return v
+        from urllib.parse import urlparse
+        import ipaddress
+        from utils.ssrf_guard import is_public_ip
+        parsed = urlparse(v.strip())
+        if parsed.scheme.lower() != 'https':
+            raise ValueError("directory_url must be an https URL")
+        host = parsed.hostname
+        if not host:
+            raise ValueError("directory_url has no host")
+        try:
+            ipaddress.ip_address(host)
+            is_ip_literal = True
+        except ValueError:
+            is_ip_literal = False
+        if is_ip_literal and not is_public_ip(host):
+            raise ValueError("directory_url must not point to a private/loopback IP address")
+        return v
+
     @model_validator(mode='after')
     def _require_provider_for_dns01(self):
         if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
@@ -900,12 +926,22 @@ async def import_le_ca_chain(authorization: str = Header(None)):
         ("https://letsencrypt.org/certs/r11.pem", "R11 Intermediate"),
     ]
     chain_parts = []
+    # v1.11.0: each download is recorded as an outbound row. The bodies are
+    # public CA certificates, not secrets, and the 8 KB body cap truncates them —
+    # what matters here is which URL failed, with what status.
+    from utils.http_instrumentation import outbound_span, TARGET_LETSENCRYPT_CA
+
     async with aiohttp.ClientSession() as session:
         for url, name in ca_urls:
             try:
-                async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
-                    if resp.status == 200:
-                        chain_parts.append(await resp.text())
+                async with outbound_span(
+                    target=TARGET_LETSENCRYPT_CA, method="GET", url=url
+                ) as span:
+                    async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
+                        text = await resp.text() if resp.status == 200 else None
+                        span.set_response(resp.status, getattr(resp, "headers", None), text)
+                        if resp.status == 200:
+                            chain_parts.append(text)
             except Exception as e:
                 logger.warning(f"Failed to download {name}: {e}")
 

backend/routers/request_logs.py

@@ -0,0 +1,514 @@
+"""v1.11.0 — the read/administration API for the unified request/response log.
+
+Endpoints (declaration order matters — see below):
+
+    GET    /api/request-logs/settings   requestlog.manage
+    PUT    /api/request-logs/settings   requestlog.manage
+    GET    /api/request-logs/stats      requestlog.read
+    POST   /api/request-logs/purge      requestlog.manage
+    GET    /api/request-logs            requestlog.read
+    GET    /api/request-logs/{log_id}   requestlog.read
+
+`/{log_id}` is a single-segment path, so FastAPI — which matches in declaration
+order — would shadow `/settings`, `/stats` and `/purge` if it came first. The
+literals are therefore declared before it. (This is the mirror image of the
+trap in routers/settings.py, where `GET /{category}` sits at the top of the
+file and swallows every literal route added after it.)
+
+Settings are stored in `system_settings` under the `requestlog` category, so
+`GET /api/settings/requestlog` still reads them, but writes go through THIS
+router: the generic `PUT /api/settings/{category}` stringifies values with
+`str(value)`, which turns `True` into `'True'` — not valid JSON, and the
+`::jsonb` cast then fails.
+"""
+import json
+import logging
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from fastapi import APIRouter, Header, HTTPException, Query
+from pydantic import BaseModel, Field, field_validator
+
+from auth_middleware import check_user_permission, get_current_user_from_token
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_settings import (
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATH_LENGTH,
+    SETTINGS_CATEGORY,
+    config_from_mapping,
+    get_config,
+    load_settings_rows,
+    refresh_config,
+    set_config,
+)
+from utils.request_log_sink import TARGET_INBOUND_AGENT, request_log_sink
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/request-logs", tags=["Request Logs"])
+
+# Columns returned by the list endpoint. Bodies and headers are detail-only:
+# a 200-row page carrying two 8 KB JSONB blobs per row is a 3 MB response.
+_LIST_COLUMNS = """
+    id, request_id, direction, target, method, url, path, status_code,
+    status_class, duration_ms, user_id, username, host(client_ip) AS client_ip,
+    error, request_body_bytes, response_body_bytes, truncated, created_at
+"""
+
+_JSONB_COLUMNS = ("query_params", "request_headers", "request_body",
+                  "response_headers", "response_body")
+
+
+class RequestLogSettings(BaseModel):
+    """Operator-tunable capture + retention policy."""
+
+    enabled: bool = True
+    capture_inbound: bool = True
+    capture_outbound: bool = True
+    capture_bodies: bool = True
+    capture_get: bool = True
+    capture_agent_success: bool = False
+    max_body_bytes: int = Field(8192, ge=0, le=262144)
+    sample_rate: float = Field(1.0, ge=0.0, le=1.0)
+    exclude_paths: List[str] = Field(
+        default_factory=lambda: list(DEFAULT_EXCLUDE_PATHS),
+        max_length=MAX_EXCLUDE_PATHS,
+    )
+    success_retention_days: int = Field(7, ge=1, le=365)
+    error_retention_days: int = Field(30, ge=1, le=365)
+    max_rows: int = Field(500000, ge=1000, le=50_000_000)
+    prune_interval_minutes: int = Field(60, ge=5, le=1440)
+
+    @field_validator("exclude_paths")
+    @classmethod
+    def _validate_paths(cls, value: List[str]) -> List[str]:
+        for entry in value:
+            if not entry.startswith("/"):
+                raise ValueError("exclude_paths entries must start with '/'")
+            if len(entry) > MAX_EXCLUDE_PATH_LENGTH:
+                raise ValueError(
+                    f"exclude_paths entries must be <= {MAX_EXCLUDE_PATH_LENGTH} characters"
+                )
+        return value
+
+
+async def _require(authorization: Optional[str], action: str) -> Dict[str, Any]:
+    """Authenticate, then enforce `requestlog.<action>`.
+
+    `current_user=` is passed through so the admin bypass in
+    check_user_permission short-circuits without a second DB round-trip.
+    """
+    current_user = await get_current_user_from_token(authorization)
+    allowed = await check_user_permission(
+        current_user["id"], "requestlog", action, current_user=current_user
+    )
+    if not allowed:
+        raise HTTPException(
+            status_code=403,
+            detail=f"Insufficient permissions: requestlog.{action} required",
+        )
+    return current_user
+
+
+async def _can_manage(current_user: Dict[str, Any]) -> bool:
+    return await check_user_permission(
+        current_user["id"], "requestlog", "manage", current_user=current_user
+    )
+
+
+def _parse_jsonb(value: Any) -> Any:
+    """asyncpg has no JSONB codec on this pool, so JSONB comes back as raw
+    text. This router is a new contract, so it parses server-side and returns
+    real JSON rather than pushing a JSON.parse() into the UI."""
+    if isinstance(value, str):
+        try:
+            return json.loads(value)
+        except (json.JSONDecodeError, ValueError):
+            return value
+    return value
+
+
+def _row_to_dict(row) -> Dict[str, Any]:
+    out = dict(row)
+    for key in _JSONB_COLUMNS:
+        if key in out:
+            out[key] = _parse_jsonb(out[key])
+    created = out.get("created_at")
+    if isinstance(created, datetime):
+        out["created_at"] = created.isoformat()
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Literal paths FIRST — see the module docstring.
+# ---------------------------------------------------------------------------
+
+
+@router.get("/settings")
+async def get_request_log_settings(authorization: Optional[str] = Header(None)):
+    """Current capture + retention policy, plus the shipped defaults so the UI
+    can offer a 'reset' without hardcoding them."""
+    await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        values = await load_settings_rows(conn)
+        config = config_from_mapping(values) if values else DEFAULT_CONFIG
+        return {
+            "settings": config.as_dict(),
+            "defaults": DEFAULT_CONFIG.as_dict(),
+            "category": SETTINGS_CATEGORY,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.put("/settings")
+async def update_request_log_settings(
+    body: RequestLogSettings,
+    authorization: Optional[str] = Header(None),
+):
+    """Persist the policy and apply it immediately.
+
+    `refresh_config()` at the end is what makes an operator's change take
+    effect on the very next request instead of up to 30 seconds later, when
+    the writer loop would otherwise pick it up.
+    """
+    current_user = await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        updated = []
+        for suffix, value in body.model_dump().items():
+            await conn.execute(
+                """
+                INSERT INTO system_settings (key, value, category, updated_at, updated_by)
+                VALUES ($1, $2::jsonb, $3, $4, $5)
+                ON CONFLICT (key) DO UPDATE SET
+                    value = EXCLUDED.value,
+                    updated_at = EXCLUDED.updated_at,
+                    updated_by = EXCLUDED.updated_by
+                """,
+                f"{SETTINGS_CATEGORY}.{suffix}",
+                json.dumps(value),
+                SETTINGS_CATEGORY,
+                datetime.utcnow(),
+                current_user.get("id"),
+            )
+            updated.append(suffix)
+
+        # Apply in-process right away, then re-read so this worker's snapshot
+        # is exactly what is on disk.
+        set_config(config_from_mapping(body.model_dump()))
+        await refresh_config()
+
+        logger.info(
+            f"Request log settings updated by {current_user.get('username')}: {len(updated)} keys"
+        )
+        return {"message": f"Updated {len(updated)} settings", "settings": get_config().as_dict()}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error updating request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to update request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/stats")
+async def get_request_log_stats(
+    authorization: Optional[str] = Header(None),
+    hours: int = Query(24, ge=1, le=720),
+):
+    """Volume and error breakdown over a window, plus table-level totals and
+    this worker's sink counters (so a saturated queue is visible)."""
+    await _require(authorization, "read")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        by_direction = await conn.fetch(
+            """
+            SELECT direction,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors,
+                   COALESCE(ROUND(AVG(duration_ms))::int, 0)                   AS avg_duration_ms,
+                   COALESCE(MAX(duration_ms), 0)                               AS max_duration_ms
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY direction
+            """,
+            str(hours),
+        )
+
+        by_status = await conn.fetch(
+            """
+            SELECT status_class, COUNT(*) AS total
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY status_class
+            ORDER BY status_class
+            """,
+            str(hours),
+        )
+
+        by_target = await conn.fetch(
+            """
+            SELECT target,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors
+            FROM request_logs
+            WHERE target IS NOT NULL
+              AND created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY target
+            ORDER BY total DESC
+            LIMIT 20
+            """,
+            str(hours),
+        )
+
+        totals = await conn.fetchrow(
+            "SELECT COUNT(*) AS total_rows, MIN(created_at) AS oldest_at, "
+            "MAX(created_at) AS newest_at FROM request_logs"
+        )
+
+        return {
+            "window_hours": hours,
+            "by_direction": [dict(r) for r in by_direction],
+            "by_status_class": [dict(r) for r in by_status],
+            "by_target": [dict(r) for r in by_target],
+            "total_rows": (totals or {}).get("total_rows", 0),
+            "oldest_at": totals["oldest_at"].isoformat() if totals and totals["oldest_at"] else None,
+            "newest_at": totals["newest_at"].isoformat() if totals and totals["newest_at"] else None,
+            # THIS WORKER only. The sink is a module global, so with
+            # UVICORN_WORKERS > 1 each process keeps its own queue and its own
+            # counters, and whichever worker happens to serve this request is
+            # the one being reported. Labelled rather than aggregated: there is
+            # no cross-process channel here, and a number that looks fleet-wide
+            # but is not would understate drops by exactly the worker count.
+            "sink": {**request_log_sink.stats, "scope": "this worker only"},
+            "retention": {
+                "success_retention_days": get_config().success_retention_days,
+                "error_retention_days": get_config().error_retention_days,
+                "max_rows": get_config().max_rows,
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log stats: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log stats")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.post("/purge")
+async def purge_request_logs(authorization: Optional[str] = Header(None)):
+    """Run a retention pass now, ignoring the watermark.
+
+    This applies the CONFIGURED retention — it is not a 'delete everything'
+    button. It exists so an operator who has just lowered the retention does
+    not have to wait for the next scheduled pass to reclaim the space.
+    """
+    current_user = await _require(authorization, "manage")
+    from utils.request_log_prune import prune_request_logs_if_due
+
+    counts = await prune_request_logs_if_due(force=True)
+    logger.info(f"Manual request log purge by {current_user.get('username')}: {counts}")
+    return {
+        "message": "Retention pass completed",
+        "removed": {
+            "success": counts.get("success", 0),
+            "error": counts.get("error", 0),
+            "overflow": counts.get("overflow", 0),
+        },
+        "ran": bool(counts.get("ran")),
+    }
+
+
+# ---------------------------------------------------------------------------
+# List, then the catch-all detail route LAST.
+# ---------------------------------------------------------------------------
+
+
+@router.get("")
+async def list_request_logs(
+    authorization: Optional[str] = Header(None),
+    direction: Optional[str] = Query(None, pattern="^(inbound|outbound)$"),
+    status_class: Optional[int] = Query(None, ge=0, le=5),
+    method: Optional[str] = Query(None, max_length=10),
+    target: Optional[str] = Query(None, max_length=32),
+    user_id: Optional[int] = Query(None, ge=1),
+    path_prefix: Optional[str] = Query(None, max_length=200),
+    q: Optional[str] = Query(None, max_length=200),
+    request_id: Optional[str] = Query(None, max_length=64),
+    errors_only: bool = Query(False),
+    since: Optional[datetime] = Query(None),
+    until: Optional[datetime] = Query(None),
+    min_duration_ms: Optional[int] = Query(None, ge=0),
+    limit: int = Query(50, ge=1, le=500),
+    offset: int = Query(0, ge=0),
+):
+    """Filtered, server-paginated list. Bodies are not included — use the
+    detail endpoint for those."""
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    where: List[str] = []
+    params: List[Any] = []
+
+    def add(clause_template: str, value: Any) -> None:
+        params.append(value)
+        where.append(clause_template.format(n=len(params)))
+
+    if direction:
+        add("direction = ${n}", direction)
+    if status_class is not None:
+        add("status_class = ${n}", status_class)
+    if method:
+        add("method = ${n}", method.upper())
+    if target:
+        add("target = ${n}", target)
+    if user_id is not None:
+        add("user_id = ${n}", user_id)
+    if path_prefix:
+        add("path LIKE ${n} || '%'", path_prefix)
+    if q:
+        # Substring search has no index to lean on; it is the deliberately slow
+        # filter and should be combined with a time window.
+        add("url ILIKE '%' || ${n} || '%'", q)
+    if request_id:
+        add("request_id = ${n}", request_id)
+    if errors_only:
+        where.append("(status_class = 0 OR status_class >= 4)")
+    if since:
+        add("created_at >= ${n}", since)
+    if until:
+        add("created_at <= ${n}", until)
+    if min_duration_ms is not None:
+        add("duration_ms >= ${n}", min_duration_ms)
+
+    # Self-scoping. Captured bodies are a broader disclosure surface than the
+    # existing activity log, so a caller holding only `requestlog.read` sees
+    # their OWN inbound requests, plus the fleet's. `requestlog.manage` (and the
+    # is_admin bypass inside it) lifts the restriction.
+    #
+    # The agent clause is not a widening for its own sake, it is what makes the
+    # `operator` grant do what the migration says it is for: "operators debug
+    # failing applies and ACME orders, so they get read access to the request
+    # log". An apply fails on the NODE, and the node reports that back over its
+    # own API key - so the row carrying the diagnosis is an agent row with
+    # `user_id IS NULL`, which own-rows-only scoping hid from exactly the role
+    # the grant was written for. Scoped on `target`, not on `user_id IS NULL`:
+    # anonymous inbound traffic (failed logins and their usernames, unauthorised
+    # probes) is NOT agent traffic and stays admin-only.
+    if not can_manage:
+        params.append(current_user["id"])
+        own = f"user_id = ${len(params)}"
+        params.append(TARGET_INBOUND_AGENT)
+        where.append(
+            f"(direction = 'inbound' AND ({own} OR target = ${len(params)}))"
+        )
+
+    where_sql = (" WHERE " + " AND ".join(where)) if where else ""
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        rows = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs{where_sql} "
+            f"ORDER BY id DESC LIMIT ${len(params) + 1} OFFSET ${len(params) + 2}",
+            *params, limit, offset,
+        )
+
+        # Bounded count: an unfiltered COUNT(*) over a multi-million-row table
+        # is a sequential scan on every page change. Cap it and tell the client
+        # the number is a floor.
+        count_cap = 10001
+        counted = await conn.fetchval(
+            f"SELECT COUNT(*) FROM (SELECT 1 FROM request_logs{where_sql} LIMIT {count_cap}) t",
+            *params,
+        )
+        total = int(counted or 0)
+
+        return {
+            "logs": [_row_to_dict(r) for r in rows],
+            "total": total,
+            "total_is_estimate": total >= count_cap,
+            "limit": limit,
+            "offset": offset,
+            "scoped_to_self": not can_manage,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing request logs: {e}")
+        raise HTTPException(status_code=500, detail="Failed to list request logs")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/{log_id}")
+async def get_request_log(log_id: int, authorization: Optional[str] = Header(None)):
+    """One exchange in full, plus every other row sharing its `request_id`.
+
+    That `related` list is the point of the feature: one inbound API call and
+    the ACME / DNS / agent calls it triggered read as a single trace.
+    """
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            "SELECT *, host(client_ip) AS client_ip_text FROM request_logs WHERE id = $1",
+            log_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        record = _row_to_dict(row)
+        record["client_ip"] = record.pop("client_ip_text", None)
+
+        if not can_manage and not (
+            record.get("direction") == "inbound"
+            and (
+                record.get("user_id") == current_user["id"]
+                or record.get("target") == TARGET_INBOUND_AGENT
+            )
+        ):
+            # Same self-scoping rule as the list endpoint. 404 rather than 403
+            # so the endpoint does not confirm that a given id exists.
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        related = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs "
+            "WHERE request_id = $1 AND id <> $2 ORDER BY id ASC LIMIT 100",
+            record["request_id"], log_id,
+        )
+
+        return {"log": record, "related": [_row_to_dict(r) for r in related]}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log {log_id}: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log entry")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)

backend/routers/settings.py

@@ -1,10 +1,12 @@
 from fastapi import APIRouter, HTTPException, Header
 from pydantic import BaseModel
 from typing import Dict, Any
+import json
 import logging
 from datetime import datetime
 
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
 
 logger = logging.getLogger(__name__)
 
@@ -50,6 +52,39 @@ async def get_settings_by_category(category: str, authorization: str = Header(No
         await close_database_connection(conn)
 
 
+def _validate_acme_challenge_backend_url(value):
+    """Validate `acme.challenge_backend_url` exactly as the config renderer reads it.
+
+    Settings values are stored as jsonb, so the renderer json.loads them before use
+    (services/haproxy_config.py). Validating the raw column text instead of the
+    decoded string would check the quoting rather than the URL.
+    """
+    decoded = value
+    if isinstance(decoded, str):
+        try:
+            decoded = json.loads(decoded)
+        except (json.JSONDecodeError, TypeError):
+            pass
+    if decoded is None:
+        return
+    try:
+        validate_acme_backend_url(str(decoded))
+    except AcmeBackendUrlError as exc:
+        raise HTTPException(
+            status_code=422,
+            detail=f"acme.challenge_backend_url: {exc}",
+        ) from None
+
+
+# Per-key validators for settings that end up in generated configuration or in
+# outbound requests. Everything else is still written through unchecked; this is a
+# deliberate allow-list of the keys where a bad value causes silent breakage rather
+# than an obvious one.
+_SETTING_VALIDATORS = {
+    "acme.challenge_backend_url": _validate_acme_challenge_backend_url,
+}
+
+
 @router.put("/{category}")
 async def update_settings_by_category(
     category: str,
@@ -59,6 +94,13 @@ async def update_settings_by_category(
     current_user = await _get_admin_user(authorization)
     conn = await get_database_connection()
     try:
+        # Validate the whole batch before writing any of it, so a rejected key cannot
+        # leave the category half-applied.
+        for key_suffix, value in body.settings.items():
+            validator = _SETTING_VALIDATORS.get(f"{category}.{key_suffix}")
+            if validator is not None:
+                validator(value)
+
         updated = []
         for key_suffix, value in body.settings.items():
             full_key = f"{category}.{key_suffix}"
@@ -104,19 +146,52 @@ async def test_acme_connection(authorization: str = Header(None), directory_url:
         finally:
             await close_database_connection(conn)
 
+    # SECURITY (GHSA-3vh4-gvxx-wm2p): validate the URL before any outbound request
+    # (https-only; block loopback/RFC1918/link-local/cloud-metadata after DNS),
+    # pin the connector to IPv4, and never follow redirects. Also do NOT reflect
+    # arbitrary upstream JSON keys back to the caller — that was an information-
+    # disclosure oracle. Only report presence of the FIXED, known ACME directory
+    # field names (never attacker-controlled data).
+    from utils.ssrf_guard import assert_public_url, safe_connector, SSRFValidationError
+
+    directory_url = str(directory_url)
+    try:
+        await assert_public_url(directory_url)
+    except SSRFValidationError as e:
+        return {"success": False, "error": f"Refused to fetch directory URL: {e}"}
+
+    _KNOWN_ACME_FIELDS = ["newNonce", "newAccount", "newOrder", "newAuthz", "revokeCert", "keyChange"]
     try:
         import aiohttp
-        async with aiohttp.ClientSession() as session:
-            async with session.get(str(directory_url), timeout=aiohttp.ClientTimeout(total=10)) as resp:
-                if resp.status == 200:
-                    data = await resp.json()
-                    return {
-                        "success": True,
-                        "directory": str(directory_url),
-                        "endpoints": list(data.keys()) if isinstance(data, dict) else []
-                    }
-                else:
-                    return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
+        # v1.11.0: this handler returns str(e) to the caller and logs nothing —
+        # the span gives the failed probe a durable record.
+        from utils.http_instrumentation import outbound_span, TARGET_SETTINGS_PROBE
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
+            async with outbound_span(
+                target=TARGET_SETTINGS_PROBE, method="GET", url=directory_url
+            ) as span:
+                async with session.get(
+                    directory_url,
+                    timeout=aiohttp.ClientTimeout(total=10),
+                    allow_redirects=False,
+                ) as resp:
+                    if resp.status == 200:
+                        data = await resp.json(content_type=None)
+                        span.set_response(resp.status, getattr(resp, "headers", None), data)
+                        if not isinstance(data, dict):
+                            return {"success": False, "error": "Directory URL did not return a JSON object"}
+                        present = [k for k in _KNOWN_ACME_FIELDS if k in data]
+                        if not present:
+                            return {"success": False, "error": "Response is not a valid ACME directory"}
+                        return {
+                            "success": True,
+                            "directory": directory_url,
+                            "endpoints": present,
+                        }
+                    else:
+                        span.set_response(resp.status, getattr(resp, "headers", None))
+                        return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
     except HTTPException:
         raise
     except Exception as e:

backend/routers/ssl.py

@@ -9,7 +9,7 @@ from datetime import datetime, timezone
 
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from models.ssl import SSLCertificate, SSLCertificateCreate, SSLCertificateUpdate, SSLCertificateResponse
 from utils.ssl_parser import parse_ssl_certificate, validate_private_key, validate_certificate_chain, format_certificate_info
 from utils.activity_log import log_user_activity
@@ -825,7 +825,8 @@ async def get_ssl_certificate(cert_id: int, authorization: str = Header(None)):
         logger.error(f"Error getting SSL certificate: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/certificates/{cert_id}/config-versions")
+@router.get("/certificates/{cert_id}/config-versions",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was unauthenticated
 async def get_ssl_certificate_config_versions(cert_id: int):
     """Get config version history for specific SSL certificate"""
     try:

backend/routers/vip.py

@@ -458,6 +458,46 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
         await close_database_connection(conn)
 
 
+# ORDER MATTERS: every literal path under this router MUST be declared before the
+# `/{vip_id}` routes below. FastAPI matches in declaration order, so a literal placed after
+# `/{vip_id}` is swallowed by it and answered with 422 ("discoveries" is not an int) — the
+# handler never runs. That is not a visible failure either: the HA/VIP page treats any
+# non-OK response as "nothing to show", so the whole adoption feature silently disappears.
+# See the v1.10.4 adoption section further down for the endpoint's own documentation.
+@router.get("/discoveries")
+async def list_vip_discoveries(cluster_id: Optional[int] = None, authorization: str = Header(None)):
+    """Unmanaged keepalived configs the agents found on their nodes.
+
+    Read-only and safe to poll: this is what the HA/VIP page shows so an existing VIP is
+    visible before anyone adopts it.
+
+    `cluster_id` scopes the result to the agents in that cluster's pool, resolved exactly like
+    the VIP list above. Without it every discovery in the fleet is returned, which is what a
+    caller that does not know about the parameter still gets.
+    """
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        try:
+            rows = await conn.fetch("""
+                SELECT d.*, a.name AS agent_name, a.pool_id, p.name AS pool_name,
+                       av.is_active AS adopted_vip_active
+                FROM vip_discoveries d
+                JOIN agents a ON a.id = d.agent_id
+                LEFT JOIN haproxy_cluster_pools p ON p.id = a.pool_id
+                LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
+                WHERE $1::int IS NULL
+                   OR a.pool_id = (SELECT pool_id FROM haproxy_clusters WHERE id = $1::int)
+                ORDER BY d.reported_at DESC, d.id DESC
+            """, cluster_id)
+        except Exception as exc:  # noqa: BLE001 — a missing relation degrades to empty (B-7)
+            logger.debug(f"vip_discoveries unavailable: {exc}")
+            return {"discoveries": []}
+        return {"discoveries": [_discovery_row_to_api(r) for r in rows]}
+    finally:
+        await close_database_connection(conn)
+
+
 @router.get("/{vip_id}")
 async def get_vip(vip_id: int, authorization: str = Header(None)):
     await _require(authorization, "read")
@@ -985,3 +1025,425 @@ async def vip_status(vip_id: int, authorization: str = Header(None)):
 # endpoint (cluster.py get_config_version_diff, vip-* branch) via render_vip_config_masked
 # above — there is no bespoke VIP preview endpoint, so VIP changes use the product's
 # standard "View Change" like every other entity (issue #27 follow-up).
+
+
+# ---------------------------------------------------------------------------
+# v1.10.4 — Adoption of a keepalived setup that already exists on the nodes
+# ---------------------------------------------------------------------------
+# The HA/VIP page starts empty on a fleet that already runs keepalived, because the flow is
+# one-way: VIPs are declared here and pushed to the node, and nothing read what was already
+# there. The agent now reports the keepalived.conf it found (read-only) into vip_discoveries;
+# these two endpoints list those findings and turn one into a managed VIP.
+#
+# Adoption REPLACES the operator's file with our render, so it is gated hard: the parser
+# reports every directive we cannot reproduce and every value we cannot know, and adoption
+# refuses while any remain. See services/keepalived_parser.py for the reasoning.
+
+
+def _discovery_row_to_api(row) -> dict:
+    """Shape a vip_discoveries row for the UI. Never includes the VRRP password: the stored
+    config copy is masked and the analysis has the secret replaced by a boolean."""
+    analysis = row["analysis"]
+    if isinstance(analysis, str):
+        try:
+            analysis = json.loads(analysis)
+        except (json.JSONDecodeError, TypeError):
+            analysis = None
+    return {
+        "agent_id": row["agent_id"],
+        "agent_name": row["agent_name"],
+        "pool_id": row["pool_id"],
+        "pool_name": row["pool_name"],
+        "config_path": row["config_path"],
+        "config_hash": row["config_hash"],
+        "is_managed": row["is_managed"],
+        "parse_error": row["parse_error"],
+        "adopted_vip_id": row["adopted_vip_id"],
+        # v1.10.8 — adoptability is derived from whether the linked VIP is STILL active, not
+        # from the link existing. `adopted_vip_id` is write-once and nothing clears it, and a
+        # VIP is only ever soft-deleted (is_active=FALSE), so the column's ON DELETE SET NULL
+        # never fires. Keying the UI on the link alone made a rejected adoption hide the node
+        # from the panel forever: the VIP was gone from the VIP list too, and the agent does not
+        # re-report an unchanged file. Deriving it here self-heals reject, undo-reject, approved
+        # teardown and anything added later, without a write on each path.
+        "adopted_vip_active": bool(row["adopted_vip_active"]) if row["adopted_vip_id"] else False,
+        "reported_at": row["reported_at"].isoformat() if row["reported_at"] else None,
+        "config_preview": row["raw_config_masked"],
+        "analysis": analysis,
+    }
+
+
+# NOTE: list_vip_discoveries lives above the `/{vip_id}` routes — see the ordering comment
+# there. `/adopt` below is a POST and no `POST /{vip_id}` exists, so it is not shadowed.
+
+
+def _find_candidate(analysis: Optional[dict], instance_name: str) -> Optional[dict]:
+    for cand in ((analysis or {}).get("candidates") or []):
+        if cand.get("instance_name") == instance_name:
+            return cand
+    return None
+
+
+async def _collect_instance_participants(conn, *, pool_id: int, vrid: int, virtual_ip: str):
+    """Every discovered node in the pool that reports the SAME vrrp_instance.
+
+    v1.10.8. Adoption used to take only the node whose row was clicked, which broke the exact
+    case the feature exists for — a running HA pair:
+
+      * adopting the BACKUP alone produced a VIP whose apply fails outright, because apply
+        requires exactly one MASTER member;
+      * adopting the MASTER alone left the peer unmanaged, and adopting it afterwards hit the
+        VRID-collision guard with a 409, so the pair could never be completed from the panel;
+      * worst, on a UNICAST pair the single-member render silently drops the unicast block —
+        render_keepalived_conf only emits it when peer_ips is non-empty — so keepalived falls
+        back to multicast on the adopted node while its peer stays unicast. They stop seeing
+        each other and BOTH claim the VIP.
+
+    Identity is (virtual_router_id, virtual address), which is what keepalived itself uses to
+    decide two nodes belong to one VRRP group, so it is the correct key. A node whose config
+    failed to parse cannot be matched and is therefore skipped — the unicast peer check in the
+    caller is what stops that turning into a silent half-adoption.
+    """
+    rows = await conn.fetch("""
+        SELECT d.agent_id, d.config_hash, d.analysis, d.parse_error, d.adopted_vip_id,
+               d.auth_pass_encrypted,
+               a.name AS agent_name, a.ip_address, av.is_active AS adopted_vip_active
+        FROM vip_discoveries d
+        JOIN agents a ON a.id = d.agent_id
+        LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
+        WHERE a.pool_id = $1 AND COALESCE(a.enabled, TRUE) = TRUE
+    """, pool_id)
+
+    participants = []
+    for r in rows:
+        if r["parse_error"]:
+            continue
+        if r["adopted_vip_id"] and r["adopted_vip_active"]:
+            continue  # already under management by a VIP that still stands
+        analysis = r["analysis"]
+        if isinstance(analysis, str):
+            try:
+                analysis = json.loads(analysis)
+            except (json.JSONDecodeError, TypeError):
+                continue
+        for cand in ((analysis or {}).get("candidates") or []):
+            v = cand.get("vip") or {}
+            if v.get("virtual_router_id") == vrid and v.get("virtual_ip") == virtual_ip:
+                participants.append({
+                    "agent_id": r["agent_id"], "agent_name": r["agent_name"],
+                    "ip_address": r["ip_address"], "config_hash": r["config_hash"],
+                    "auth_pass_encrypted": r["auth_pass_encrypted"],
+                    "candidate": cand,
+                })
+                break
+    return participants
+
+
+@router.post("/adopt")
+async def adopt_vip(payload: dict, request: Request, authorization: str = Header(None)):
+    """Turn one discovered vrrp_instance into a managed VIP.
+
+    Body: agent_id, instance_name, name, [description], [prefix_length], [accept_data_loss].
+
+    Refuses while the parser reports blockers. Two of them are resolvable by the operator
+    rather than fatal:
+      * a missing prefix length can be supplied as `prefix_length` (we never guess a netmask
+        for a live VIP);
+      * "we would delete this directive" can be accepted with `accept_data_loss: true`, which
+        is an explicit choice to lose e.g. a notify hook. Everything else — an unknown VRID, a
+        fractional advert_int, an unsupported auth_type — is not a loss but an impossibility,
+        and no flag overrides it.
+
+    The VIP is created PENDING like any other, so nothing reaches the node until the operator
+    applies it from Apply Management.
+    """
+    current_user = await _require(authorization, "create")
+    agent_id = payload.get("agent_id")
+    instance_name = (payload.get("instance_name") or "").strip()
+    name = (payload.get("name") or "").strip()
+    if not agent_id or not instance_name or not name:
+        raise HTTPException(status_code=400, detail="agent_id, instance_name and name are required")
+
+    conn = await get_database_connection()
+    try:
+        disc = await conn.fetchrow("""
+            SELECT d.*, a.name AS agent_name, a.pool_id, a.ip_address,
+                   av.is_active AS adopted_vip_active
+            FROM vip_discoveries d
+            JOIN agents a ON a.id = d.agent_id
+            LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
+            WHERE d.agent_id = $1
+        """, int(agent_id))
+        if not disc:
+            raise HTTPException(status_code=404, detail="No discovered keepalived config for that agent")
+        # Only an adoption that is still STANDING blocks a new one. A rejected adoption leaves
+        # adopted_vip_id pointing at a soft-deleted VIP, and nothing clears it, so keying on the
+        # link alone made the node permanently unadoptable (v1.10.8).
+        if disc["adopted_vip_id"] and disc["adopted_vip_active"]:
+            raise HTTPException(status_code=409, detail="This discovery has already been adopted")
+        if not disc["pool_id"]:
+            raise HTTPException(status_code=400, detail="The agent is not in a pool; assign it first")
+        if disc["parse_error"]:
+            raise HTTPException(status_code=422,
+                                detail=f"Config could not be parsed: {disc['parse_error']}")
+
+        analysis = disc["analysis"]
+        if isinstance(analysis, str):
+            analysis = json.loads(analysis)
+        cand = _find_candidate(analysis, instance_name)
+        if not cand:
+            raise HTTPException(status_code=404,
+                                detail=f"No vrrp_instance '{instance_name}' in the report")
+
+        vip_fields = dict(cand.get("vip") or {})
+        member = dict(cand.get("member") or {})
+
+        # The operator may supply the one value we refuse to guess.
+        supplied_prefix = payload.get("prefix_length")
+        if vip_fields.get("prefix_length") is None and supplied_prefix is not None:
+            try:
+                vip_fields["prefix_length"] = int(supplied_prefix)
+            except (TypeError, ValueError):
+                raise HTTPException(status_code=400, detail="prefix_length must be an integer")
+
+        from services.keepalived_parser import remaining_blockers
+
+        vrid = vip_fields.get("virtual_router_id")
+        virtual_ip = vip_fields.get("virtual_ip")
+        if vrid is None or not virtual_ip or not member.get("network_interface"):
+            raise HTTPException(status_code=422,
+                                detail="Incomplete candidate (vrid/address/interface)")
+
+        # v1.10.8 — adopt the whole VRRP instance. Every node in the pool reporting this same
+        # VRID + address becomes a member, each with the role, priority and interface ITS OWN
+        # file declares. See _collect_instance_participants for why single-node adoption was
+        # unsafe on a unicast pair.
+        participants = await _collect_instance_participants(
+            conn, pool_id=disc["pool_id"], vrid=int(vrid), virtual_ip=virtual_ip)
+        if not any(p["agent_id"] == int(agent_id) for p in participants):
+            # The reporting node must be in its own instance; if it is not, something changed
+            # underneath us (re-report, concurrent adoption) — refuse rather than guess.
+            raise HTTPException(status_code=409,
+                                detail="The discovery changed while adopting; refresh and try again")
+
+        # One source of truth for which blockers an operator may resolve (see the docstring on
+        # remaining_blockers): a supplied prefix, and an explicit acceptance of directives our
+        # renderer would delete. Nothing else is waivable. Checked for EVERY node we are about
+        # to overwrite, not only the one that was clicked.
+        for p in participants:
+            rem = remaining_blockers(
+                list(p["candidate"].get("blockers") or []),
+                prefix_supplied=vip_fields.get("prefix_length") is not None,
+                accept_data_loss=bool(payload.get("accept_data_loss")),
+            )
+            if rem:
+                raise HTTPException(status_code=422, detail={
+                    "message": f"This keepalived config cannot be adopted as-is ({p['agent_name']})",
+                    "node": p["agent_name"],
+                    "blockers": rem,
+                })
+            if not (p["candidate"].get("member") or {}).get("network_interface"):
+                raise HTTPException(status_code=422,
+                                    detail=f"{p['agent_name']} does not declare an interface for this instance")
+
+        # STRANDING GUARD. Participant resolution can only match a node it can READ, that is
+        # ENABLED, and that is in THIS pool. Every one of those is a door through which a real
+        # member of the VRRP group leaves the set silently — and a silent exit is the whole
+        # failure mode this release exists to close, because the nodes we do adopt get rewritten
+        # while the one that left keeps running an unmanaged config on the same address.
+        #
+        # So rather than guard each door, ask the question directly: is there any reported
+        # keepalived.conf that mentions THIS virtual address and is not among the nodes we are
+        # about to adopt? Scoping on the address keeps an unrelated file elsewhere in the fleet
+        # from blocking every adoption.
+        #
+        # Two exclusions are legitimate rather than stranding:
+        #   * a node already held by a VIP that still STANDS is under management already — and it
+        #     cannot be this instance, because uq_vip_vrid_active forbids a second active VIP with
+        #     this VRID in the pool (the clash check above would have refused first);
+        #   * a node whose file carries our ownership marker (is_managed) is ours already.
+        adopted_ids = [p["agent_id"] for p in participants]
+        stranded = await conn.fetch("""
+            SELECT a.name AS agent_name, a.pool_id, COALESCE(a.enabled, TRUE) AS enabled,
+                   d.parse_error
+            FROM vip_discoveries d
+            JOIN agents a ON a.id = d.agent_id
+            LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
+            WHERE d.raw_config_masked LIKE '%' || $1 || '%'
+              AND NOT (a.id = ANY($2::int[]))
+              AND COALESCE(av.is_active, FALSE) = FALSE
+              AND d.is_managed = FALSE
+            ORDER BY a.name
+        """, virtual_ip, adopted_ids)
+        if stranded:
+            def _why(r):
+                if r["parse_error"]:
+                    return f"its config could not be parsed: {r['parse_error']}"
+                if not r["enabled"]:
+                    return "the agent is disabled, so it can never receive a config"
+                if r["pool_id"] != disc["pool_id"]:
+                    return "it is in a different agent pool, and a VIP's members must share one pool"
+                return "it does not report this vrrp_instance"
+            named = "; ".join(f"{r['agent_name']} — {_why(r)}" for r in stranded)
+            raise HTTPException(status_code=422, detail=(
+                f"These node(s) also reference {virtual_ip} but cannot be taken over with the rest "
+                f"of the instance, so adopting now would rewrite the others and leave them running "
+                f"an unmanaged config on the same address: {named}. Resolve that first, then adopt."))
+
+        # AGREEMENT ON THE SHARED VIP FIELDS. Everything below is stored once on the VIP row and
+        # re-rendered onto EVERY member, so a value taken from the node that happened to be
+        # clicked would be imposed on nodes whose own file said something else. prefix_length is
+        # the sharpest: the design refuses to GUESS a netmask for a live VIP, and quietly copying
+        # one node's netmask onto another is the same change by another name.
+        def _vfield(p, key):
+            return (p["candidate"].get("vip") or {}).get(key)
+
+        for key, label in (("prefix_length", "prefix length"),
+                           ("use_unicast", "unicast/multicast mode"),
+                           ("track_haproxy", "HAProxy tracking")):
+            seen = {}
+            for p in participants:
+                seen.setdefault(_vfield(p, key), []).append(p["agent_name"])
+            if len(seen) > 1:
+                spread = "; ".join(f"{v!r}: {', '.join(names)}" for v, names in seen.items())
+                raise HTTPException(status_code=422, detail=(
+                    f"The nodes of this instance disagree on {label} ({spread}). One value is "
+                    f"stored on the VIP and re-rendered onto every member, so adopting would "
+                    f"impose one node's setting on the others. Align the files first."))
+
+        # The VRRP secret is stored per discovery as its own Fernet token, and Fernet is
+        # non-deterministic, so the ciphertexts cannot be compared — decrypt and compare the
+        # plaintexts. Nothing is logged. A node we cannot decrypt is treated as a mismatch rather
+        # than assumed equal.
+        secrets = {}
+        for p in participants:
+            enc = p.get("auth_pass_encrypted")
+            plain = decrypt_vrrp_secret(enc) if enc else None
+            if enc and not plain:
+                raise HTTPException(status_code=409, detail=(
+                    f"The VRRP secret reported by {p['agent_name']} cannot be decrypted "
+                    f"(encryption key changed?). Re-report or fix that node before adopting."))
+            secrets.setdefault(plain, []).append(p["agent_name"])
+        if len(secrets) > 1:
+            groups = "; ".join(
+                ("no password: " if k is None else "one password: ") + ", ".join(v)
+                for k, v in secrets.items())
+            raise HTTPException(status_code=422, detail=(
+                f"The nodes of this instance do not share one VRRP password ({groups}). "
+                f"Adoption stores a single secret and renders it onto every member, so it would "
+                f"silently change authentication on the others. Align the files first."))
+
+        # keepalived requires one MASTER and a matching advertisement interval across the group.
+        # Both are checked here so the operator learns at adoption time instead of at apply.
+        roles = [(p["candidate"].get("member") or {}).get("role") or "BACKUP" for p in participants]
+        if roles.count("MASTER") != 1:
+            listed = ", ".join(f"{p['agent_name']}={r}" for p, r in zip(participants, roles))
+            raise HTTPException(status_code=422, detail=(
+                f"VRID {vrid} is reported by {len(participants)} node(s) with "
+                f"{roles.count('MASTER')} MASTER ({listed}); exactly one must be MASTER. If a "
+                f"member is missing, install or enable its agent so it reports its keepalived.conf, "
+                f"then adopt again."))
+        adverts = {int((p["candidate"].get("vip") or {}).get("advert_int") or 1) for p in participants}
+        if len(adverts) > 1:
+            raise HTTPException(status_code=422, detail=(
+                f"The nodes disagree on advert_int ({sorted(adverts)}). keepalived needs the same "
+                f"advertisement interval across a VRRP group, so align the files first."))
+
+        # UNICAST SAFETY. Our renderer emits the unicast block only when it has peer addresses,
+        # so a peer that is not a member would be dropped and keepalived would fall back to
+        # multicast on this node while the real peer stays unicast — both would then claim the
+        # VIP. Refuse instead, naming the address that is unaccounted for.
+        declared_peers = {str(x) for p in participants for x in (p["candidate"].get("peers") or [])}
+        if declared_peers:
+            missing_ip = [p["agent_name"] for p in participants if not p["ip_address"]]
+            if missing_ip:
+                raise HTTPException(status_code=422, detail=(
+                    f"These nodes have not reported an IP address yet: {', '.join(missing_ip)}. "
+                    f"The unicast peer list cannot be verified until they do."))
+            member_ips = {str(p["ip_address"]) for p in participants}
+            unknown = sorted(declared_peers - member_ips)
+            if unknown:
+                raise HTTPException(status_code=422, detail=(
+                    f"This instance is unicast and lists peer(s) {', '.join(unknown)} that are not "
+                    f"among the nodes being adopted. Adopting would drop them from the peer list and "
+                    f"keepalived would silently fall back to multicast, so both sides could end up "
+                    f"holding the VIP. Register those nodes as agents so they report their config, "
+                    f"then adopt again."))
+
+        # One active VIP per agent — the delivery endpoint serves a single keepalived.conf per
+        # node, so a second active membership never converges. create/update enforce this via
+        # _validate_members_against_pool; adoption did not call it at all.
+        busy = await conn.fetchrow("""
+            SELECT a.name AS agent_name, v.name AS vip_name
+            FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id
+            JOIN agents a ON a.id = vm.agent_id
+            WHERE vm.agent_id = ANY($1::int[]) AND v.is_active = TRUE LIMIT 1
+        """, [p["agent_id"] for p in participants])
+        if busy:
+            raise HTTPException(status_code=409, detail=(
+                f"node '{busy['agent_name']}' is already a member of VIP '{busy['vip_name']}'"))
+
+        # Adoption must keep the VRRP identity it found. A different VRID would create a second
+        # VRRP domain on the wire, so a collision inside the pool is a hard conflict, never an
+        # auto-reallocation the way create_vip does it.
+        clash = await conn.fetchrow("""
+            SELECT id, name FROM vip_instances
+            WHERE pool_id = $1 AND is_active = TRUE AND virtual_router_id = $2
+        """, disc["pool_id"], int(vrid))
+        if clash:
+            raise HTTPException(status_code=409, detail=(
+                f"VRID {vrid} is already used by VIP '{clash['name']}' in this pool; the adopted "
+                f"config must keep its VRID, so resolve the collision first"))
+
+        async with conn.transaction():
+            vip_id = await conn.fetchval("""
+                INSERT INTO vip_instances
+                  (name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
+                   advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
+                   is_active, last_config_status, adopted_at, created_by)
+                VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',CURRENT_TIMESTAMP,$11)
+                RETURNING id
+            """, name, payload.get("description") or f"Adopted from {disc['agent_name']}",
+                disc["pool_id"], virtual_ip, int(vip_fields["prefix_length"]), int(vrid),
+                int(vip_fields.get("advert_int") or 1),
+                disc["auth_pass_encrypted"],           # already Fernet-encrypted at ingest
+                bool(vip_fields.get("use_unicast")), bool(vip_fields.get("track_haproxy")),
+                current_user["id"])
+            # Every node of the instance becomes a member with the role/priority/interface ITS
+            # OWN file declares, and each carries its own one-shot takeover authorisation pinned
+            # to the hash of the file we analysed on THAT node. The guard stays per-node: a file
+            # edited on one member between adoption and Apply is still refused there.
+            for p in participants:
+                pm = p["candidate"].get("member") or {}
+                await conn.execute("""
+                    INSERT INTO vip_members
+                        (vip_id, agent_id, network_interface, role, priority, takeover_expected_hash)
+                    VALUES ($1,$2,$3,$4,$5,$6)
+                """, vip_id, p["agent_id"], pm["network_interface"],
+                    pm.get("role") or "BACKUP", int(pm.get("priority") or 100),
+                    p["config_hash"])
+                await conn.execute(
+                    "UPDATE vip_discoveries SET adopted_vip_id = $2 WHERE agent_id = $1",
+                    p["agent_id"], vip_id)
+
+        await _stage_vip_version(conn, vip_id, "adopt", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="adopt", resource_type="vip",
+            resource_id=str(vip_id),
+            details={"name": name, "virtual_ip": virtual_ip, "vrid": vrid,
+                     "adopted_from_agent": disc["agent_name"],
+                     "members": [p["agent_name"] for p in participants],
+                     "accepted_data_loss": bool(payload.get("accept_data_loss"))},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+
+        node_names = [p["agent_name"] for p in participants]
+        return {
+            "id": vip_id,
+            "members": node_names,
+            "message": (
+                f"VIP adopted with {len(node_names)} member node(s): {', '.join(node_names)} "
+                f"(PENDING — review it in Apply Management, then apply to hand their "
+                f"keepalived.conf over to OpenManager)"),
+        }
+    finally:
+        await close_database_connection(conn)

backend/routers/waf.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Request, Header
+from fastapi import APIRouter, HTTPException, Request, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 import logging
 import time
@@ -182,7 +183,8 @@ async def get_waf_stats(cluster_id: Optional[int] = None, authorization: str = H
         logger.error(f"Error fetching WAF stats: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules")
+@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): WAF rule definitions aid bypass crafting
 async def get_waf_rules(cluster_id: Optional[int] = None):
     """
     # Get WAF Rules

backend/services/acme_diagnostics.py

@@ -8,9 +8,13 @@ suitable for an Antd Tabs/Steps display.
 Key constraints (Section 3.3 of the v1.5.0 plan):
 - DNS resolution uses stdlib socket.gethostbyname_ex via run_in_executor (we
   intentionally avoid pulling aiodns as a runtime dep for v1.5.0).
-- Port-80 probe is HEAD-only, target locked to the order's domains, success on
-  HTTP 200 OR 404, warns on egress timeout (don't fail-hard — corp egress
-  policies often blackhole outbound 80).
+- Port-80 probe is a GET (not HEAD) locked to the order's domains, because the
+  status code alone cannot tell a working challenge endpoint from a web UI: a
+  reverse proxy that has lost its /.well-known/acme-challenge/ location serves
+  its SPA with HTTP 200. The body's shape decides. Warns rather than fails on
+  egress timeout (corp egress policies often blackhole outbound 80) and on a
+  wrong responder (this probe sees the PUBLIC domain, not the challenge backend,
+  so it is evidence rather than a verdict).
 - All checks have hard wall-clock timeouts (asyncio.wait_for) to bound impact
   on the API event loop.
 - humanize_error_detail covers >= 11 RFC8555 problem types and is backwards
@@ -330,10 +334,41 @@ async def check_dns(domains: List[str]) -> Dict[str, Any]:
     )
 
 
+# Enough to classify a response without turning the diagnostic into a way to pull
+# arbitrary amounts of a third party's content into our JSON.
+_PROBE_BODY_LIMIT = 65536
+
+
+def _classify_probe_body(body: bytes, content_type: str) -> str:
+    """Coarse shape of a probe response: html | json | text | empty | binary.
+
+    The body itself is deliberately NOT retained anywhere — the shape is all that is
+    needed to tell "served me a web page" from "served me a token", and keeping the
+    bytes would open a new read surface onto whatever is behind the address.
+    """
+    if not body:
+        return "empty"
+    ct = (content_type or "").lower()
+    head = body[:512].lstrip().lower()
+    if ct.startswith("text/html") or head.startswith((b"<!doctype", b"<html")):
+        return "html"
+    if ct.startswith("application/json") or head[:1] in (b"{", b"["):
+        return "json"
+    try:
+        body.decode("utf-8")
+    except UnicodeDecodeError:
+        return "binary"
+    return "text"
+
+
 async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict[str, Any]:
-    """Probe HTTP-01 readiness on port 80 with a HEAD request to a synthetic
-    challenge URL. Success on 200 OR 404 (404 means the well-known path is
-    served but no challenge yet — fine).
+    """Probe HTTP-01 readiness on port 80 with a GET to a synthetic challenge URL.
+
+    404 means the path is served but no challenge is outstanding, which is fine. A
+    200 is only fine if the body is NOT a web page: a proxy that has lost its
+    /.well-known/acme-challenge/ route falls through to its catch-all and answers
+    200 with index.html, which a status-code-only check accepts as healthy while
+    every real validation fails.
 
     On egress timeout we WARN rather than FAIL because many corporate egress
     policies blackhole port 80 outbound; that does not impair LE's ingress
@@ -398,12 +433,84 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
                 continue
             url = f"http://{d}/.well-known/acme-challenge/diagnostic-probe"
             try:
-                async with session.head(url, allow_redirects=False) as resp:
-                    targets.append({
-                        "domain": d,
-                        "status": resp.status,
-                        "ok": resp.status in (200, 404),
-                    })
+                # v1.11.0: recorded as an outbound row so a failing port-80 probe is
+                # diagnosable after the fact, not only while the panel is open.
+                #
+                # MERGE NOTE: the feature branch instrumented a `session.head(...)`
+                # probe, which is what this function did when that branch was cut.
+                # It has since become a GET, because the status code alone cannot
+                # tell a working challenge endpoint from a SPA catch-all serving
+                # index.html with HTTP 200 (v1.10.x). Reverting to HEAD to gain the
+                # log row would put that bug straight back, so the GET probe below
+                # is the one that is wrapped.
+                #
+                # GET, not HEAD: the status code alone cannot tell a working challenge
+                # endpoint from a SPA. A reverse proxy that has lost its
+                # /.well-known/acme-challenge/ location falls through to its catch-all
+                # and serves index.html with HTTP 200 — which the old
+                # `status in (200, 404)` rule accepted as healthy while every real
+                # validation failed. Only the body distinguishes them.
+                from utils.http_instrumentation import outbound_span, TARGET_ACME_DIAG
+
+                async with outbound_span(
+                    target=TARGET_ACME_DIAG, method="GET", url=url, capture_body=False
+                ) as span:
+                    async with session.get(url, allow_redirects=False) as resp:
+                        # The body is EVIDENCE, not a precondition. If it cannot be read —
+                        # connection reset mid-response, a server that hangs after headers —
+                        # fall back to the status-only semantics this check has always had
+                        # rather than turning a healthy 404 into a hard failure. The stricter
+                        # rule below applies only when there is something to judge.
+                        try:
+                            body = await resp.content.read(_PROBE_BODY_LIMIT)
+                        except Exception:
+                            body = None
+                        content_type = (resp.headers.get("content-type") or "").split(";")[0].strip()
+                        body_class = (
+                            _classify_probe_body(body, content_type) if body is not None else "unread"
+                        )
+                        target = {
+                            "domain": d,
+                            "status": resp.status,
+                            "content_type": content_type or None,
+                            "body_len": len(body) if body is not None else None,
+                            "body_class": body_class,
+                        }
+                        # The VERDICT goes in the log row, not the body. `body` here is
+                        # up to 64 KB of a third party's page (`_PROBE_BODY_LIMIT`);
+                        # storing it would put an arbitrary remote document into the
+                        # audit table for every probed domain. The classification is
+                        # what an operator reads back anyway.
+                        span.set_response(
+                            resp.status,
+                            getattr(resp, "headers", None),
+                            {
+                                "probe": "acme-http01",
+                                "content_type": content_type or None,
+                                "body_len": len(body) if body is not None else None,
+                                "body_class": body_class,
+                            },
+                        )
+                        if resp.status == 200 and body_class == "html":
+                            # Reachable, wrong responder. Reported as a warning rather than
+                            # a failure: this check probes the PUBLIC domain and cannot see
+                            # the challenge backend, so it is evidence, not a verdict — and
+                            # a new `fail` here would block the site wizard on upgrade day
+                            # for every install.
+                            target["warn"] = True
+                            target["diagnosis"] = (
+                                "responded 200 with an HTML page, not a challenge token — "
+                                "the request is reaching a web UI instead of the ACME endpoint"
+                            )
+                        elif resp.status in (301, 302, 303, 307, 308):
+                            target["warn"] = True
+                            target["redirect_location"] = resp.headers.get("location")
+                            target["diagnosis"] = (
+                                "redirected instead of serving the challenge path"
+                            )
+                        else:
+                            target["ok"] = resp.status in (200, 404)
+                        targets.append(target)
             except asyncio.TimeoutError:
                 targets.append({"domain": d, "error": "egress timeout", "warn": True})
                 skip_reason = "egress timeout"
@@ -425,7 +532,11 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
             details={"targets": targets},
             duration_ms=duration_ms,
         )
-    if warns and not [t for t in targets if t.get("ok")]:
+    # Surface warnings even when OTHER domains answered correctly. The old condition
+    # ("warn only if nothing succeeded") hid the single most diagnostic outcome there
+    # is: a multi-domain certificate where one name reaches a web UI instead of the
+    # challenge endpoint reported a clean pass.
+    if warns:
         # R18b audit fix (round 7): branch the rollup message on the
         # actual cause. Pre-fix the message was always "Egress to
         # port 80 appears blocked" — even when every target was
@@ -435,6 +546,36 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
         # corporate firewall logs while the real cause was an
         # internal-only DNS A record. Also harden against
         # `skip_reason=None` so the message never reads "(None)".
+        # Wrong-responder warnings take priority over every other cause: they are the
+        # only ones that mean "your server answered, and answered wrong", which is a
+        # different problem from "we could not test".
+        wrong_responder = [t for t in targets if t.get("diagnosis")]
+        if wrong_responder:
+            first = wrong_responder[0]
+            if first.get("body_class") == "html":
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} with an HTML "
+                    f"page ({first.get('content_type') or 'unknown type'}, "
+                    f"{first.get('body_len')} bytes) instead of a challenge token. The "
+                    "path is reaching a web interface, not the ACME endpoint — check "
+                    "that the reverse proxy in front of OpenManager routes "
+                    "/.well-known/acme-challenge/ to the API."
+                )
+            else:
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} "
+                    f"({first.get('diagnosis')})"
+                )
+            return _check_result(
+                "port80",
+                "Port 80 reachability",
+                "warn",
+                human,
+                severity="warn",
+                details={"targets": targets},
+                duration_ms=duration_ms,
+            )
+
         ssrf_skip = any(
             "non-public" in (t.get("skip") or "")
             or "SSRF" in (t.get("skip") or "")
@@ -491,11 +632,20 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             duration_ms=int((time.time() - started) * 1000),
         )
 
+    # The WHERE clause is deliberately identical to the pre-existing one, so `not rows`
+    # still means exactly what it meant before and the `fail` branch below cannot fire
+    # in any situation where it previously passed. Narrowing it here (e.g. by adding a
+    # mode filter) would turn a tcp-only port-80 cluster from "ok" into "fail", and the
+    # site wizard blocks submit on any failing check — locking those installs the day
+    # this ships. Mode is examined afterwards, in Python, and only ever downgrades to
+    # `warn`.
     rows = await conn.fetch(
         """
-        SELECT id, name, bind_address, bind_port, mode, default_backend
-        FROM frontends
-        WHERE cluster_id = ANY($1::int[]) AND is_active = TRUE AND bind_port = 80
+        SELECT f.id, f.name, f.bind_address, f.bind_port, f.mode, f.default_backend,
+               f.cluster_id, c.acme_enabled
+        FROM frontends f
+        JOIN haproxy_clusters c ON c.id = f.cluster_id
+        WHERE f.cluster_id = ANY($1::int[]) AND f.is_active = TRUE AND f.bind_port = 80
         """,
         cluster_ids,
     )
@@ -510,13 +660,144 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             details={"cluster_ids": cluster_ids},
             duration_ms=duration_ms,
         )
+
+    # Character-for-character the renderer's normalisation (services/haproxy_config.py),
+    # so this can never disagree with what actually gets emitted.
+    http_rows = [r for r in rows if (r["mode"] or "http").strip().lower() == "http"]
+    if not http_rows:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                "The only port-80 frontend(s) in the target cluster(s) are in tcp mode. "
+                "A tcp-mode frontend cannot carry the /.well-known/acme-challenge/ ACL, "
+                "so HTTP-01 cannot be served — use DNS-01, or add an http-mode frontend "
+                "on port 80."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows]},
+            duration_ms=duration_ms,
+        )
+    rows = http_rows
+
+    # A frontend row proves only that the DATABASE describes port-80 routing. The
+    # renderer gates the challenge ACL on `acme_enabled`, and the nodes run whatever
+    # config was last APPLIED — so the row said "ok" during an incident where the
+    # live config had no usable challenge route at all. Check the two things the row
+    # cannot tell us. Both report `warn`, never `fail`: the site wizard blocks submit
+    # on any `fail`, so a new failing condition would lock every install on the day
+    # it ships.
+    # `.get()` rather than `[]`: the column gates a WARNING, so a row shape without
+    # it should not blow up the whole diagnostic. Absent means "assume enabled" —
+    # the applied-config check below is the authoritative one either way.
+    acme_off = sorted({r["cluster_id"] for r in rows if not r.get("acme_enabled", True)})
+    if acme_off:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {acme_off} have ACME Challenge Routing disabled, so the "
+                "generated config contains no /.well-known/acme-challenge/ route. "
+                "Enable it in Cluster Management, then apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "acme_disabled_clusters": acme_off},
+            duration_ms=duration_ms,
+        )
+
+    missing_in_applied = []
+    challenge_backends = {}
+    for cluster_id in sorted({r["cluster_id"] for r in rows}):
+        # Selector matched to the one the AGENT uses to fetch its config
+        # (routers/agent.py: status='APPLIED' AND is_active=TRUE), because the question
+        # here is "what are the nodes running right now?". Without `is_active` this can
+        # read a superseded row and report on a config that was never delivered.
+        # Extracting in SQL rather than pulling whole configs back per cluster: these
+        # files run to hundreds of KB on real installs.
+        applied = await conn.fetchrow(
+            """
+            SELECT position('use_backend _acme_challenge_backend' in config_content) > 0
+                       AS has_route,
+                   substring(config_content from 'server _acme_mgmt [^\\n]*') AS server_line
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+              AND config_content IS NOT NULL
+            ORDER BY created_at DESC LIMIT 1
+            """,
+            cluster_id,
+        )
+        if not applied or not applied["has_route"]:
+            missing_in_applied.append(cluster_id)
+            continue
+        server_line = (applied["server_line"] or "").strip()
+        challenge_backends[cluster_id] = (
+            server_line[len("server _acme_mgmt "):].strip() if server_line else None
+        )
+
+    if missing_in_applied:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {missing_in_applied} have no applied configuration carrying "
+                "the challenge route. The change exists in the database but the nodes are "
+                "still running an older config — apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_not_applied": missing_in_applied},
+            duration_ms=duration_ms,
+        )
+
+    # A backend section with no `server` line: the route exists, `haproxy -c` passes,
+    # and every challenge request gets a 503 from an empty backend. Without this branch
+    # the falsy target slips past the loopback filter below and the check reports "ok".
+    serverless = sorted(cid for cid, target in challenge_backends.items() if not target)
+    if serverless:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {serverless} route the challenge path to a backend that has "
+                "no server line, so every request returns 503. The configured ACME "
+                "Challenge Backend URL could not be resolved into an address — check it "
+                "in Cluster Management, or Settings > ACME for the global value."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_without_server": serverless},
+            duration_ms=duration_ms,
+        )
+
+    loopback = {
+        cid: target for cid, target in challenge_backends.items()
+        if target and target.split(":")[0].strip("[]").lower()
+        in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+    }
+    if loopback:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"The applied config points the challenge backend at {sorted(loopback.values())}. "
+                "HAProxy resolves that on the HAProxy node, so it means the node itself, not "
+                "this management server. Set ACME Challenge Backend URL to a routable address."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
+            duration_ms=duration_ms,
+        )
+
     return _check_result(
         "routing",
         "HAProxy routing",
         "ok",
-        f"Found {len(rows)} HTTP frontend(s) on port 80",
+        f"Found {len(rows)} HTTP frontend(s) on port 80; challenge route present in applied config",
         severity="info",
-        details={"frontends": [dict(r) for r in rows]},
+        details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
         duration_ms=duration_ms,
     )
 

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -65,26 +69,61 @@ class ACMEService:
             if cached.get('_fetched_at', 0) > time.time() - 3600:
                 return cached
 
-        async with aiohttp.ClientSession() as session:
-            async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
-                if resp.status != 200:
-                    raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
-                data = await resp.json()
-                if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
-                data['_fetched_at'] = time.time()
-                self._directory_cache[directory_url] = data
-                return data
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): directory_url can come from a stored
+        # account row; validate it (https + public IP, no redirects) before the
+        # server-side fetch so it cannot be pointed at internal/metadata targets.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(directory_url)
+
+        # v1.11.0: recorded in request_logs as an outbound call so an operator
+        # can see exactly which CA was contacted and what it answered.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(target=TARGET_ACME, method="GET", url=directory_url) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    if resp.status != 200:
+                        span.set_response(resp.status, dict(resp.headers))
+                        raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
+                    data = await resp.json()
+                    span.set_response(resp.status, dict(resp.headers), data)
+                    if 'Replay-Nonce' in resp.headers:
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
+                    data['_fetched_at'] = time.time()
+                    self._directory_cache[directory_url] = data
+                    return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
-            nonce = self._nonce
-            self._nonce = None
-            return nonce
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         directory = await self.get_directory(directory_url)
-        async with aiohttp.ClientSession() as session:
-            async with session.head(directory['newNonce']) as resp:
-                return resp.headers['Replay-Nonce']
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): newNonce is taken from the (attacker-
+        # influenceable) directory JSON and is fetched here BEFORE the guarded
+        # _signed_request POST, so it must be guarded too — otherwise a directory
+        # that returns an internal newNonce (and omits Replay-Nonce) is a live SSRF.
+        # https + public IP only, IPv4-pinned connector, no redirects, bounded timeout.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        nonce_url = directory['newNonce']
+        await assert_public_url(nonce_url)
+
+        # v1.11.0: a HEAD with no body and no status check — capture the status
+        # and the allowlisted headers only. `Replay-Nonce` itself is redacted by
+        # the header rules: it is a single-use credential.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(
+            target=TARGET_ACME, method="HEAD", url=nonce_url, capture_body=False
+        ) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.head(nonce_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    span.set_response(resp.status, dict(resp.headers))
+                    return resp.headers['Replay-Nonce']
 
     def _generate_account_key(self) -> Tuple[str, dict]:
         private_key = rsa.generate_private_key(
@@ -179,39 +218,82 @@ class ACMEService:
 
         body = self._sign_jws(private_key, protected, payload)
 
-        async with aiohttp.ClientSession() as session:
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): `url` is taken from the CA directory /
+        # order responses. The directory is already fetched from a validated
+        # public CA, but guard the follow-up POST target too (defence in depth)
+        # so a tampered/malicious directory cannot steer the request internally.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(url)
+
+        # v1.11.0: instrument each ATTEMPT separately (the span goes inside the
+        # retry loop, the session stays outside it) so a badNonce retry shows up
+        # as its own row instead of being folded into the successful one.
+        #
+        # capture_body=False is mandatory here. The JWS body is
+        # {protected, payload, signature}: `protected` carries the nonce and the
+        # account kid/jwk, and `signature` is made with the account private key.
+        # The key itself never crosses the wire, but a stored (protected,
+        # signature) pair is a REPLAYABLE ACME credential for the lifetime of the
+        # nonce. We log a description of the request instead of the request.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
             for attempt in range(3):
-                async with session.post(
-                    url,
-                    json=body,
-                    headers={"Content-Type": "application/jose+json"},
-                    timeout=aiohttp.ClientTimeout(total=30),
-                ) as resp:
-                    if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                jws_summary = {
+                    "jws": True,
+                    "acme_url": protected.get("url"),
+                    "kid_present": bool(protected.get("kid")),
+                    "jwk_present": bool(protected.get("jwk")),
+                    "payload_empty": payload == "",
+                    "attempt": attempt + 1,
+                }
+                async with outbound_span(
+                    target=TARGET_ACME,
+                    method="POST",
+                    url=url,
+                    request_body=jws_summary,
+                    capture_body=False,
+                ) as span:
+                    async with session.post(
+                        url,
+                        json=body,
+                        headers={"Content-Type": "application/jose+json"},
+                        timeout=aiohttp.ClientTimeout(total=30),
+                        allow_redirects=False,
+                    ) as resp:
+                        if 'Replay-Nonce' in resp.headers:
+                            self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
-                        err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
-                            nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
-                            protected['nonce'] = nonce
-                            body = self._sign_jws(private_key, protected, payload)
-                            continue
+                        if resp.status == 400 and attempt < 2:
+                            err = await resp.json()
+                            etype = (err.get('type') or '')
+                            edetail = (err.get('detail') or '').lower()
+                            # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                            # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                            # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                            # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                            if etype.endswith('badNonce') or 'nonce' in edetail:
+                                span.set_response(resp.status, dict(resp.headers), err)
+                                nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
+                                protected['nonce'] = nonce
+                                body = self._sign_jws(private_key, protected, payload)
+                                continue
 
-                    resp_data = {}
-                    content_type = resp.headers.get('Content-Type', '')
-                    if 'json' in content_type:
-                        resp_data = await resp.json()
-                    elif resp.status < 300:
-                        text = await resp.text()
-                        if text:
-                            try:
-                                resp_data = json.loads(text)
-                            except json.JSONDecodeError:
-                                resp_data = {"raw": text}
+                        resp_data = {}
+                        content_type = resp.headers.get('Content-Type', '')
+                        if 'json' in content_type:
+                            resp_data = await resp.json()
+                        elif resp.status < 300:
+                            text = await resp.text()
+                            if text:
+                                try:
+                                    resp_data = json.loads(text)
+                                except json.JSONDecodeError:
+                                    resp_data = {"raw": text}
 
-                    headers = dict(resp.headers)
-                    return resp.status, resp_data, headers
+                        headers = dict(resp.headers)
+                        span.set_response(resp.status, headers, resp_data)
+                        return resp.status, resp_data, headers
 
         raise Exception(f"ACME request to {url} failed after retries")
 

backend/services/csr_service.py

@@ -0,0 +1,461 @@
+"""
+csr_service: CSR (Certificate Signing Request) generation + signed-certificate
+import (v1.9.0).
+
+Flow:
+  1. `generate_csr_bundle` builds a private key + CSR locally (pure crypto,
+     no DB/IO — callers MUST run it via `asyncio.to_thread`: RSA-4096
+     generation takes seconds and would stall the single-worker event loop).
+  2. The bundle is persisted to `ssl_csrs` (`insert_csr_row`); the operator
+     downloads the CSR PEM and has it signed by an external CA.
+  3. `import_signed_certificate` pairs the CA response with the stored key,
+     creates a normal `ssl_certificates` row (source='csr',
+     last_config_status='PENDING' — agents never see it before Apply) and
+     NULLs the key copy on the CSR row.
+
+The CSR builder generalises the in-repo ACME reference
+(services/acme_service.py finalize_order): PEM output instead of DER, full
+subject instead of CN-only, ECDSA support, same PKCS8/NoEncryption key
+serialisation (the agent concatenates cert+key+chain into one PEM and HAProxy
+cannot read passphrase-protected keys).
+
+Private keys are ENCRYPTED AT REST from v1.10.1 (Issue #53): the Fernet token
+replaces the PEM in the same `ssl_csrs.private_key_pem` column, so there is no
+schema change and no SCHEMA_VERSION bump. Rows written earlier hold a raw PEM
+and are still read transparently — see utils/csr_key_crypto.py for the format
+discriminator and the key-rotation caveat. The pending CSR key is the one key
+in the system worth encrypting: it sits idle for the whole signing window and
+is never transmitted, unlike ssl_certificates.private_key_content and the ACME
+order keys, which agents must receive in plaintext on every poll.
+The key is NEVER returned by any CSR API response — `csr_row_to_dict` strips
+it unconditionally.
+"""
+
+import json
+import logging
+from typing import Any, Dict, List, Optional
+from types import SimpleNamespace
+
+import asyncpg
+from fastapi import HTTPException
+
+from cryptography import x509
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import NameOID
+
+from services import ssl_service
+from utils.csr_key_crypto import decrypt_csr_private_key, encrypt_csr_private_key
+
+logger = logging.getLogger(__name__)
+
+
+_KEY_FACTORIES = {
+    'rsa-2048': lambda: rsa.generate_private_key(public_exponent=65537, key_size=2048),
+    'rsa-4096': lambda: rsa.generate_private_key(public_exponent=65537, key_size=4096),
+    'ecdsa-p256': lambda: ec.generate_private_key(ec.SECP256R1()),
+    'ecdsa-p384': lambda: ec.generate_private_key(ec.SECP384R1()),
+}
+
+# (payload attribute, x509 OID, subject-JSON key)
+_SUBJECT_OID_MAP = [
+    ('organization', NameOID.ORGANIZATION_NAME, 'O'),
+    ('organizational_unit', NameOID.ORGANIZATIONAL_UNIT_NAME, 'OU'),
+    ('locality', NameOID.LOCALITY_NAME, 'L'),
+    ('state', NameOID.STATE_OR_PROVINCE_NAME, 'ST'),
+    ('country', NameOID.COUNTRY_NAME, 'C'),
+    ('email', NameOID.EMAIL_ADDRESS, 'emailAddress'),
+]
+
+
+def generate_csr_bundle(payload: Any) -> Dict[str, Any]:
+    """Generate a private key + CSR for a validated SSLCSRCreate payload.
+
+    Pure CPU-bound crypto — no DB, no network. Callers must offload via
+    `asyncio.to_thread` (see module docstring).
+
+    Returns {'csr_pem', 'private_key_pem', 'sans', 'subject'}.
+    """
+    key = _KEY_FACTORIES[payload.key_algorithm]()
+
+    attrs = [x509.NameAttribute(NameOID.COMMON_NAME, payload.common_name)]
+    subject_json: Dict[str, str] = {}
+    for attr_name, oid, json_key in _SUBJECT_OID_MAP:
+        value = getattr(payload, attr_name, None)
+        if value and str(value).strip():
+            cleaned = str(value).strip()
+            attrs.append(x509.NameAttribute(oid, cleaned))
+            subject_json[json_key] = cleaned
+
+    # CN always first in the SAN list, then the extra names, deduped with
+    # order preserved (mirrors the ACME flow where domains[0] is the CN).
+    sans = list(dict.fromkeys([payload.common_name, *(payload.sans or [])]))
+
+    builder = (
+        x509.CertificateSigningRequestBuilder()
+        .subject_name(x509.Name(attrs))
+        .add_extension(
+            x509.SubjectAlternativeName([x509.DNSName(d) for d in sans]),
+            critical=False,
+        )
+    )
+    csr = builder.sign(key, hashes.SHA256())
+
+    return {
+        'csr_pem': csr.public_bytes(serialization.Encoding.PEM).decode('utf-8'),
+        'private_key_pem': key.private_bytes(
+            serialization.Encoding.PEM,
+            serialization.PrivateFormat.PKCS8,
+            serialization.NoEncryption(),
+        ).decode('utf-8'),
+        'sans': sans,
+        'subject': subject_json,
+    }
+
+
+def diff_domains(csr_sans: Optional[List[str]], cert_domains: Optional[List[str]]) -> List[str]:
+    """Human-readable warnings for SAN drift between the CSR and the signed
+    certificate (case-insensitive set diff). CAs legitimately add/normalise
+    SANs, so drift is WARN-only — the hard gate is the key match."""
+    csr_set = {d.lower() for d in (csr_sans or []) if d}
+    cert_set = {d.lower() for d in (cert_domains or []) if d}
+    warnings: List[str] = []
+    added = sorted(cert_set - csr_set)
+    dropped = sorted(csr_set - cert_set)
+    if added:
+        warnings.append(
+            f"The CA added domains that were not in the CSR: {', '.join(added)}"
+        )
+    if dropped:
+        warnings.append(
+            f"The CA dropped domains that were requested in the CSR: {', '.join(dropped)}"
+        )
+    return warnings
+
+
+def _maybe_json_list(value: Any) -> List[str]:
+    """asyncpg returns JSONB columns as str unless a codec is registered."""
+    if isinstance(value, str):
+        try:
+            parsed = json.loads(value)
+            return parsed if isinstance(parsed, list) else []
+        except Exception:
+            return []
+    return list(value) if value else []
+
+
+def csr_row_to_dict(row: Any, include_pem: bool = False) -> Dict[str, Any]:
+    """Row → API dict. ALWAYS strips private_key_pem — the key never leaves
+    the server via a CSR endpoint. csr_pem included only on demand
+    (detail/create responses, not lists)."""
+    d = dict(row)
+    d.pop('private_key_pem', None)
+    if not include_pem:
+        d.pop('csr_pem', None)
+    for key in ('subject', 'sans'):
+        if key in d and isinstance(d[key], str):
+            try:
+                d[key] = json.loads(d[key])
+            except Exception:
+                pass
+    return d
+
+
+async def assert_csr_name_available(conn, name: str) -> None:
+    """Reject a CSR name that is already taken by an ACTIVE certificate or
+    another PENDING CSR. Called BEFORE key generation (cheap fail-fast) and
+    re-run inside `insert_csr_row` (the unique index closes the race)."""
+    existing_cert = await conn.fetchval(
+        "SELECT id FROM ssl_certificates WHERE name = $1 AND is_active = TRUE",
+        name,
+    )
+    if existing_cert:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"An active SSL certificate named '{name}' already exists. "
+                "The CSR name becomes the certificate name at import — choose "
+                "a different name or remove the existing certificate first."
+            ),
+        )
+    existing_csr = await conn.fetchval(
+        "SELECT id FROM ssl_csrs WHERE name = $1 AND status = 'pending'",
+        name,
+    )
+    if existing_csr:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{name}' already exists (id={existing_csr}). "
+                "Import or delete it first, or choose a different name."
+            ),
+        )
+
+
+async def insert_csr_row(conn, payload: Any, bundle: Dict[str, Any], user_id: Optional[int]) -> int:
+    """Persist a freshly generated CSR bundle. Returns the new csr id.
+
+    Issue #53 (v1.10.1): the private key is Fernet-encrypted before it is stored. The token goes
+    into the SAME private_key_pem TEXT column — no schema change — and is only ever decrypted
+    in-process by import_signed_certificate. No CSR endpoint returns the column either way.
+    """
+    await assert_csr_name_available(conn, payload.name)
+    stored_key = encrypt_csr_private_key(bundle['private_key_pem'])
+    try:
+        csr_id = await conn.fetchval(
+            """
+            INSERT INTO ssl_csrs
+            (name, common_name, subject, sans, key_algorithm, csr_pem,
+             private_key_pem, status, created_by)
+            VALUES ($1, $2, $3::jsonb, $4::jsonb, $5, $6, $7, 'pending', $8)
+            RETURNING id
+            """,
+            payload.name,
+            payload.common_name,
+            json.dumps(bundle['subject']),
+            json.dumps(bundle['sans']),
+            payload.key_algorithm,
+            bundle['csr_pem'],
+            stored_key,
+            user_id,
+        )
+    except asyncpg.exceptions.UniqueViolationError:
+        # uq_ssl_csrs_name_pending — a concurrent request won the name.
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{payload.name}' was just created by a "
+                "concurrent request — choose a different name."
+            ),
+        )
+    return csr_id
+
+
+async def import_signed_certificate(conn, csr_id: int, imp: Any, user_id: Optional[int]) -> Dict[str, Any]:
+    """Pair the CA-signed certificate with the stored CSR key and create the
+    ssl_certificates row. Atomic: cert row + CSR state change commit together.
+
+    Returns {'certificate_id', 'certificate_name', 'primary_domain',
+    'warnings', 'reactivated'}. Raises HTTPException on every failure
+    (404 missing, 409 already completed, 400 validation).
+    """
+    async with conn.transaction():
+        # Row lock serialises concurrent imports AND a concurrent DELETE of
+        # the same CSR; works across multiple uvicorn workers (DB-level lock).
+        row = await conn.fetchrow(
+            "SELECT * FROM ssl_csrs WHERE id = $1 FOR UPDATE", csr_id
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        if row['status'] == 'completed':
+            raise HTTPException(
+                status_code=409,
+                detail=(
+                    f"CSR '{row['name']}' is already completed — certificate "
+                    f"id {row['ssl_certificate_id']} was imported from it. "
+                    "Create a new CSR to reissue."
+                ),
+            )
+        if not row['private_key_pem']:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Stored CSR private key is missing — the CSR row is "
+                    "corrupt. Delete it and create a new CSR."
+                ),
+            )
+        # Issue #53: the column holds a Fernet token from v1.10.1 on, and a raw PEM for rows
+        # written before it. decrypt_csr_private_key accepts both, so no data migration is
+        # needed. A None here means the token cannot be decrypted — SECRET_KEY was rotated
+        # without CSR_ENCRYPTION_KEY set. Fail loudly: the key is gone, so the CA's certificate
+        # can never be paired with it, and silently falling through would surface as the far
+        # more confusing "certificate does not match this CSR's private key".
+        stored_key = decrypt_csr_private_key(row['private_key_pem'])
+        if not stored_key:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    f"The stored private key for CSR '{row['name']}' cannot be decrypted. This "
+                    "happens when SECRET_KEY was rotated while CSR_ENCRYPTION_KEY was not set. "
+                    "The key is unrecoverable, so this CSR can no longer be completed — delete "
+                    "it and create a new one (then have the new CSR signed)."
+                ),
+            )
+
+        effective_name = getattr(imp, 'name', None) or row['name']
+
+        # Parse the pasted certificate FIRST so a malformed/truncated CA
+        # response gets the manual flow's 400, not a 500 from the key-match
+        # step below (verify_certificate_key_match reports an unparseable
+        # cert as match=None, which we treat as an integrity failure).
+        from utils.ssl_parser import parse_ssl_certificate, verify_certificate_key_match
+        precheck = parse_ssl_certificate(imp.certificate_content)
+        if precheck.get('error'):
+            raise HTTPException(
+                status_code=400,
+                detail=f"Invalid SSL certificate: {precheck['error']}",
+            )
+
+        # THE defining check of this feature: the CA response must match the
+        # key we generated. Deliberately stricter than create_cert_row's
+        # lenient fallback — we generated this key ourselves, so an
+        # unverifiable pair is an integrity failure, not operator input.
+        match_result = verify_certificate_key_match(imp.certificate_content, stored_key)
+        if match_result.get('match') is False:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    "The signed certificate does not match this CSR's private "
+                    "key — the CA response likely belongs to a different "
+                    "CSR/key. Verify you pasted the certificate that was "
+                    "issued for this exact CSR."
+                ),
+            )
+        if match_result.get('match') is not True:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Could not verify the certificate/key pair: "
+                    f"{match_result.get('reason', 'unknown')}"
+                ),
+            )
+
+        # Full parse/validation pipeline shared with the manual + wizard
+        # flows: invalid PEM, bad chain and already-expired certs all 400.
+        payload = SimpleNamespace(
+            name=effective_name,
+            certificate_content=imp.certificate_content,
+            private_key_content=stored_key,
+            chain_content=getattr(imp, 'chain_content', None),
+            usage_type=getattr(imp, 'usage_type', 'frontend') or 'frontend',
+        )
+        fields = ssl_service._prepare_cert_fields(payload)
+
+        # Global name uniqueness (ssl_certificates.cluster_id is always NULL
+        # under the R38 schema, so name is effectively a global namespace).
+        existing = await conn.fetchrow(
+            "SELECT id, is_active FROM ssl_certificates WHERE name = $1 LIMIT 1",
+            effective_name,
+        )
+        if existing and existing['is_active']:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    f"An active SSL certificate named '{effective_name}' "
+                    "already exists (created after this CSR). Delete or "
+                    "rename it, or pass a different `name` in the import "
+                    "request — the CSR stays pending and can be re-imported."
+                ),
+            )
+
+        reactivated = False
+        if existing and not existing['is_active']:
+            # Reactivate the soft-deleted row (mirrors create_cert_row):
+            # preserves the row id so historical references keep working.
+            await conn.execute(
+                "DELETE FROM ssl_certificate_clusters WHERE ssl_certificate_id = $1",
+                existing['id'],
+            )
+            await conn.execute(
+                """
+                UPDATE ssl_certificates
+                SET is_active = TRUE,
+                    last_config_status = 'PENDING',
+                    certificate_content = $2,
+                    private_key_content = $3,
+                    chain_content = $4,
+                    primary_domain = $5,
+                    all_domains = $6::jsonb,
+                    expiry_date = $7,
+                    usage_type = $8,
+                    issuer = $9,
+                    fingerprint = $10,
+                    status = $11,
+                    days_until_expiry = $12,
+                    source = 'csr',
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = $1
+                """,
+                existing['id'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['primary_domain'],
+                json.dumps(fields['all_domains']),
+                fields['expiry_date'],
+                fields['usage_type'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+            )
+            cert_id = existing['id']
+            reactivated = True
+            logger.info(
+                f"csr_service.import_signed_certificate: reactivated "
+                f"soft-deleted cert '{effective_name}' (id={cert_id}) for CSR {csr_id}"
+            )
+        else:
+            cert_id = await conn.fetchval(
+                """
+                INSERT INTO ssl_certificates (
+                    name, primary_domain, certificate_content, private_key_content,
+                    chain_content, expiry_date, issuer, fingerprint, status,
+                    days_until_expiry, all_domains, is_active, cluster_id,
+                    last_config_status, usage_type, source
+                ) VALUES (
+                    $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb,
+                    TRUE, NULL, 'PENDING', $12, 'csr'
+                )
+                RETURNING id
+                """,
+                effective_name,
+                fields['primary_domain'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['expiry_date'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+                json.dumps(fields['all_domains']),
+                fields['usage_type'],
+            )
+
+        # Cluster bindings: global = zero junction rows (existing convention).
+        if not getattr(imp, 'is_global', False):
+            for cluster_id in (getattr(imp, 'cluster_ids', None) or []):
+                await ssl_service.ensure_cluster_junction(conn, cert_id, cluster_id)
+
+        # Complete the CSR and destroy the key copy — the key now lives on
+        # the certificate row only, like every other key in the system.
+        await conn.execute(
+            """
+            UPDATE ssl_csrs
+            SET status = 'completed',
+                ssl_certificate_id = $2,
+                private_key_pem = NULL,
+                completed_at = CURRENT_TIMESTAMP,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $1
+            """,
+            csr_id,
+            cert_id,
+        )
+
+    warnings = diff_domains(_maybe_json_list(row['sans']), fields['all_domains'])
+    if reactivated:
+        warnings.append(
+            f"A soft-deleted certificate named '{effective_name}' was "
+            f"reactivated (row id {cert_id}) — existing entities that still "
+            "reference that id now serve the newly imported certificate."
+        )
+
+    return {
+        'certificate_id': cert_id,
+        'certificate_name': effective_name,
+        'primary_domain': fields['primary_domain'],
+        'warnings': warnings,
+        'reactivated': reactivated,
+    }

backend/services/dns_providers/__init__.py

@@ -5,8 +5,8 @@ A small adapter layer so DNS-01 challenges can publish/clean up the
 additive at the RRset level (add/remove a single value by name+content, never
 overwrite-by-name) so multiple coexisting values at one name (wildcard + apex) work.
 
-MVP providers: manual (user publishes the TXT themselves) and Cloudflare. New providers
-plug in via the registry without touching the orchestration.
+Providers: manual (user publishes the TXT themselves), Cloudflare, and GoDaddy (v1.10.0). New
+providers plug in via the registry without touching the orchestration.
 """
 from .base import DnsProvider, DnsProviderError
 from .registry import get_provider, list_providers, is_supported

backend/services/dns_providers/cloudflare.py

@@ -73,26 +73,42 @@ class CloudflareDNSProvider(DnsProvider):
         """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
         DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
         url = f"{CLOUDFLARE_API_BASE}{path}"
+        # v1.11.0: single funnel for every Cloudflare call, so instrumenting here
+        # covers all five logical endpoints. `safe_error_only=True` records only
+        # the exception TYPE — the same stance the handlers below already take,
+        # because a raw message can carry the request URL and through it the zone
+        # identifier. The Authorization header is dropped to a presence marker by
+        # the header allowlist.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_CLOUDFLARE
+
         try:
-            async with session.request(
-                method, url, headers=self._headers(), allow_redirects=False, **kwargs
-            ) as resp:
-                try:
-                    body = await resp.json()
-                except Exception:  # noqa: BLE001
-                    body = {}
-                if resp.status in (401, 403):
-                    raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
-                if resp.status >= 400 or not body.get("success", False):
-                    # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
-                    # human message text, never the request (which carries the token header).
-                    msgs = "; ".join(
-                        str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
-                    )
-                    raise DnsProviderError(
-                        f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
-                    )
-                return body
+            async with outbound_span(
+                target=TARGET_DNS_CLOUDFLARE,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        body = await resp.json()
+                    except Exception:  # noqa: BLE001
+                        body = {}
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    if resp.status in (401, 403):
+                        raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                    if resp.status >= 400 or not body.get("success", False):
+                        # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                        # human message text, never the request (which carries the token header).
+                        msgs = "; ".join(
+                            str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                        )
+                        raise DnsProviderError(
+                            f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                        )
+                    return body
         except DnsProviderError:
             raise
         except aiohttp.ClientError as exc:

backend/services/dns_providers/godaddy.py

@@ -0,0 +1,527 @@
+"""GoDaddy DNS provider for ACME DNS-01 (Issue #35 follow-up, v1.10.0).
+
+Uses the GoDaddy Domains API v1 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name selects which zone is touched), which is the same reason cloudflare.py is exempt from
+utils/ssrf_guard.py. Every failure is wrapped in DnsProviderError with a SANITIZED message: the
+API Key and Secret are scrubbed out of any text that could reach a log, an order event, or
+letsencrypt_orders.error_detail.
+
+Two GoDaddy-specific hazards drive the shape of this module — neither exists on Cloudflare:
+
+1. NO PER-VALUE WRITE. `PUT /v1/domains/{d}/records/TXT/{name}` REPLACES the entire RRset at that
+   type+name; it does not merge. A certificate for `example.com` + `*.example.com` publishes two
+   DIFFERENT TXT values at the SAME name `_acme-challenge.example.com` (base.py's additive
+   contract), so a naive single-value PUT would silently destroy the sibling and fail the wildcard
+   authorization. Every mutation here is therefore read-modify-write: GET the current RRset, merge,
+   PUT the whole list back. An EMPTY array is rejected (422 INVALID_BODY, "Records must be
+   specified"), so removing the LAST value must use DELETE — never `PUT []`.
+
+2. ZONE-DESTRUCTIVE SIBLING PATHS. `PUT /v1/domains/{d}/records/TXT` (three segments, no name)
+   wipes EVERY TXT in the zone — SPF, DKIM, DMARC, Microsoft/Google verification — and
+   `PUT /v1/domains/{d}/records` wipes the whole zone (this is dehydrated issue #430 verbatim).
+   The record path is built only by _rrset_path(), which refuses an empty zone or relative name so
+   a URL can never collapse onto one of those endpoints.
+
+Concurrency: v1 has no ETag, no If-Match and no per-record id, so read-modify-write can lose an
+update if two mutations at one name overlap. Today they cannot: orders are advanced sequentially
+(`for oid in claimed_ids: await advance_dns01_order(oid)` in main.py) and an order's challenges are
+published sequentially (`for ch in challenges: await provider.add_txt_record(...)` in
+dns01_orchestrator.py), so the apex+wildcard pair is strictly ordered and the second publish sees
+the first. _rrset_lock() makes that safety structural rather than incidental. Across REPLICAS the
+window is real but narrow (two orders publishing at the same record name in overlapping cycles) and
+self-healing: a lost publish ends `invalid` and the bounded retry chain mints a fresh order, a lost
+cleanup is retried by the reconcile sweep, and an orphaned `_acme-challenge` TXT is inert. The real
+fix is the v3 API (POST + DELETE by recordId, natively per-value), which is PAT-only and a
+follow-up; it is deliberately not used here because v1 + sso-key is what operators can use today.
+
+Credentials: an API Key + Secret pair from https://developer.godaddy.com/keys. It must be a
+PRODUCTION key — the first key the dashboard issues is an OTE (test) key and an OTE credential
+against api.godaddy.com returns 401. A Personal Access Token also works: paste it as the API Key
+and leave the Secret blank, and the Authorization header becomes `Bearer <token>`. That path is not
+cosmetic — GoDaddy marks sso-key "deprecated, supported through 2026" and the current v1 OpenAPI
+advertises only bearer auth, so the PAT is the migration target, not an alternative.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any, Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+GODADDY_API_BASE = "https://api.godaddy.com/v1"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# GoDaddy enforces a 600s (10 min) TTL floor at request time. The published v1 OpenAPI declares no
+# minimum, so a smaller value is not caught by the schema — it fails with
+# 422 {"code":"INVALID_BODY","fields":[{"message":"must have a minimum value of 600", ...}]}.
+# Pin the floor; DNS-01 has no reason to want anything longer.
+_TXT_TTL = 600
+
+# Read-modify-write serialization, keyed by the RRset (record name), not the zone — the RRset is the
+# actual unit of contention, and keying on it avoids serializing unrelated subdomains of one zone.
+# The orchestrator is sequential today (see the module docstring), so this is defence in depth: it
+# is what stops a future `asyncio.gather()` over the publish loop from silently breaking every
+# wildcard+apex certificate. Bounded in practice by the certificate inventory of one process, so
+# there is no eviction; the entries are empty Lock objects.
+_RRSET_LOCKS: Dict[str, asyncio.Lock] = {}
+
+
+def _rrset_lock(record_name: str) -> asyncio.Lock:
+    key = (record_name or "").rstrip(".").lower()
+    lock = _RRSET_LOCKS.get(key)
+    if lock is None:
+        # Safe without a guard: a single event loop never preempts between the get and the assign.
+        lock = _RRSET_LOCKS[key] = asyncio.Lock()
+    return lock
+
+
+def _scrub(text: str, *secrets: str) -> str:
+    """Remove credential substrings from a message before it can reach a log or an order event.
+
+    GoDaddy error bodies do not echo the Authorization header, so this is belt-and-braces — but it
+    makes base.py's "never leak a secret" invariant structural instead of a matter of care. Short
+    strings are skipped so a 1-2 char credential fragment cannot blank out ordinary prose.
+    """
+    out = text or ""
+    for secret in secrets:
+        if secret and len(secret) >= 4:
+            out = out.replace(secret, "***")
+    return out[:300]
+
+
+def _relative_name(fqdn: str, zone: str) -> str:
+    """Convert an absolute record name to the zone-relative form GoDaddy's API requires.
+
+    GoDaddy record names are RELATIVE to the zone with NO trailing dot, and the zone apex is the
+    literal "@" — never an empty string (which would collapse the URL onto the zone-wide TXT
+    endpoint) and never the domain name itself.
+
+      ("_acme-challenge.example.com",         "example.com") -> "_acme-challenge"
+      ("_acme-challenge.foo.bar.example.com", "example.com") -> "_acme-challenge.foo.bar"
+      ("example.com",                         "example.com") -> "@"
+    """
+    f = (fqdn or "").rstrip(".").lower()
+    z = (zone or "").rstrip(".").lower()
+    if z and f == z:
+        return "@"
+    if z and f.endswith("." + z):
+        return f[: -(len(z) + 1)]
+    # Defensive: callers always pass a zone that _resolve_domain derived from this very name.
+    return f or "@"
+
+
+def _rrset_path(zone: str, rel_name: str) -> str:
+    """Build the 4-segment record path `/domains/{zone}/records/TXT/{name}`.
+
+    SAFETY GATE: an empty rel_name would collapse the URL to `/domains/{zone}/records/TXT` — the
+    endpoint that replaces EVERY TXT record in the zone (SPF, DKIM, DMARC, domain verifications).
+    A "." or ".." segment does the same thing one step later: `quote()` leaves both untouched
+    (they are unreserved) and yarl normalizes dot segments away when it builds the URL, so
+    ".../records/TXT/.." would resolve to ".../records" — the whole-zone endpoint. Refuse both
+    rather than build them. `safe=''` percent-encodes the apex "@" as "%40" (accepted bare too,
+    but safer through proxies); "_", "-" and "." are unreserved and pass through unchanged, so a
+    multi-label relative name stays one readable path segment.
+    """
+    if not zone or not rel_name:
+        raise DnsProviderError("Internal error: refusing to build a zone-wide GoDaddy TXT record path.")
+    if rel_name.strip(".") == "" or any(part in (".", "..") for part in rel_name.split("/")):
+        raise DnsProviderError("Internal error: refusing to build a GoDaddy TXT path from a dot segment.")
+    return f"/domains/{quote(zone, safe='')}/records/TXT/{quote(rel_name, safe='')}"
+
+
+def _live_values(records: List[Dict]) -> List[str]:
+    """The non-empty `data` values in an RRset read.
+
+    GoDaddy leaves tombstone rows with `"data": ""` behind at a name after some removals. Echoing
+    one back in a PUT body is rejected with 422 INVALID_BODY, so every field implementation
+    (lego, acme.sh, Posh-ACME) filters them independently — so do we.
+    """
+    out: List[str] = []
+    for rec in records or []:
+        data = (rec or {}).get("data") or ""
+        if data:
+            out.append(data)
+    return out
+
+
+def _merge_add(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that adds `value` while preserving every coexisting sibling value.
+
+    Returns None when `value` is already present — an idempotent no-op, which is where an ACME
+    retry cycle lands.
+    """
+    live = _live_values(existing)
+    if value in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live] + [{"data": value, "ttl": _TXT_TTL}]
+
+
+def _merge_remove(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that removes ONLY `value`, keeping every sibling.
+
+    Three-state result, because GoDaddy needs three different calls:
+      None -> `value` is not there; already gone, tolerate (base.py's remove contract).
+      []   -> it was the last value; the caller must DELETE, since `PUT []` is rejected.
+      list -> PUT this body.
+    """
+    live = _live_values(existing)
+    if value not in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live if d != value]
+
+
+def _require_rrset(body: Any) -> List[Dict]:
+    """The RRset read, or a refusal.
+
+    FAIL CLOSED. A read that did not come back as a JSON array must never be treated as "the RRset
+    is empty" — the very next call is a full-RRset PUT, so coercing an unreadable read to [] would
+    replace every coexisting sibling value with just ours. Failing instead is free: the orchestrator
+    reverts the publish flag and retries next cycle, while a destructive PUT is unrecoverable.
+    """
+    if not isinstance(body, list):
+        raise DnsProviderError(
+            "GoDaddy returned an unreadable TXT record list; refusing to replace the record set."
+        )
+    return body
+
+
+def _error_fields(body: Any) -> Tuple[str, str]:
+    """The whitelisted (code, message) pair from a GoDaddy error body.
+
+    Only these two string fields are ever read; the raw body is never interpolated into a
+    user-facing message.
+    """
+    if not isinstance(body, dict):
+        return "", ""
+    code = body.get("code")
+    message = body.get("message")
+    return (code if isinstance(code, str) else ""), (message if isinstance(message, str) else "")
+
+
+def _retry_after_seconds(headers, body: Any) -> int:
+    """Seconds to wait after a 429.
+
+    The current platform sends `Retry-After` and `ratelimit-reset` headers with no body, while the
+    legacy v1 OpenAPI documents an `ErrorLimit` body carrying `retryAfterSec`. All three shapes are
+    live in the wild — and so is none of them, hence the 60s default.
+    """
+    for key in ("Retry-After", "ratelimit-reset"):
+        raw = (headers or {}).get(key)
+        if raw:
+            try:
+                return max(1, int(str(raw).strip()))
+            except (TypeError, ValueError):
+                pass
+    if isinstance(body, dict):
+        raw = body.get("retryAfterSec")
+        if isinstance(raw, int) and raw > 0:
+            return raw
+    return 60
+
+
+class _GoDaddyHTTPError(DnsProviderError):
+    """A DnsProviderError that also carries the HTTP status and GoDaddy `code`.
+
+    Callers INSIDE this module branch on the status (tolerate a 404 read-back, fall through a
+    zone probe), while everything outside — dns01_orchestrator, letsencrypt.py — still sees a
+    plain sanitized DnsProviderError and needs no change.
+    """
+
+    def __init__(self, message: str, status: int, code: str = ""):
+        super().__init__(message)
+        self.status = status
+        self.code = code
+
+
+class GoDaddyDNSProvider(DnsProvider):
+    name = "godaddy"
+    label = "GoDaddy"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_key",
+            "label": "API Key",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": ("Production API Key from developer.godaddy.com/keys — the first key the dashboard "
+                     "issues is an OTE (test) key and will be rejected. A Personal Access Token also "
+                     "works: paste it here and leave the Secret blank."),
+        },
+        {
+            "key": "api_secret",
+            "label": "API Secret",
+            "type": "password",
+            "required": False,
+            "max_length": 200,
+            "help": ("The Secret half of the same API Key pair. Leave blank ONLY if the field above "
+                     "holds a Personal Access Token. The account also needs at least one registered "
+                     "domain for GoDaddy to allow DNS API access at all."),
+        },
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        # Normalize, never validate: dns01_orchestrator.py calls get_provider() OUTSIDE any
+        # DnsProviderError guard, so a constructor that raised on malformed credentials would escape
+        # as an unhandled exception in the 60s background cycle. The UI drops blank fields before
+        # submitting, so a left-blank field arrives as a MISSING key rather than "" — `.get() or ""`
+        # covers both.
+        self._api_key = (self.credentials.get("api_key") or "").strip()
+        self._api_secret = (self.credentials.get("api_secret") or "").strip()
+        # Per-INSTANCE zone cache. A module-level cache would leak one ACME account's zone visibility
+        # into another's; an instance lives for exactly one orchestrator step, which is precisely the
+        # scope where caching pays off (apex + wildcard resolve the same zone from the same name).
+        self._zone_cache: Dict[str, str] = {}
+
+    def _auth_header(self) -> str:
+        """`sso-key <key>:<secret>` when a Secret is present, else `Bearer <token>` for a PAT.
+
+        Literal prefix, one space, a single colon — no base64, no URL-encoding, no quotes. Keeping
+        this as one swappable string is what makes GoDaddy's sso-key sunset a credential change
+        rather than a code change.
+        """
+        if self._api_secret:
+            return f"sso-key {self._api_key}:{self._api_secret}"
+        return f"Bearer {self._api_key}"
+
+    def _headers(self) -> Dict[str, str]:
+        # Accept is not optional: these endpoints content-negotiate application/xml and
+        # text/javascript. Content-Type is required on every write or GoDaddy answers 400/415.
+        return {
+            "Authorization": self._auth_header(),
+            "Accept": "application/json",
+            "Content-Type": "application/json",
+        }
+
+    def _http_error(self, status: int, code: str, message: str, retry_after: Optional[int]) -> _GoDaddyHTTPError:
+        """Map an HTTP status to a sanitized, operator-actionable DnsProviderError.
+
+        These strings land in acme_order_events and letsencrypt_orders.error_detail and are shown
+        in the order timeline, so each one names what to fix. GoDaddy's own `code`/`message` is
+        appended when present because the two 403 causes — account not eligible for the DNS API vs.
+        a PAT missing `domains.dns:update` — are indistinguishable by status alone. Scrubbing
+        happens HERE, at the single point where provider-supplied text enters a message, so a new
+        caller cannot forget it.
+        """
+        code = _scrub(code, self._api_key, self._api_secret)
+        message = _scrub(message, self._api_key, self._api_secret)
+        if status == 401:
+            detail = ("GoDaddy rejected the API credentials. Check they are a PRODUCTION Key/Secret pair "
+                      "from developer.godaddy.com/keys — the first key the dashboard issues is an OTE "
+                      "(test) key and is not valid here.")
+        elif status == 403:
+            detail = ("GoDaddy denied access to the DNS API. The account needs at least one registered "
+                      "domain, and a Personal Access Token needs the domains.domain:read and "
+                      "domains.dns:update scopes.")
+        elif status == 404:
+            detail = ("GoDaddy has no zone for this domain (check it is registered in this account and "
+                      "uses GoDaddy nameservers).")
+        elif status == 409:
+            detail = "GoDaddy reports this domain is not eligible to have its DNS records changed."
+        elif status == 422:
+            detail = "GoDaddy rejected the record change as invalid (HTTP 422)."
+        elif status == 429:
+            detail = f"GoDaddy rate limit reached; retry in ~{retry_after or 60}s."
+        else:
+            detail = f"GoDaddy API error (HTTP {status})."
+        if code or message:
+            detail += f" (GoDaddy: {code}{': ' + message if message else ''})"
+        return _GoDaddyHTTPError(detail, status=status, code=code)
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> Any:
+        """One GoDaddy API call. Returns the parsed JSON body, or None for the empty-bodied writes.
+
+        Raises a SANITIZED _GoDaddyHTTPError / DnsProviderError — never the credentials, never the
+        request, never a response body verbatim.
+        """
+        url = f"{GODADDY_API_BASE}{path}"
+        # v1.11.0: single funnel for every GoDaddy call. `safe_error_only=True`
+        # keeps the recorded error to the exception TYPE, matching the stance the
+        # handlers below already take — a raw message can carry the request URL.
+        # The `Authorization: sso-key <key>:<secret>` header never reaches the log:
+        # the header allowlist reduces it to a presence marker.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_GODADDY
+
+        try:
+            async with outbound_span(
+                target=TARGET_DNS_GODADDY,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        # content_type=None: every GoDaddy write answers 200/204 with an EMPTY body, and
+                        # aiohttp would otherwise raise on the missing/other content type before parsing.
+                        body = await resp.json(content_type=None)
+                    except ValueError:
+                        # ONLY a decode failure (JSONDecodeError subclasses ValueError) is swallowed —
+                        # an empty write body, or an HTML error page on a >=400. A transport failure
+                        # mid-read (ClientPayloadError, TimeoutError) must NOT land here: it would look
+                        # identical to "empty body", and a caller that reads an RRset would then see
+                        # None and could mistake it for an empty RRset. Those propagate to the handlers
+                        # below and become a real DnsProviderError.
+                        body = None
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    # 2xx only. Redirects are deliberately not followed (aiohttp would forward the
+                    # Authorization header), so a 3xx is a failed call — treating `< 400` as success
+                    # would report a redirected write as a silent no-op.
+                    if 200 <= resp.status < 300:
+                        return body
+                    code, message = _error_fields(body)
+                    retry_after = _retry_after_seconds(resp.headers, body) if resp.status == 429 else None
+                    raise self._http_error(resp.status, code, message, retry_after)
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Only the exception TYPE is interpolated: an aiohttp client error's str() can carry the
+            # request URL, and the message is persisted to the order timeline.
+            raise DnsProviderError(f"Could not reach the GoDaddy API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected GoDaddy API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._api_key:
+            return {"ok": False, "detail": "No GoDaddy API Key provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                # Cheapest read-only check: one request, no zone needed. Deliberately NOT
+                # GET /v1/domains/{domain} — GoDaddy has rejected that details call for small
+                # accounts since 2024-05 while record-level calls keep working, so verifying with it
+                # produces false negatives on accounts where DNS-01 would succeed.
+                body = await self._request(session, "GET", "/domains?limit=1")
+            if not isinstance(body, list):
+                return {"ok": False, "detail": "GoDaddy returned an unexpected response to the credential check."}
+            if not body:
+                # An empty list is NOT a failure: sub-zones delegated to GoDaddy nameservers are
+                # manageable via the records API but never appear in the domain listing.
+                return {"ok": True, "detail": ("GoDaddy credentials valid, but no domains are visible in this "
+                                               "account — the domain you validate must be registered here, or "
+                                               "be a zone delegated to GoDaddy nameservers.")}
+            return {"ok": True, "detail": "GoDaddy credentials valid."}
+        except DnsProviderError as exc:
+            detail = str(exc)
+            if not self._api_secret:
+                # The Bearer path is silent otherwise, and a half-filled form is the likeliest cause.
+                detail += (" Note: no API Secret was entered, so the API Key was sent as a Personal Access "
+                           "Token (Bearer). If you have a Key + Secret pair, enter both halves.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the GoDaddy credentials."}
+
+    async def _resolve_domain(self, session: aiohttp.ClientSession, record_name: str) -> str:
+        """Find the most-specific (longest-suffix) GoDaddy-managed zone for an absolute record name.
+
+        GoDaddy has no `/zones?name=` equivalent, so this walks suffixes longest-to-shortest and
+        probes `GET /v1/domains/{candidate}/records/NS`. That probe (rather than the domain listing
+        or the domain-details call) is deliberate: it finds sub-zones delegated to GoDaddy
+        nameservers, which never appear in `GET /v1/domains` at all, and it does not depend on the
+        details endpoint that small accounts are rejected from.
+        """
+        cached = self._zone_cache.get(record_name)
+        if cached:
+            return cached
+        labels = record_name.rstrip(".").lower().split(".")
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break  # a zone needs at least two labels
+            try:
+                body = await self._request(
+                    session, "GET", f"/domains/{quote(candidate, safe='')}/records/NS"
+                )
+            except _GoDaddyHTTPError as exc:
+                if exc.status in (404, 422):
+                    continue  # not a zone in this account — keep walking
+                # 401/403/409/429/5xx are credential, eligibility or platform failures, not
+                # "wrong zone". Continuing would burn the rate-limit budget re-failing on every
+                # remaining suffix and would bury the real cause under "no managed domain".
+                raise
+            if isinstance(body, list) and body:
+                self._zone_cache[record_name] = candidate
+                return candidate
+        raise DnsProviderError(f"No managed GoDaddy domain found for {record_name}.")
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                zone = await self._resolve_domain(session, name)
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status != 404:
+                        raise
+                    # Some accounts 404 reading back a record set in a zone whose WRITES succeed
+                    # (acme.sh #6517). Reachable only when the NS probe resolved the zone but the
+                    # TXT read 404s — if the NS probe itself 404s we never get here and the caller
+                    # sees "No managed GoDaddy domain found", which is the honest answer. We cannot
+                    # merge what we cannot read, and a single-value PUT would destroy any coexisting
+                    # sibling, so PATCH is the only correct recovery: it is the one genuinely
+                    # ADDITIVE primitive in v1 ("Appends DNS records ... Existing records with the
+                    # same type and name are preserved"). It cannot dedupe, but a duplicate
+                    # identical TXT is harmless for validation and cleanup removes the whole RRset.
+                    await self._request(
+                        session, "PATCH", f"/domains/{quote(zone, safe='')}/records",
+                        json=[{"type": "TXT", "name": _relative_name(name, zone),
+                               "data": value, "ttl": _TXT_TTL}],
+                    )
+                    return
+                body = _merge_add(_require_rrset(existing), value)
+                if body is None:
+                    return  # already published — idempotent, this is where ACME retries land
+                await self._request(session, "PUT", path, json=body)
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                try:
+                    zone = await self._resolve_domain(session, name)
+                except _GoDaddyHTTPError as exc:
+                    # Raise only what a later sweep could plausibly succeed at. reconcile_dns01_cleanup
+                    # swallows the error and leaves dns_record_cleaned FALSE, so the row is re-selected
+                    # every cycle — and its query takes a bare LIMIT 50, so rows that can NEVER succeed
+                    # (revoked key, account lost DNS-API eligibility) would monopolise the whole
+                    # cleanup budget and starve every other account. For those terminal statuses we
+                    # give up quietly: the orphaned `_acme-challenge` TXT is inert, and the same
+                    # credential failure is already loud on the publish path, where it is actionable.
+                    if exc.status == 429 or exc.status >= 500:
+                        raise
+                    return
+                except DnsProviderError:
+                    return  # zone genuinely not resolvable — nothing we could clean up
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status == 404:
+                        return  # RRset (or the read) is gone — tolerate
+                    raise
+                body = _merge_remove(_require_rrset(existing), value)
+                if body is None:
+                    return  # our value is not there — already gone, tolerate
+                if not body:
+                    # The LAST value at this name. `PUT []` is rejected (422 INVALID_BODY, "Records
+                    # must be specified"), so emptying an RRset REQUIRES DELETE. This removes only
+                    # TXT at this exact name; other names and other record types are preserved.
+                    # Do NOT fall back to the "write an empty string to delete" folklore — that hack
+                    # is what creates the tombstone rows _live_values has to filter.
+                    try:
+                        await self._request(session, "DELETE", path)
+                    except _GoDaddyHTTPError as exc:
+                        if exc.status == 404:
+                            return  # raced with another cleanup — tolerate
+                        raise
+                    return
+                await self._request(session, "PUT", path, json=body)

backend/services/dns_providers/registry.py

@@ -10,11 +10,13 @@ from typing import Dict, List, Type
 
 from .base import DnsProvider
 from .cloudflare import CloudflareDNSProvider
+from .godaddy import GoDaddyDNSProvider
 from .manual import ManualDNSProvider
 
 _PROVIDERS: Dict[str, Type[DnsProvider]] = {
     ManualDNSProvider.name: ManualDNSProvider,
     CloudflareDNSProvider.name: CloudflareDNSProvider,
+    GoDaddyDNSProvider.name: GoDaddyDNSProvider,
 }
 
 

backend/services/frontend_service.py

@@ -56,14 +56,14 @@ async def create_frontend_row(
             acl_rules, redirect_rules, use_backend_rules,
             request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
             rate_limit, compression, log_separate, monitor_uri,
-            cluster_id, maxconn, updated_at
+            cluster_id, maxconn, log_format, filters, updated_at
         ) VALUES (
             $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12,
             $13, $14, $15, $16, $17, $18, $19,
             $20, $21, $22,
             $23, $24, $25, $26, $27, $28,
             $29, $30, $31, $32,
-            $33, $34, CURRENT_TIMESTAMP
+            $33, $34, $35, $36, CURRENT_TIMESTAMP
         )
         RETURNING id
         """,
@@ -101,6 +101,8 @@ async def create_frontend_row(
         getattr(payload, "monitor_uri", None),
         cluster_id,
         getattr(payload, "maxconn", None),
+        getattr(payload, "log_format", None),  # Issue #38
+        getattr(payload, "filters", None),     # Issue #38
     )
 
     if mark_pending:

backend/services/haproxy_config.py

@@ -6,9 +6,98 @@ import json
 import urllib.parse
 from typing import Optional, List, Dict, Any
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import resolve_acme_backend_target
 
 logger = logging.getLogger(__name__)
 
+# Sentinel prefixes returned by `generate_haproxy_config_for_cluster` INSTEAD of a
+# configuration when generation fails. They are plain strings (not exceptions) for
+# historical reasons: the function's outer `except` swallows everything and returns
+# a one-line comment as the "config".
+#
+# That is a data-loss primitive on its own: an exception anywhere in the generator —
+# e.g. `urllib.parse.urlparse('http://host:99999').port` raising ValueError for an
+# out-of-range port in `acme_backend_url` — collapses a whole cluster's haproxy.cfg
+# into a single comment line, which the apply path then stores as APPLIED and pushes
+# to every agent. Callers that PERSIST the returned text MUST reject it first; use
+# `is_config_generation_error()` rather than matching the string by hand.
+CONFIG_GENERATION_ERROR_PREFIXES = (
+    "# Error generating configuration:",
+    "# Error: Cluster not found",
+)
+
+
+def is_config_generation_error(config_content: Optional[str]) -> bool:
+    """True when `config_content` is a generator failure sentinel, not a configuration.
+
+    Persisting or shipping a sentinel silently destroys a cluster's configuration, so
+    every call site that writes the generator's output to `config_versions` (or hands
+    it to an agent) must guard with this.
+    """
+    if not config_content:
+        return True
+    return config_content.lstrip().startswith(CONFIG_GENERATION_ERROR_PREFIXES)
+
+
+def select_acme_backend_source(candidates: List[tuple]):
+    """Pick the first candidate that resolves into a usable address.
+
+    `candidates` is an ordered list of ``(source_name, url)`` from most to least
+    specific. Returns ``(source_name, url, target, skipped)`` where ``skipped`` lists
+    the ``(source_name, url, target)`` of candidates that were rejected.
+
+    Falling through on UNUSABLE values, not just empty ones, is the point. Values
+    predating validation are common — the settings field was free text — and a
+    scheme-less ``10.90.1.4:8080`` cannot be resolved. Stopping at the first non-empty
+    candidate would emit a backend section with no ``server`` line: ``haproxy -c``
+    still passes because the section exists, Apply succeeds, and every challenge
+    request then 503s from an empty backend with nothing to show for it.
+    """
+    skipped = []
+    for source, url in candidates:
+        if not url:
+            continue
+        target = resolve_acme_backend_target(url)
+        if target.error_code:
+            skipped.append((source, url, target))
+            continue
+        return source, url, target, skipped
+
+    # Nothing usable. Report against the last non-empty candidate so the rendered
+    # comment and the log name a concrete value rather than an empty one.
+    if skipped:
+        source, url, target = skipped[-1]
+        return source, url, target, skipped[:-1]
+    source, url = candidates[0] if candidates else ('none', '')
+    return source, url, resolve_acme_backend_target(url), skipped
+
+
+def extract_acme_backend_target(config_content: Optional[str]) -> Optional[str]:
+    """Return the `server _acme_mgmt` argument string from a rendered config.
+
+    e.g. ``"10.90.1.4:80"`` or ``"mgmt.internal:443 ssl verify none"``; ``None`` when
+    the cluster renders no ACME challenge backend at all.
+
+    Used to decide whether an ACME-related edit actually CHANGES the shipped
+    configuration. Comparing whole config texts would report a difference on every
+    unrelated pending edit; comparing this one line answers the only question that
+    matters here — "would the HAProxy nodes start talking to a different address?"
+    """
+    if not config_content:
+        return None
+    in_section = False
+    for line in config_content.splitlines():
+        stripped = line.strip()
+        if stripped.startswith("backend "):
+            in_section = stripped == "backend _acme_challenge_backend"
+            continue
+        if stripped.startswith(("frontend ", "listen ", "defaults", "global")):
+            in_section = False
+            continue
+        if in_section and stripped.startswith("server _acme_mgmt "):
+            return stripped[len("server _acme_mgmt "):].strip()
+    return None
+
 
 def _format_redirect_rule(rule: Any) -> Optional[str]:
     """Render a single redirect rule into a HAProxy `redirect ...` line.
@@ -393,6 +482,12 @@ def _categorize_haproxy_directive(line: str) -> str:
         return "prelude"
     if s.startswith("acl "):
         return "acl"
+    # Issue #38: SPOE (and other) `filter` directives must be declared BEFORE
+    # the `http-request send-spoe-group` rules that use them, otherwise HAProxy
+    # fails with "unable to find SPOE engine". Own bucket, flushed right after
+    # `prelude` and before tcp_req/acl/http_req (see flush order below).
+    if s.startswith("filter "):
+        return "filter"
     if s.startswith("stick-table") or s.startswith("stick "):
         return "stick"
     if s.startswith("tcp-request"):
@@ -414,6 +509,7 @@ def _categorize_haproxy_directive(line: str) -> str:
         or s.startswith("compression ")
         or s.startswith("monitor-uri")
         or s.startswith("log ")
+        or s.startswith("log-format")  # Issue #38: log-format / log-format-sd
         or s.startswith("description ")
         or s.startswith("disabled")
         or s.startswith("enabled")
@@ -871,7 +967,15 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                         f"    bind {frontend['bind_address']}:{frontend['bind_port']}"
                     )
             
-            config_lines.append(f"    mode {frontend['mode']}")
+            # `frontends.mode` is `VARCHAR(10) DEFAULT 'http'` but NULLABLE, and rows
+            # can arrive with it unset via agent sync or config import. Interpolating
+            # the raw value then emits a literal `mode None`, which HAProxy rejects —
+            # taking down the whole cluster config, not just this frontend. Normalise
+            # once here and use the result everywhere below, so the ACME gate, the
+            # backend-mode check and the rendered line can never disagree with each
+            # other about what mode this frontend is in.
+            frontend_mode = (frontend.get('mode') or 'http').strip().lower()
+            config_lines.append(f"    mode {frontend_mode}")
 
             # ─────────────────────────────────────────────────────────────────
             # R2.3 / R3.3 (PR-1 hotfix): emit ordering buckets.
@@ -903,7 +1007,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # "stick-table already declared").
             # ─────────────────────────────────────────────────────────────────
             _fe_buckets: Dict[str, List[str]] = {
-                "prelude": [], "stick": [], "tcp_req": [],
+                "prelude": [], "filter": [], "stick": [], "tcp_req": [],
                 "acl": [], "http_req": [], "http_resp": [],
                 "redirect": [], "use_be": [], "default_be": [],
             }
@@ -983,7 +1087,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 _fe_buckets[cat].append(line)
 
             # ACME HTTP-01 Challenge routing (auto-managed)
-            if frontend['mode'] == 'http' and cluster_info.get('acme_enabled', False):
+            if frontend_mode == 'http' and cluster_info.get('acme_enabled', False):
                 _emit_fe("    acl is_acme_challenge path_beg /.well-known/acme-challenge/")
                 _emit_fe("    http-request allow if is_acme_challenge")
                 _emit_fe("    use_backend _acme_challenge_backend if is_acme_challenge")
@@ -996,20 +1100,31 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
                         _emit_fe(f"    {line_stripped}")
 
+            # Issue #38: emit frontend log-format and SPOE (etc.) filter directives.
+            # `log_format` routes to the `prelude` bucket, `filters` to the `filter`
+            # bucket (both via _emit_fe → _categorize_haproxy_directive), guaranteeing
+            # `filter ...` is rendered before the `http-request send-spoe-group` rules.
+            for _fld in ('log_format', 'filters'):
+                if frontend.get(_fld):
+                    for line in frontend[_fld].split('\n'):
+                        line_stripped = line.strip()
+                        if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
+                            _emit_fe(f"    {line_stripped}")
+
             # CRITICAL: Validate frontend-backend mode compatibility
             if frontend.get('default_backend'):
                 default_backend_name = frontend['default_backend'].strip() if frontend['default_backend'] else ''
                 if default_backend_name and default_backend_name not in ('[]', '{}', 'null', 'None'):
                     backend_mode = backend_modes.get(default_backend_name)
 
-                    if backend_mode and backend_mode != frontend['mode']:
-                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend['mode']}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
+                    if backend_mode and backend_mode != frontend_mode:
+                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend_mode}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
                         # FIX-10 marker: 'BACKEND-MODE-WARNING' keyword in
                         # the comment body routes it to the 'default_be'
                         # bucket via _categorize_haproxy_directive, so the
                         # warning emits next to the actual default_backend
                         # directive instead of at the top of the block.
-                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend['mode']}'")
+                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend_mode}'")
                         _emit_fe(f"    # BACKEND-MODE-WARNING: HAProxy will reject this configuration! Please fix the mode mismatch in UI.")
 
                     _emit_fe(f"    default_backend {default_backend_name}")
@@ -1223,6 +1338,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # ─────────────────────────────────────────────────────────────
             for _bucket_key in (
                 "prelude",
+                "filter",
                 "stick",
                 "tcp_req",
                 "acl",
@@ -1570,36 +1686,82 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     f"frontend found (would create orphan backend section)."
                 )
             else:
-                acme_url = cluster_info.get('acme_backend_url') or ''
-                if not acme_url:
-                    try:
-                        acme_settings = await db_conn.fetchrow(
-                            "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
-                        )
-                        if acme_settings and acme_settings['value']:
-                            val = acme_settings['value']
-                            if isinstance(val, str):
-                                try:
-                                    val = json.loads(val)
-                                except (json.JSONDecodeError, TypeError):
-                                    pass
-                            if val:
-                                acme_url = str(val)
-                    except Exception:
-                        pass
-                if not acme_url:
-                    from config import MANAGEMENT_BASE_URL
-                    acme_url = MANAGEMENT_BASE_URL
+                # Track WHERE the effective URL came from. Support has no way today to
+                # tell an operator-set value from the shipped `localhost` default, and
+                # this whole block emits no log line at all (contrast the skip branches
+                # above), so a wrong challenge backend is invisible until Let's Encrypt
+                # fails. `acme_source` is logged with the rendered host:port below.
+                acme_candidates = [
+                    ('cluster.acme_backend_url', cluster_info.get('acme_backend_url') or '')
+                ]
+                _settings_url = ''
+                try:
+                    acme_settings = await db_conn.fetchrow(
+                        "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
+                    )
+                    if acme_settings and acme_settings['value']:
+                        val = acme_settings['value']
+                        if isinstance(val, str):
+                            try:
+                                val = json.loads(val)
+                            except (json.JSONDecodeError, TypeError):
+                                pass
+                        if val:
+                            _settings_url = str(val)
+                except Exception:
+                    pass
+                acme_candidates.append(
+                    ('system_settings.acme.challenge_backend_url', _settings_url)
+                )
+                from config import MANAGEMENT_BASE_URL
+                acme_candidates.append(('config.MANAGEMENT_BASE_URL', MANAGEMENT_BASE_URL))
 
-                parsed = urllib.parse.urlparse(acme_url)
-                host = parsed.hostname or 'localhost'
-                port = parsed.port or (443 if parsed.scheme == 'https' else 8080)
-                ssl_flag = ' ssl verify none' if parsed.scheme == 'https' else ''
+                acme_source, acme_url, target, _skipped = select_acme_backend_source(
+                    acme_candidates
+                )
+                for _s_source, _s_url, _s_target in _skipped:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} skipping unusable value from "
+                        f"{_s_source} (reason={_s_target.error_code}): "
+                        f"{_s_target.error_message} value={_s_url!r}"
+                    )
 
                 config_lines.append("# ACME Challenge Backend (auto-managed by HAProxy OpenManager)")
                 config_lines.append("backend _acme_challenge_backend")
                 config_lines.append("    mode http")
-                config_lines.append(f"    server _acme_mgmt {host}:{port}{ssl_flag}")
+                if target.error_code:
+                    # The value cannot produce an address. Emit the section without a
+                    # `server` line rather than guessing: every HTTP frontend already
+                    # carries `use_backend _acme_challenge_backend`, and a use_backend
+                    # with no matching backend is fatal to `haproxy -c`. The operator's
+                    # raw value is deliberately NOT echoed into the file — a value
+                    # containing a newline would inject directives into a config pushed
+                    # to every node. It goes to the log instead.
+                    config_lines.append(
+                        f"    # ACME challenge backend unavailable ({target.error_code}) — "
+                        f"see Cluster Management > ACME Challenge Backend URL"
+                    )
+                    logger.error(
+                        f"ACME-BACKEND: cluster {cluster_id} has an unusable challenge backend "
+                        f"URL (source={acme_source}, reason={target.error_code}): "
+                        f"{target.error_message} value={acme_url!r}"
+                    )
+                else:
+                    config_lines.append(
+                        f"    server _acme_mgmt {target.host}:{target.port}{target.ssl_flag}"
+                    )
+                    # One greppable line per render. `ACME-BACKEND` is the support
+                    # keyword: it answers "what address did we actually ship, and who
+                    # chose it?" without shell access to the node.
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} challenge backend rendered as "
+                        f"{target.host}:{target.port}{target.ssl_flag} "
+                        f"(source={acme_source}, url={acme_url!r})"
+                    )
+                for _warning in target.warnings:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} (source={acme_source}): {_warning}"
+                    )
                 config_lines.append("")
 
         # Only close the connection if it was created within this function

backend/services/keepalived_parser.py

@@ -0,0 +1,533 @@
+"""Issue #27 follow-up — parse an EXISTING keepalived.conf so a hand-maintained VIP can be
+adopted into OpenManager's model (v1.10.4).
+
+Standalone and DB-free, like keepalived_config.py: the agent reports the file it found on a
+node, this module turns it into the fields `vip_instances` / `vip_members` need, and the
+adoption endpoint decides whether taking ownership is safe.
+
+WHY A PARSER AND NOT THE HEARTBEAT. The heartbeat carries two keepalived facts —
+`keepalive_state` (MASTER/BACKUP, best-effort from logs) and `keepalive_ip` (the first
+address grepped out of `virtual_ipaddress`). Rendering a node's config needs eleven:
+virtual_router_id, auth_pass, interface, priority, prefix_length, advert_int, unicast
+peers, track_haproxy, role and the address itself. Guessing the missing ones is not a
+cosmetic risk — a wrong VRID puts the nodes in two separate VRRP domains and a wrong
+auth_pass makes them reject each other, and either way both nodes claim the VIP.
+
+THE SAFETY CONTRACT. Adoption REPLACES the operator's file with our render, so anything in
+their file that `render_keepalived_conf` cannot reproduce would be silently destroyed on
+takeover — a `notify_master` failover hook, an LVS `virtual_server` section, a second
+address in one instance, a sync group. Extracting the fields is the easy half; the half
+that matters is `unsupported`, the list of directives we would drop. The caller must treat
+a non-empty `unsupported` as a refusal to adopt, not a warning to log.
+
+Secrets: a parsed instance carries `auth_pass` in cleartext because that is the only way to
+re-render an identical config. NEVER log a parse result. Callers persist it through
+`encrypt_vrrp_secret` and mask it in anything UI-facing, exactly as the VIP router already
+does for `auth_pass` in version diffs.
+"""
+from __future__ import annotations
+
+import ipaddress
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+# Directives `render_keepalived_conf` emits, and therefore the only ones a takeover can
+# reproduce. Anything else found in a vrrp_instance is reported in `unsupported`.
+_SUPPORTED_INSTANCE_KEYS = {
+    "state", "interface", "virtual_router_id", "priority", "advert_int",
+    "authentication", "unicast_src_ip", "unicast_peer", "virtual_ipaddress", "track_script",
+}
+# Top-level blocks we can account for. `vrrp_script` is reproduced only when it is the
+# check script we generate ourselves (see _classify_script).
+_SUPPORTED_TOP_KEYS = {"global_defs", "vrrp_script", "vrrp_instance"}
+
+# global_defs entries our render emits. An operator's file usually carries more (notification
+# email, router_id, ...) and losing those is a real change, so they are reported too.
+_SUPPORTED_GLOBAL_KEYS = {"enable_script_security", "script_user"}
+
+_IDENT_RE = re.compile(r"^[A-Za-z0-9._:-]+$")
+
+
+class KeepalivedParseError(ValueError):
+    """The text is not a keepalived.conf we can reason about (unbalanced braces etc.)."""
+
+
+# ---------------------------------------------------------------------------
+# Tokenizer / block reader
+# ---------------------------------------------------------------------------
+def _strip_comment(line: str) -> str:
+    """Drop a trailing comment. keepalived treats BOTH `#` and `!` as comment starters, and
+    neither is meaningful inside the quoted script paths we care about, so a quote-aware
+    scan is enough (a `#` inside quotes stays)."""
+    out: List[str] = []
+    quote: Optional[str] = None
+    for ch in line:
+        if quote:
+            out.append(ch)
+            if ch == quote:
+                quote = None
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            out.append(ch)
+            continue
+        if ch in ("#", "!"):
+            break
+        out.append(ch)
+    return "".join(out)
+
+
+def _split_tokens(line: str) -> List[str]:
+    """Whitespace split that keeps quoted strings whole and isolates braces, so
+    `virtual_ipaddress { 10.0.0.1/24 dev eth0 }` tokenizes the same as its multi-line form."""
+    tokens: List[str] = []
+    buf: List[str] = []
+    quote: Optional[str] = None
+
+    def flush() -> None:
+        if buf:
+            tokens.append("".join(buf))
+            buf.clear()
+
+    for ch in line:
+        if quote:
+            if ch == quote:
+                quote = None
+            else:
+                buf.append(ch)
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            continue
+        if ch.isspace():
+            flush()
+        elif ch in ("{", "}"):
+            flush()
+            tokens.append(ch)
+        else:
+            buf.append(ch)
+    flush()
+    return tokens
+
+
+def _read_blocks(text: str) -> List[Dict[str, Any]]:
+    """Parse the file into nested entries.
+
+    Each entry is either
+      {"kind": "block", "name": str, "args": [str], "body": [entries], "line": int}
+      {"kind": "line",  "tokens": [str], "line": int}
+
+    Line boundaries matter: inside `virtual_ipaddress` and `unicast_peer` each line is one
+    bare value, so a flat token stream could not tell two addresses apart.
+    """
+    root: List[Dict[str, Any]] = []
+    stack: List[List[Dict[str, Any]]] = [root]
+    # Blocks whose opening `{` we have seen, so a stray `}` can be reported with context.
+    open_blocks: List[str] = []
+
+    for lineno, raw in enumerate(text.splitlines(), start=1):
+        pending: List[str] = []
+        for tok in _split_tokens(_strip_comment(raw)):
+            if tok == "{":
+                name = pending[0] if pending else ""
+                args = pending[1:]
+                block = {"kind": "block", "name": name, "args": args, "body": [], "line": lineno}
+                stack[-1].append(block)
+                stack.append(block["body"])
+                open_blocks.append(name)
+                pending = []
+            elif tok == "}":
+                if pending:
+                    stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+                    pending = []
+                if len(stack) == 1:
+                    raise KeepalivedParseError(f"unbalanced '}}' on line {lineno}")
+                stack.pop()
+                open_blocks.pop()
+            else:
+                pending.append(tok)
+        if pending:
+            stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+
+    if len(stack) != 1:
+        raise KeepalivedParseError(f"unclosed block '{open_blocks[-1] or '?'}' at end of file")
+    return root
+
+
+# ---------------------------------------------------------------------------
+# Interpretation
+# ---------------------------------------------------------------------------
+def _as_int(tokens: List[str]) -> Optional[int]:
+    if len(tokens) < 2:
+        return None
+    try:
+        return int(tokens[1])
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_vip_entry(tokens: List[str]) -> Optional[Dict[str, Any]]:
+    """One `virtual_ipaddress` line: `<addr>[/<prefix>] [dev <iface>] [label ...]`.
+
+    Returns None when the first token is not an address — a shape we do not understand must
+    surface as unsupported rather than be silently dropped.
+    """
+    spec = tokens[0]
+    addr, _, prefix = spec.partition("/")
+    try:
+        ip = ipaddress.ip_address(addr)
+    except ValueError:
+        return None
+    entry: Dict[str, Any] = {
+        "address": str(ip),
+        "prefix_length": None,
+        "dev": None,
+        "extra": [],
+    }
+    if prefix:
+        try:
+            entry["prefix_length"] = int(prefix)
+        except ValueError:
+            return None
+    rest = tokens[1:]
+    i = 0
+    while i < len(rest):
+        if rest[i] == "dev" and i + 1 < len(rest):
+            entry["dev"] = rest[i + 1]
+            i += 2
+            continue
+        # `label`, `scope`, `brd`, ... — all real directives we do not render.
+        entry["extra"].append(rest[i])
+        i += 1
+    return entry
+
+
+def _classify_script(block: Dict[str, Any]) -> Tuple[str, Optional[str]]:
+    """Return (name, script_path) for a vrrp_script block."""
+    name = block["args"][0] if block["args"] else (block["name"] or "")
+    path = None
+    for entry in block["body"]:
+        if entry["kind"] == "line" and entry["tokens"] and entry["tokens"][0] == "script":
+            path = " ".join(entry["tokens"][1:]) or None
+    return name, path
+
+
+def _parse_instance(block: Dict[str, Any]) -> Dict[str, Any]:
+    """Interpret one `vrrp_instance` block into VIP-model fields plus its own unsupported list."""
+    inst: Dict[str, Any] = {
+        "instance_name": block["args"][0] if block["args"] else "",
+        "state": None,
+        "interface": None,
+        "virtual_router_id": None,
+        "priority": None,
+        "advert_int": None,
+        "auth_type": None,
+        "auth_pass": None,
+        "unicast_src_ip": None,
+        "unicast_peers": [],
+        "virtual_ips": [],
+        "track_scripts": [],
+        "unsupported": [],
+        "line": block["line"],
+    }
+
+    def unsupported(what: str, lineno: int) -> None:
+        inst["unsupported"].append({"directive": what, "line": lineno})
+
+    for entry in block["body"]:
+        if entry["kind"] == "line":
+            tokens = entry["tokens"]
+            key = tokens[0]
+            if key == "state":
+                inst["state"] = (tokens[1].upper() if len(tokens) > 1 else None)
+            elif key == "interface":
+                inst["interface"] = tokens[1] if len(tokens) > 1 else None
+            elif key == "virtual_router_id":
+                inst["virtual_router_id"] = _as_int(tokens)
+            elif key == "priority":
+                inst["priority"] = _as_int(tokens)
+            elif key == "advert_int":
+                # keepalived accepts sub-second floats; our model column is an integer.
+                raw = tokens[1] if len(tokens) > 1 else ""
+                try:
+                    val = float(raw)
+                except (TypeError, ValueError):
+                    val = None
+                if val is None:
+                    unsupported(f"advert_int {raw}", entry["line"])
+                elif val != int(val):
+                    # Rounding would change VRRP timing, so refuse rather than adopt-and-alter.
+                    unsupported(f"advert_int {raw} (fractional; model stores whole seconds)",
+                                entry["line"])
+                else:
+                    inst["advert_int"] = int(val)
+            elif key == "unicast_src_ip":
+                inst["unicast_src_ip"] = tokens[1] if len(tokens) > 1 else None
+            else:
+                unsupported(" ".join(tokens), entry["line"])
+            continue
+
+        name = entry["name"]
+        if name == "authentication":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    continue
+                k = sub["tokens"][0]
+                if k == "auth_type":
+                    inst["auth_type"] = (sub["tokens"][1].upper() if len(sub["tokens"]) > 1 else None)
+                elif k == "auth_pass":
+                    # Everything after the keyword: a VRRP password may contain spaces.
+                    inst["auth_pass"] = " ".join(sub["tokens"][1:]) or None
+                else:
+                    unsupported(f"authentication/{' '.join(sub['tokens'])}", sub["line"])
+        elif name == "unicast_peer":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["unicast_peers"].append(sub["tokens"][0])
+                else:
+                    unsupported("unicast_peer/<block>", entry["line"])
+        elif name == "virtual_ipaddress":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    unsupported("virtual_ipaddress/<block>", entry["line"])
+                    continue
+                parsed = _parse_vip_entry(sub["tokens"])
+                if parsed is None:
+                    unsupported(f"virtual_ipaddress/{' '.join(sub['tokens'])}", sub["line"])
+                else:
+                    if parsed["extra"]:
+                        unsupported(
+                            f"virtual_ipaddress/{parsed['address']} "
+                            f"({' '.join(parsed['extra'])})", sub["line"])
+                    inst["virtual_ips"].append(parsed)
+        elif name == "track_script":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["track_scripts"].append(sub["tokens"][0])
+        else:
+            unsupported(f"{name} {{...}}", entry["line"])
+
+    return inst
+
+
+def parse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse a keepalived.conf into VIP-model fields plus everything we could not model.
+
+    Raises KeepalivedParseError on structurally broken input. Never log the result: parsed
+    instances carry `auth_pass` in cleartext.
+    """
+    root = _read_blocks(text or "")
+    result: Dict[str, Any] = {
+        "instances": [],
+        "scripts": {},
+        "global_defs": {},
+        "unsupported": [],   # top-level directives our render would drop
+        "sync_groups": [],
+    }
+
+    for entry in root:
+        if entry["kind"] == "line":
+            # A bare top-level directive (e.g. `include /etc/keepalived/conf.d/*.conf`).
+            result["unsupported"].append(
+                {"directive": " ".join(entry["tokens"]), "line": entry["line"]})
+            continue
+        name = entry["name"]
+        if name == "global_defs":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    key = sub["tokens"][0]
+                    result["global_defs"][key] = " ".join(sub["tokens"][1:])
+                    if key not in _SUPPORTED_GLOBAL_KEYS:
+                        result["unsupported"].append(
+                            {"directive": f"global_defs/{' '.join(sub['tokens'])}",
+                             "line": sub["line"]})
+                else:
+                    result["unsupported"].append(
+                        {"directive": f"global_defs/{sub.get('name', '?')} {{...}}",
+                         "line": sub["line"]})
+        elif name == "vrrp_script":
+            script_name, path = _classify_script(entry)
+            result["scripts"][script_name] = {"script": path, "line": entry["line"]}
+        elif name == "vrrp_instance":
+            result["instances"].append(_parse_instance(entry))
+        elif name == "vrrp_sync_group":
+            # A sync group ties instances together so they fail over as a unit. Our render has
+            # no equivalent, and dropping it changes failover semantics — never adopt silently.
+            group = entry["args"][0] if entry["args"] else ""
+            result["sync_groups"].append({"name": group, "line": entry["line"]})
+            result["unsupported"].append(
+                {"directive": f"vrrp_sync_group {group}", "line": entry["line"]})
+        else:
+            # virtual_server (LVS), static_routes, bfd_instance, ...
+            args = " ".join(entry["args"])
+            result["unsupported"].append(
+                {"directive": f"{name} {args} {{...}}".replace("  ", " "), "line": entry["line"]})
+
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Mapping to the VIP model + the adoption gate
+# ---------------------------------------------------------------------------
+# keepalived defaults we are willing to apply when a directive is absent, because the value
+# is unambiguous and re-rendering it changes nothing on the wire.
+_DEFAULT_ADVERT_INT = 1
+_DEFAULT_PRIORITY = 100
+_DEFAULT_STATE = "BACKUP"
+
+# The only track_script our renderer emits (keepalived_config.build_haproxy_check_script).
+OUR_CHECK_SCRIPT_NAME = "chk_haproxy"
+
+
+def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -> Dict[str, Any]:
+    """Map one parsed `vrrp_instance` onto vip_instances / vip_members fields.
+
+    Returns `adoptable` plus `blockers`. A blocker means taking ownership would change what
+    is running — either because our render cannot reproduce something in the file, or because
+    a value we must write is not knowable from the file. Adoption REPLACES the operator's
+    config, so "we could not read it" and "we would change it" are the same hazard, and both
+    have to stop the flow rather than be logged.
+
+    Never log the return value: `vip.auth_pass` is cleartext.
+    """
+    blockers: List[str] = []
+
+    # Directives we would drop. Report the file's own line numbers so the operator can look.
+    dropped = list(parsed.get("unsupported") or []) + list(instance.get("unsupported") or [])
+    for d in dropped:
+        blockers.append(
+            f"line {d['line']}: `{d['directive']}` — OpenManager's renderer cannot reproduce "
+            f"this, so adopting would delete it")
+
+    vips = instance.get("virtual_ips") or []
+    if len(vips) == 0:
+        blockers.append("the instance declares no virtual_ipaddress — nothing to adopt")
+    elif len(vips) > 1:
+        addrs = ", ".join(v["address"] for v in vips)
+        blockers.append(
+            f"the instance carries {len(vips)} addresses ({addrs}); a managed VIP holds exactly "
+            f"one, so adopting would drop all but the first")
+
+    vip_entry = vips[0] if vips else None
+
+    if instance.get("virtual_router_id") is None:
+        blockers.append("no virtual_router_id — it cannot be guessed: a wrong VRID puts the "
+                        "nodes in separate VRRP domains and both would claim the VIP")
+    if not instance.get("interface"):
+        blockers.append("no interface — required to render the instance and the address")
+
+    # An explicit prefix is required. Our renderer ALWAYS writes `<addr>/<prefix>`, the model
+    # column defaults to 24, and keepalived's own default for a bare address is a host route.
+    # Picking either one for the operator would silently change the VIP's netmask, so ask.
+    if vip_entry is not None and vip_entry.get("prefix_length") is None:
+        blockers.append(
+            f"`{vip_entry['address']}` has no explicit prefix length; state it during adoption "
+            f"so the netmask cannot change on takeover")
+
+    # The address must live on the instance's interface — that is the only `dev` we can render.
+    if vip_entry is not None and vip_entry.get("dev") and instance.get("interface") \
+            and vip_entry["dev"] != instance["interface"]:
+        blockers.append(
+            f"the address is bound to `dev {vip_entry['dev']}` but the instance uses "
+            f"`interface {instance['interface']}`; the render always uses the instance interface")
+
+    auth_type = instance.get("auth_type")
+    if auth_type not in (None, "PASS"):
+        blockers.append(f"auth_type {auth_type} is not supported (only PASS is rendered)")
+
+    # A tracked script that is not ours would be replaced by our HAProxy check.
+    tracked = [t for t in (instance.get("track_scripts") or [])]
+    foreign = [t for t in tracked if t != OUR_CHECK_SCRIPT_NAME]
+    if foreign:
+        blockers.append(
+            f"track_script {', '.join(foreign)} would be replaced by OpenManager's HAProxy "
+            f"health check")
+
+    state = instance.get("state") or _DEFAULT_STATE
+    if state not in ("MASTER", "BACKUP"):
+        blockers.append(f"state {state} is not MASTER or BACKUP")
+
+    peers = list(instance.get("unicast_peers") or [])
+    src = instance.get("unicast_src_ip")
+    # Our renderer emits unicast_src_ip and unicast_peer together, or neither.
+    if bool(src) != bool(peers):
+        which = "unicast_src_ip without unicast_peer" if src else "unicast_peer without unicast_src_ip"
+        blockers.append(f"{which} — the render emits both or neither")
+
+    candidate: Dict[str, Any] = {
+        "instance_name": instance.get("instance_name") or "",
+        "adoptable": not blockers,
+        "blockers": blockers,
+        "dropped_directives": dropped,
+        "vip": {
+            "virtual_ip": vip_entry["address"] if vip_entry else None,
+            "prefix_length": vip_entry.get("prefix_length") if vip_entry else None,
+            "virtual_router_id": instance.get("virtual_router_id"),
+            "advert_int": instance.get("advert_int") if instance.get("advert_int") is not None
+            else _DEFAULT_ADVERT_INT,
+            "use_unicast": bool(peers),
+            "track_haproxy": OUR_CHECK_SCRIPT_NAME in tracked,
+            "auth_pass": instance.get("auth_pass"),
+        },
+        "member": {
+            "network_interface": instance.get("interface"),
+            "role": state,
+            "priority": instance.get("priority") if instance.get("priority") is not None
+            else _DEFAULT_PRIORITY,
+        },
+        "peers": peers,
+        "unicast_src_ip": src,
+        # Which values came from a keepalived default rather than the file, so the UI can say so.
+        "defaulted": [
+            k for k, present in (
+                ("advert_int", instance.get("advert_int") is not None),
+                ("priority", instance.get("priority") is not None),
+                ("state", instance.get("state") is not None),
+            ) if not present
+        ],
+    }
+    return candidate
+
+
+# Substrings that identify the two blocker classes an operator is allowed to resolve. They are
+# matched rather than typed because the blocker text is what the UI shows; keeping the marker in
+# the sentence means the message and the rule cannot drift apart.
+_LOSS_MARKER = "would delete it"
+_PREFIX_MARKER = "no explicit prefix length"
+
+
+def remaining_blockers(blockers: List[str], *, prefix_supplied: bool = False,
+                       accept_data_loss: bool = False) -> List[str]:
+    """Blockers that survive what the operator is permitted to resolve.
+
+    Exactly two classes are resolvable, and the distinction is the whole safety argument:
+
+      * a missing prefix length is *unknown*, and the operator can supply it — we refuse to pick
+        a netmask for a live VIP ourselves;
+      * "our renderer cannot reproduce this, so adopting would delete it" is a *loss*, and losing
+        it can be an informed choice.
+
+    Everything else — an unknown virtual_router_id, a fractional advert_int, an unsupported
+    auth_type, an address on a different interface — is neither unknown nor a loss but an
+    impossibility, and no flag may wave it through. This is the single source of truth for that
+    rule; the endpoint and the UI both derive from it.
+    """
+    out: List[str] = []
+    for b in blockers or []:
+        if prefix_supplied and _PREFIX_MARKER in b:
+            continue
+        if accept_data_loss and _LOSS_MARKER in b:
+            continue
+        out.append(b)
+    return out
+
+
+def analyse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse + map in one call: the shape the discovery endpoint stores and the UI renders."""
+    parsed = parse_keepalived_conf(text)
+    return {
+        "instance_count": len(parsed["instances"]),
+        "sync_groups": parsed["sync_groups"],
+        "global_defs": parsed["global_defs"],
+        "candidates": [build_adoption_candidate(parsed, inst) for inst in parsed["instances"]],
+    }

backend/services/ssl_service.py

@@ -40,10 +40,12 @@ flow.
   (callers translate to wizard step-jumpback toasts).
 """
 
+import hashlib
 import json
 import logging
+import time
 from datetime import datetime, timezone
-from typing import Any, Optional
+from typing import Any, List, Optional
 
 from fastapi import HTTPException
 
@@ -106,23 +108,22 @@ def _recompute_status_from_expiry(
         return cert_info_status or "valid", cert_info_days or 0
 
 
-async def create_cert_row(
-    conn,
-    payload: Any,
-    cluster_id: int,
-) -> int:
-    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
-    binding to the given cluster_id. Returns new ssl_certificate_id.
+def _prepare_cert_fields(payload: Any) -> dict:
+    """Parse + validate the PEM material on `payload` and derive every
+    ssl_certificates column value from it (v1.9.0 extraction — shared by
+    `create_cert_row` and the CSR import flow in services/csr_service.py,
+    byte-identical to the former inline body of `create_cert_row`).
 
     payload is expected to expose:
       name, certificate_content, private_key_content, chain_content,
       usage_type (optional, default 'frontend').
 
-    All cert metadata (primary_domain, all_domains, expiry_date,
-    issuer, fingerprint, status, days_until_expiry) is now parsed
-    FROM the PEM content via `parse_ssl_certificate` — operator-
-    supplied values on the payload are accepted as a graceful
-    fallback only when parsing fails (which itself raises 400).
+    Raises HTTPException(400) on any parse/validation failure (invalid PEM,
+    bad private key, cert/key mismatch, bad chain, already-expired cert).
+
+    Returns a dict with keys: cert_content, private_key_content,
+    chain_content, cert_info, primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry, usage_type.
     """
     cert_content = getattr(payload, "certificate_content", None) or ""
     if not cert_content.strip():
@@ -213,6 +214,53 @@ async def create_cert_row(
     )
     usage_type = getattr(payload, "usage_type", "frontend") or "frontend"
 
+    return {
+        "cert_content": cert_content,
+        "private_key_content": private_key_content,
+        "chain_content": chain_content,
+        "cert_info": cert_info,
+        "primary_domain": primary_domain,
+        "all_domains": all_domains,
+        "expiry_date": expiry_date,
+        "issuer": issuer,
+        "fingerprint": fingerprint,
+        "status": status,
+        "days_until_expiry": days_until_expiry,
+        "usage_type": usage_type,
+    }
+
+
+async def create_cert_row(
+    conn,
+    payload: Any,
+    cluster_id: int,
+) -> int:
+    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
+    binding to the given cluster_id. Returns new ssl_certificate_id.
+
+    payload is expected to expose:
+      name, certificate_content, private_key_content, chain_content,
+      usage_type (optional, default 'frontend').
+
+    All cert metadata (primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry) is now parsed
+    FROM the PEM content via `parse_ssl_certificate` — operator-
+    supplied values on the payload are accepted as a graceful
+    fallback only when parsing fails (which itself raises 400).
+    """
+    fields = _prepare_cert_fields(payload)
+    cert_content = fields["cert_content"]
+    private_key_content = fields["private_key_content"]
+    chain_content = fields["chain_content"]
+    expiry_date = fields["expiry_date"]
+    primary_domain = fields["primary_domain"]
+    all_domains = fields["all_domains"]
+    issuer = fields["issuer"]
+    fingerprint = fields["fingerprint"]
+    status = fields["status"]
+    days_until_expiry = fields["days_until_expiry"]
+    usage_type = fields["usage_type"]
+
     existing = await conn.fetchrow(
         """
         SELECT s.id, s.is_active
@@ -408,3 +456,81 @@ async def validate_server_ca_bundle_eligibility(
         cluster_id,
     )
     return row is not None
+
+
+async def stage_ssl_config_versions(
+    conn,
+    cert_id: int,
+    cluster_ids: List[int],
+    action: str = "create",
+    created_by: Optional[int] = None,
+) -> List[dict]:
+    """Stage one PENDING config version per affected cluster after an SSL
+    certificate mutation (v1.9.0 — distilled from the routers/ssl.py POST
+    /certificates staging loop; used by the CSR import flow).
+
+    Uses the EXACT `ssl-{cert_id}-{action}-{timestamp}` version-name scheme of
+    the manual SSL flow so Apply Management, the `has_pending_config`
+    LIKE-filter ('ssl-' || id || '-%'), and the agent delivery predicates
+    treat CSR-imported certificates identically to manually uploaded ones.
+    Agents are NOT notified here — the operator applies manually.
+
+    Per-cluster failures are caught and reported in the returned
+    sync_results list (the DB save has already succeeded — same semantics as
+    the manual flow, where a config-generation failure never rolls back the
+    certificate row).
+    """
+    # Local import: keeps services/haproxy_config free to import ssl helpers
+    # without a module-level cycle.
+    from services.haproxy_config import generate_haproxy_config_for_cluster
+
+    sync_results: List[dict] = []
+    for cluster_id in cluster_ids:
+        try:
+            config_content = await generate_haproxy_config_for_cluster(cluster_id)
+            config_hash = hashlib.sha256(config_content.encode()).hexdigest()
+            version_name = f"ssl-{cert_id}-{action}-{int(time.time())}"
+
+            version_created_by = created_by
+            if version_created_by is None:
+                version_created_by = await conn.fetchval(
+                    "SELECT id FROM users WHERE username = 'admin' LIMIT 1"
+                ) or 1
+
+            await conn.fetchval(
+                """
+                INSERT INTO config_versions
+                (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
+                VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                RETURNING id
+                """,
+                cluster_id,
+                version_name,
+                config_content,
+                config_hash,
+                version_created_by,
+            )
+            logger.info(
+                f"APPLY WORKFLOW: Created PENDING config version {version_name} "
+                f"for cluster {cluster_id} (ssl_service.stage_ssl_config_versions)"
+            )
+            sync_results.append({
+                'node': 'pending',
+                'success': True,
+                'cluster_id': cluster_id,
+                'version': version_name,
+                'status': 'PENDING',
+                'message': 'SSL certificate staged. Click Apply to activate.',
+            })
+        except Exception as e:
+            logger.error(
+                f"Cluster config staging failed for SSL certificate {cert_id} "
+                f"on cluster {cluster_id}: {e}"
+            )
+            sync_results.append({
+                'node': 'cluster',
+                'success': False,
+                'cluster_id': cluster_id,
+                'error': str(e),
+            })
+    return sync_results

backend/tests/test_acl_pattern_file_allow.py

@@ -0,0 +1,190 @@
+"""Issue #38 follow-up — ACL `-f <file>` pattern-file support (v1.8.9).
+
+The Bulgu #12 hard rejects were removed: pattern files are
+operator-managed host files (same policy as the SPOE
+`filter ... config <path>` reference preserved since v1.8.8), bulk
+import always accepted `-f`, and the agent runs `haproxy -c` before
+every reload so a missing file fails safely. These tests pin:
+
+1. ACCEPT — the manual FrontendConfig model and the wizard models
+   accept `-f` in every rule field (string + dict shapes).
+2. GUARDS KEPT — `$(`/backtick shell-substitution rejects and the
+   `X !X` contradiction machinery are unchanged.
+3. WARNINGS — `_pattern_file_warnings` emits exactly one advisory
+   listing the referenced files, and NOTHING for `-f`-free rules
+   (zero-noise: existing users see no new output).
+4. ADVISORY — the bulk-import preview advisory block scans
+   acl/use_backend rules (and only those fields).
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from models.frontend import FrontendConfig  # noqa: E402
+from routers.frontend import _pattern_file_warnings  # noqa: E402
+
+
+ACL_F = "blacklisted src -f /etc/haproxy/blacklist.lst"
+UB_F = "be-secure if { src -f /etc/haproxy/allowlist.lst }"
+REDIR_F = "location /blocked if { src -f /etc/haproxy/blacklist.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 1. ACCEPT — manual FrontendConfig model
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_frontend_config_accepts_acl_file_flag():
+    fe = FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[ACL_F])
+    assert fe.acl_rules == [ACL_F]
+
+
+def test_frontend_config_accepts_use_backend_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", use_backend_rules=[UB_F])
+    assert fe.use_backend_rules == [UB_F]
+
+
+def test_frontend_config_accepts_redirect_string_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[REDIR_F])
+    assert fe.redirect_rules == [REDIR_F]
+
+
+def test_frontend_config_accepts_redirect_dict_file_flag():
+    rule = {"type": "scheme", "scheme": "https",
+            "condition": "if { src -f /etc/haproxy/blacklist.lst }"}
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[rule])
+    assert fe.redirect_rules == [rule]
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 2. GUARDS KEPT — dangerous-content rejects unchanged
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "acl1 path $(rm -rf /)",
+    "acl1 path `id`",
+])
+def test_acl_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[bad_rule])
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "be1 if $(whoami)",
+    "be1 if `id`",
+])
+def test_use_backend_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(
+            name="fe1", bind_port=80, mode="http", use_backend_rules=[bad_rule])
+
+
+def test_contradiction_detection_still_works_on_file_flag_rules():
+    """Interaction guard: a `-f` rule with an `X !X` contradiction is
+    still caught by the handler-level contradiction machinery — the
+    `-f` relaxation must not weaken that gate."""
+    from models.frontend import _frontend_has_acl_contradiction
+    assert _frontend_has_acl_contradiction(
+        "be1 if blacklisted !blacklisted") is True
+    # And a normal -f rule is NOT a contradiction.
+    assert _frontend_has_acl_contradiction(UB_F) is False
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 3. WARNINGS — _pattern_file_warnings (zero-noise contract)
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_pattern_file_warnings_lists_unique_paths():
+    warnings = _pattern_file_warnings(
+        acl_rules=[ACL_F, "other src -f /etc/haproxy/blacklist.lst"],
+        use_backend_rules=[UB_F],
+        redirect_rules=[{"condition": "if { src -f /etc/haproxy/geo.lst }"}],
+    )
+    assert len(warnings) == 1
+    w = warnings[0]
+    assert "/etc/haproxy/blacklist.lst" in w
+    assert "/etc/haproxy/allowlist.lst" in w
+    assert "/etc/haproxy/geo.lst" in w
+    # Duplicate path listed once.
+    assert w.count("/etc/haproxy/blacklist.lst") == 1
+    # Non-blocking framing: mentions fail-safe haproxy -c.
+    assert "haproxy -c" in w
+
+
+def test_pattern_file_warnings_empty_without_file_flag():
+    """Zero-noise: operators who don't use `-f` must see NO warning."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_api path_beg /api", "is_admin src 10.0.0.0/24"],
+        use_backend_rules=["be-api if is_api"],
+        redirect_rules=[{"type": "scheme", "scheme": "https",
+                         "condition": "if !{ ssl_fc }"}],
+    ) == []
+    assert _pattern_file_warnings() == []
+
+
+def test_pattern_file_warnings_ignores_dash_f_substrings():
+    """`-file`/`-foo` substrings must not trigger the advisory."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_self path_beg /self-config-file",
+                   "is_foo path_beg /foo -m beg"],
+    ) == []
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 4. Wizard models accept `-f` (string + dict) — parity
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_wizard_models_accept_file_flag():
+    from models.site_wizard import FrontendStep
+
+    fe = FrontendStep(
+        name="fe1", mode="http", bind_address="*", bind_port=80,
+        acl_rules=[ACL_F],
+        use_backend_rules=["be-x if blacklisted"],
+        redirect_rules=[{"type": "scheme", "target": "https",
+                         "condition": "if { src -f /etc/haproxy/x.lst }"}],
+    )
+    assert fe.acl_rules == [ACL_F]
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/x.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 5. Bulk-import preview advisory — source-level pin
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_parse_bulk_advisory_scans_only_structured_rule_fields():
+    """The preview advisory scans acl_rules/use_backend_rules but NOT
+    request_headers/tcp_request_rules (always-free-form fields —
+    warning there would add new noise for existing users)."""
+    src = Path(__file__).resolve().parents[1] / "routers" / "config.py"
+    text = src.read_text()
+    block_start = text.index("pattern-file advisory")
+    block = text[block_start:block_start + 1200]
+    assert 'acl_rules' in block
+    assert 'use_backend_rules' in block
+    assert 'request_headers' not in block.split("_pattern_paths")[1], (
+        "advisory must not scan request_headers")
+
+
+def test_no_dash_f_reject_left_in_models():
+    """No model file may still hard-reject the `-f` flag."""
+    for rel in ("models/frontend.py", "models/site_wizard.py"):
+        text = (Path(__file__).resolve().parents[1] / rel).read_text()
+        for m in re.finditer(r"-f\(\\s\|\$\)", text):
+            ctx = text[max(0, m.start() - 400):m.start() + 400]
+            assert "raise ValueError" not in ctx, (
+                f"{rel}: a `-f` reject regex still sits next to a raise")

backend/tests/test_acme_backend_url.py

@@ -0,0 +1,291 @@
+"""ACME challenge backend URL validation, resolution and change detection.
+
+These pin the behaviour behind a real incident: a split deployment rendered
+`server _acme_mgmt <mgmt>:8080` against a port with no listener, HTTP-01 failed for
+weeks while DNS-01 kept working, and every existing check reported success. The
+three mechanisms below are what make that impossible to repeat silently.
+"""
+import pytest
+
+from services.haproxy_config import (
+    extract_acme_backend_target,
+    is_config_generation_error,
+    select_acme_backend_source,
+)
+from utils.acme_backend_url import (
+    AcmeBackendUrlError,
+    resolve_acme_backend_target,
+    validate_acme_backend_url,
+)
+
+
+# ---------------------------------------------------------------------------
+# 1. Boundary validation — what an operator may type.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "value,expected",
+    [
+        ("http://10.90.1.4:80", "http://10.90.1.4:80"),
+        ("http://10.90.1.4", "http://10.90.1.4"),
+        ("https://mgmt.internal:8443", "https://mgmt.internal:8443"),
+        # RFC1918 is the NORMAL answer here, unlike utils/ssrf_guard's policy: the
+        # operator is naming their own management host, which on a split deployment
+        # is private by definition.
+        ("http://192.168.1.5:8080", "http://192.168.1.5:8080"),
+        # Empty means "inherit from the next level of the resolution chain".
+        (None, None),
+        ("", None),
+        ("   ", None),
+        # Surrounding whitespace is normalised, not rejected — and the NORMALISED
+        # value is what callers persist, so it can never reach haproxy.cfg.
+        ("  http://10.0.0.5:80  ", "http://10.0.0.5:80"),
+    ],
+)
+def test_accepts_and_normalises_usable_values(value, expected):
+    assert validate_acme_backend_url(value) == expected
+
+
+@pytest.mark.parametrize(
+    "value,code",
+    [
+        # Scheme-less values used to be accepted and then silently became `localhost`
+        # in the renderer — the trap that makes a correct diagnosis un-actionable.
+        ("10.90.1.4:8080", "no_scheme"),
+        ("localhost:8080", "no_scheme"),
+        ("ftp://10.0.0.5", "bad_scheme"),
+        # A newline would inject directives into a file pushed to every node.
+        ("http://10.90.1.4\nbind :9", "whitespace"),
+        ("http://10.90.1.4 x", "whitespace"),
+        # Loopback by number AND by name: `localhost` is what both shipped defaults
+        # contain, so catching only the numeric form would miss the common case.
+        ("http://localhost:8080", "loopback"),
+        ("http://LOCALHOST", "loopback"),
+        ("http://127.0.0.1", "loopback"),
+        ("http://[::1]:80", "loopback"),
+        ("http://0.0.0.0:80", "unspecified"),
+        ("http://169.254.169.254", "link_local"),
+        ("http://u:p@10.0.0.5", "userinfo"),
+        ("http://10.0.0.5/api", "has_path"),
+        ("http://10.0.0.5?x=1", "has_path"),
+        # urlparse defers port parsing to attribute access; unguarded this raises
+        # inside the config generator and destroys the cluster's whole config.
+        ("http://10.0.0.5:99999", "bad_port"),
+        ("http://10.0.0.5:abc", "bad_port"),
+        ("http://-bad-.com", "invalid_host"),
+    ],
+)
+def test_rejects_unusable_values_with_stable_codes(value, code):
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url(value)
+    assert exc.value.code == code
+    assert str(exc.value), "every rejection must carry operator-facing prose"
+
+
+def test_rejects_values_longer_than_the_column():
+    # VARCHAR(500); without this the write fails as an opaque asyncpg 22001 -> 500.
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url("http://" + "a" * 600 + ".com")
+    assert exc.value.code == "too_long"
+
+
+# ---------------------------------------------------------------------------
+# 2. Render-time resolution — never rejects, never raises.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "url,host,port,ssl_flag",
+    [
+        ("http://10.90.1.4:80", "10.90.1.4", 80, ""),
+        # Port-less http stays 8080, NOT the scheme default 80: the bundled compose
+        # publishes nginx on 8080, so installs relying on this have a working path
+        # today and changing it would break them silently in the renewal loop.
+        ("http://10.90.1.4", "10.90.1.4", 8080, ""),
+        ("https://m.io", "m.io", 443, " ssl verify none"),
+        ("http://localhost:8080", "localhost", 8080, ""),
+    ],
+)
+def test_resolution_preserves_existing_rendering(url, host, port, ssl_flag):
+    target = resolve_acme_backend_target(url)
+    assert (target.host, target.port, target.ssl_flag) == (host, port, ssl_flag)
+    assert target.error_code is None
+
+
+@pytest.mark.parametrize(
+    "url",
+    ["", None, "   ", "10.0.0.5:80", "http://h:99999", "http://10.0.0.5\nx", "http://ba d"],
+)
+def test_resolution_reports_instead_of_raising(url):
+    target = resolve_acme_backend_target(url)
+    assert target.error_code, "unusable values must be reported, not raised"
+    assert target.error_message
+
+
+def test_resolution_warns_on_loopback_rather_than_refusing():
+    # Refusing here would make every existing install unappliable: the shipped
+    # defaults ARE loopback, and the failure would block changes unrelated to ACME.
+    target = resolve_acme_backend_target("http://localhost:8080")
+    assert target.error_code is None
+    assert target.warnings and "Loopback" in target.warnings[0]
+
+
+def test_resolution_warns_when_the_port_is_omitted():
+    target = resolve_acme_backend_target("http://10.0.0.5")
+    assert target.port == 8080
+    assert any("port" in w.lower() for w in target.warnings)
+
+
+# ---------------------------------------------------------------------------
+# 3. Change detection — what makes a panel edit actually reach the nodes.
+# ---------------------------------------------------------------------------
+
+
+_CONFIG = """global
+    daemon
+
+frontend fe_http
+    bind 10.90.1.100:80
+    mode http
+    acl is_acme_challenge path_beg /.well-known/acme-challenge/
+    use_backend _acme_challenge_backend if is_acme_challenge
+    default_backend app
+
+backend app
+    server s1 10.0.0.9:8080
+
+# ACME Challenge Backend (auto-managed by HAProxy OpenManager)
+backend _acme_challenge_backend
+    mode http
+    server _acme_mgmt 10.90.1.4:80
+"""
+
+
+def test_extracts_the_challenge_backend_target():
+    assert extract_acme_backend_target(_CONFIG) == "10.90.1.4:80"
+
+
+def test_extracts_target_with_ssl_flag():
+    cfg = _CONFIG.replace("10.90.1.4:80", "m.io:443 ssl verify none")
+    assert extract_acme_backend_target(cfg) == "m.io:443 ssl verify none"
+
+
+def test_ignores_server_lines_in_other_backends():
+    # Comparing the whole config would flag every unrelated pending edit as a change;
+    # this must key on the ACME section alone.
+    cfg = _CONFIG.replace("backend _acme_challenge_backend", "backend something_else")
+    assert extract_acme_backend_target(cfg) is None
+
+
+def test_returns_none_when_the_section_has_no_server_line():
+    cfg = (
+        "backend _acme_challenge_backend\n"
+        "    mode http\n"
+        "    # ACME challenge backend unavailable (loopback)\n"
+    )
+    assert extract_acme_backend_target(cfg) is None
+
+
+@pytest.mark.parametrize("value", ["", None])
+def test_extraction_tolerates_empty_input(value):
+    assert extract_acme_backend_target(value) is None
+
+
+def test_url_change_that_renders_the_same_target_is_not_a_change():
+    # `http://10.90.1.4` and `http://10.90.1.4:8080` are different strings but the
+    # same shipped address; minting a config version for that would put a no-op
+    # pending change in front of the operator.
+    a = resolve_acme_backend_target("http://10.90.1.4")
+    b = resolve_acme_backend_target("http://10.90.1.4:8080")
+    assert (a.host, a.port, a.ssl_flag) == (b.host, b.port, b.ssl_flag)
+
+
+# ---------------------------------------------------------------------------
+# 4. Source selection — an unusable value must not shadow a usable one.
+# ---------------------------------------------------------------------------
+
+
+def test_prefers_the_most_specific_usable_source():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "http://10.0.0.1:80"),
+        ("settings", "http://10.0.0.2:80"),
+        ("env", "http://10.0.0.3:80"),
+    ])
+    assert (source, url, target.host) == ("cluster", "http://10.0.0.1:80", "10.0.0.1")
+    assert skipped == []
+
+
+def test_skips_an_unusable_value_and_uses_the_next_source():
+    # The regression this guards: a scheme-less value left over from the era when the
+    # settings field was free text resolves to nothing. Stopping there would emit a
+    # backend section with no `server` line — `haproxy -c` passes, Apply succeeds, and
+    # every challenge request 503s with no visible cause.
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.90.1.4:8080"),
+        ("settings", ""),
+        ("env", "http://10.90.1.4:8080"),
+    ])
+    assert source == "env"
+    assert target.error_code is None and target.host == "10.90.1.4"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_empty_sources_are_skipped_without_being_reported():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""),
+        ("settings", None),
+        ("env", "http://10.0.0.9:80"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert skipped == []
+
+
+def test_reports_the_last_attempted_value_when_nothing_resolves():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.0.0.1:80"),
+        ("env", "not a url"),
+    ])
+    assert (source, url) == ("env", "not a url")
+    assert target.error_code, "the caller needs an error to render and log"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_all_sources_empty_yields_an_error_not_a_crash():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""), ("settings", ""), ("env", ""),
+    ])
+    assert source == "cluster" and target.error_code == "empty" and skipped == []
+
+
+def test_loopback_is_usable_enough_to_render():
+    # Warned about, never skipped: the shipped defaults are loopback, so treating it as
+    # unusable would make the fallback chain fall off its own end on a stock install.
+    source, _url, target, _skipped = select_acme_backend_source([
+        ("env", "http://localhost:8080"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert target.warnings
+
+
+# ---------------------------------------------------------------------------
+# 5. The generator's failure sentinel must never be mistaken for a config.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "content",
+    [
+        "# Error generating configuration: boom",
+        "# Error: Cluster not found",
+        "",
+        None,
+    ],
+)
+def test_detects_generation_failure_sentinels(content):
+    assert is_config_generation_error(content) is True
+
+
+@pytest.mark.parametrize("content", [_CONFIG, "global\n    daemon\n"])
+def test_real_configs_are_not_flagged(content):
+    assert is_config_generation_error(content) is False

backend/tests/test_acme_diagnostics.py

@@ -104,13 +104,30 @@ async def test_check_dns_empty_ips_marks_failure(monkeypatch):
 
 
 # ----------------------------------------------------------------------------
-# Port-80 check (HEAD probe)
+# Port-80 check (GET probe)
+#
+# The probe is a GET, not a HEAD: a reverse proxy that has lost its
+# /.well-known/acme-challenge/ location falls through to its catch-all and serves
+# an SPA with HTTP 200, which a status-code-only check accepts as healthy while
+# every real validation fails. The fakes below therefore carry a body.
 # ----------------------------------------------------------------------------
 
 
-class _FakeHEADResp:
-    def __init__(self, status):
+class _FakeContent:
+    def __init__(self, body):
+        self._body = body
+
+    async def read(self, n=-1):
+        if self._body is None:
+            raise ConnectionResetError("reset mid-body")
+        return self._body if n is None or n < 0 else self._body[:n]
+
+
+class _FakeGETResp:
+    def __init__(self, status, body=b"", content_type="text/plain"):
         self.status = status
+        self.headers = {"content-type": content_type}
+        self.content = _FakeContent(body)
 
     async def __aenter__(self):
         return self
@@ -120,10 +137,13 @@ class _FakeHEADResp:
 
 
 class _FakeSession:
-    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False):
+    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False,
+                 bodies=None, content_types=None):
         self._statuses = list(statuses or [])
         self._raise_timeout = raise_timeout
         self._raise_client_error = raise_client_error
+        self._bodies = list(bodies or [])
+        self._content_types = list(content_types or [])
 
     async def __aenter__(self):
         return self
@@ -131,14 +151,16 @@ class _FakeSession:
     async def __aexit__(self, *args):
         return False
 
-    def head(self, url, allow_redirects=False):
+    def get(self, url, allow_redirects=False):
         if self._raise_timeout:
             raise asyncio.TimeoutError()
         if self._raise_client_error:
             import aiohttp
             raise aiohttp.ClientError("connection refused")
         status = self._statuses.pop(0) if self._statuses else 200
-        return _FakeHEADResp(status)
+        body = self._bodies.pop(0) if self._bodies else b""
+        ctype = self._content_types.pop(0) if self._content_types else "text/plain"
+        return _FakeGETResp(status, body, ctype)
 
 
 def _mock_public_dns(monkeypatch, ip="93.184.216.34"):
@@ -178,6 +200,65 @@ async def test_check_port80_ok_on_404(monkeypatch):
     assert out["status"] == "ok"
 
 
+@pytest.mark.asyncio
+async def test_check_port80_warns_when_200_carries_a_web_page(monkeypatch):
+    """The failure that motivated the GET probe.
+
+    A reverse proxy whose /.well-known/acme-challenge/ location has drifted away
+    falls through to its catch-all and serves the SPA. The status is 200, so the old
+    `status in (200, 404)` rule called the install healthy while every validation
+    failed. Only the body distinguishes them.
+    """
+    _mock_public_dns(monkeypatch)
+    spa = b'<!doctype html><html><head><title>HAProxy OpenManager</title></head>'
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200], bodies=[spa], content_types=["text/html"])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    target = out["details"]["targets"][0]
+    assert target["body_class"] == "html"
+    assert not target.get("ok")
+    assert "HTML" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_port80_falls_back_to_status_when_the_body_cannot_be_read(monkeypatch):
+    """A body that cannot be read is missing evidence, not a verdict.
+
+    Turning a connection reset mid-response into a hard failure would make a healthy
+    404 fail intermittently, so the check keeps its original status-only semantics
+    whenever there is nothing to judge.
+    """
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[404], bodies=[None])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "ok"
+    assert out["details"]["targets"][0]["body_class"] == "unread"
+
+
+@pytest.mark.asyncio
+async def test_check_port80_warns_on_a_redirect(monkeypatch):
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[301])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    assert out["details"]["targets"][0]["diagnosis"]
+
+
 @pytest.mark.asyncio
 async def test_check_port80_warn_on_egress_timeout(monkeypatch):
     """Corporate egress blocks port 80 outbound — warn, don't fail."""
@@ -337,16 +418,92 @@ async def test_check_routing_fail_when_no_port80_frontend():
     assert "No HTTP frontend" in out["message"]
 
 
+def _routing_row(**over):
+    row = {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
+           "mode": "http", "default_backend": "be", "cluster_id": 1, "acme_enabled": True}
+    row.update(over)
+    return row
+
+
+def _applied(has_route=True, server_line="server _acme_mgmt 10.90.1.4:80"):
+    return {"has_route": has_route, "server_line": server_line}
+
+
 @pytest.mark.asyncio
-async def test_check_routing_ok_when_port80_frontend_present():
+async def test_check_routing_ok_when_challenge_route_is_in_the_applied_config():
+    # A port-80 frontend row alone is NOT enough. It describes what the database
+    # wants; the nodes run whatever was last applied. During the incident this
+    # function reported "ok" from the row count while the live config had no usable
+    # challenge route at all.
     conn = AsyncMock()
-    conn.fetch.return_value = [
-        {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
-         "mode": "http", "default_backend": "be"},
-    ]
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied()
     out = await check_routing(conn, ["a.example.com"], [1])
     assert out["status"] == "ok"
     assert len(out["details"]["frontends"]) == 1
+    assert out["details"]["challenge_backends"] == {1: "10.90.1.4:80"}
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_acme_is_disabled_on_the_cluster():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(acme_enabled=False)]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "disabled" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_route_is_not_in_the_applied_config():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(has_route=False)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "apply the cluster" in out["message"].lower()
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_has_no_server_line():
+    # `haproxy -c` passes because the section exists, so nothing else catches this;
+    # every challenge request 503s from an empty backend.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line=None)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "503" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_is_loopback():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line="server _acme_mgmt 127.0.0.1:8080")
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "HAProxy node" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_rather_than_fails_on_a_tcp_only_port80_cluster():
+    # The `fail` branch must stay reachable only when NO port-80 frontend exists at
+    # all: SiteWizard blocks submit on any failing check, so turning this into a
+    # failure would lock tcp-only installs the day it ships.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode="tcp")]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "tcp mode" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_treats_null_mode_as_http_like_the_renderer():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode=None)]
+    conn.fetchrow.return_value = _applied()
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "ok"
 
 
 # ----------------------------------------------------------------------------

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_csr_import.py

@@ -0,0 +1,469 @@
+"""
+v1.9.0 CSR creation — unit tests for the signed-certificate import flow and
+config-version staging (pattern: test_ssl_service_extraction.py, AsyncMock conn).
+
+Pins the security-relevant invariants:
+- key match is a HARD gate: match=False → 400 before any INSERT, and
+  match=None (unverifiable) → 500, never a lenient pass (we generated the
+  key ourselves — deliberate divergence from create_cert_row's fallback).
+- the new cert row is cluster_id=NULL / last_config_status='PENDING' /
+  source='csr' (PENDING keeps it invisible to agents until Apply).
+- completing the CSR NULLs the private key copy.
+- staging reuses the exact `ssl-{id}-create-{ts}` version-name scheme.
+"""
+import json
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from fastapi import HTTPException
+
+from models.csr import SSLCSRImport
+from services.csr_service import (
+    assert_csr_name_available,
+    import_signed_certificate,
+    insert_csr_row,
+)
+from services.ssl_service import stage_ssl_config_versions
+
+
+_VALID_PARSE = {
+    "primary_domain": "www.example.com",
+    "all_domains": ["www.example.com"],
+    "expiry_date": datetime(2099, 1, 1, tzinfo=timezone.utc),
+    "issuer": "CN=Test CA",
+    "fingerprint": "AA:BB:CC",
+    "status": "valid",
+    "days_until_expiry": 365,
+}
+
+_FAKE_CERT = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+_FAKE_KEY = "-----BEGIN PRIVATE KEY-----\nY\n-----END PRIVATE KEY-----"
+
+
+def _csr_row(**overrides):
+    row = {
+        "id": 5,
+        "name": "csr-www",
+        "common_name": "www.example.com",
+        "subject": "{}",
+        "sans": json.dumps(["www.example.com"]),
+        "key_algorithm": "rsa-2048",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nZ\n-----END CERTIFICATE REQUEST-----",
+        "private_key_pem": _FAKE_KEY,
+        "status": "pending",
+        "ssl_certificate_id": None,
+    }
+    row.update(overrides)
+    return row
+
+
+def _mk_conn():
+    conn = AsyncMock()
+    # asyncpg's conn.transaction() is a SYNC call returning an async CM.
+    conn.transaction = MagicMock()
+    return conn
+
+
+def _import_payload(**overrides):
+    base = dict(
+        certificate_content=_FAKE_CERT,
+        chain_content=None,
+        usage_type="frontend",
+        is_global=False,
+        cluster_ids=[1, 2],
+        name=None,
+    )
+    base.update(overrides)
+    return SSLCSRImport(**base)
+
+
+@contextmanager
+def _patched(match=None, parse=None):
+    """Patch every parser touchpoint of the import path: the function-local
+    imports in csr_service (utils.ssl_parser.*) and the module-level imports
+    in ssl_service._prepare_cert_fields (services.ssl_service.*)."""
+    match_result = match if match is not None else {"match": True}
+    parse_result = dict(parse or _VALID_PARSE)
+    with patch("utils.ssl_parser.verify_certificate_key_match", return_value=match_result), \
+         patch("utils.ssl_parser.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.validate_private_key", return_value=True), \
+         patch("services.ssl_service.validate_certificate_chain", return_value=True):
+        yield
+
+
+# ----------------------------------------------------------------------------
+# import_signed_certificate
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_import_happy_path_inserts_pending_csr_sourced_cert():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]  # FOR UPDATE row, no name clash
+    conn.fetchval.return_value = 42                 # INSERT ... RETURNING id
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert result["reactivated"] is False
+
+    # Concurrency invariants: everything runs inside a transaction and the
+    # CSR row is locked FOR UPDATE (serialises double-import and delete-races).
+    assert conn.transaction.call_count == 1
+    lock_sql = conn.fetchrow.call_args_list[0].args[0]
+    assert "FOR UPDATE" in lock_sql
+
+    insert_sql, *insert_args = conn.fetchval.call_args.args
+    assert "INSERT INTO ssl_certificates" in insert_sql
+    assert "NULL, 'PENDING'" in insert_sql, "cert must stay invisible to agents until Apply"
+    assert "'csr'" in insert_sql, "source column must record the CSR origin"
+    # The stored CSR key — not any request-supplied key — must be persisted.
+    assert _FAKE_KEY in insert_args
+
+    # One junction row per requested cluster.
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert len(junction_calls) == 2
+    assert {c.args[2] for c in junction_calls} == {1, 2}
+
+    # CSR completion must destroy the key copy.
+    completion_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_csrs" in c.args[0]
+    ]
+    assert len(completion_calls) == 1
+    assert "private_key_pem = NULL" in completion_calls[0].args[0]
+    assert "status = 'completed'" in completion_calls[0].args[0]
+    assert completion_calls[0].args[1] == 5   # csr_id
+    assert completion_calls[0].args[2] == 42  # cert_id
+
+
+@pytest.mark.asyncio
+async def test_import_global_creates_zero_junction_rows():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        await import_signed_certificate(
+            conn, 5, _import_payload(is_global=True, cluster_ids=None), user_id=7
+        )
+
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert junction_calls == [], "global cert = zero junction rows (existing convention)"
+
+
+@pytest.mark.asyncio
+async def test_import_key_mismatch_rejected_400_before_any_write():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": False, "reason": "public key mismatch"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "does not match" in exc_info.value.detail
+    assert not conn.fetchval.await_count, "nothing must be inserted on mismatch"
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_unverifiable_key_match_is_hard_error_not_lenient():
+    """match=None means OUR stored key is unreadable — integrity failure,
+    never the lenient pass create_cert_row historically allows."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": None, "reason": "key could not be parsed"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 500
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_expired_certificate_rejected_400():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    expired = dict(_VALID_PARSE)
+    expired["status"] = "expired"
+    expired["days_until_expiry"] = -10
+    with _patched(parse=expired):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "expired" in exc_info.value.detail.lower()
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_malformed_certificate_rejected_400_not_500():
+    """A cert with PEM markers but unparseable content (truncated CA response)
+    is OPERATOR INPUT — it must get the manual flow's 400, not the 500 that
+    the strict key-match branch reserves for a corrupt STORED key."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(parse={"error": "Could not parse certificate"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "Invalid SSL certificate" in exc_info.value.detail
+    assert not conn.fetchval.await_count
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_completed_csr_conflicts_409():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(status="completed", ssl_certificate_id=42)]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 409
+    assert "already completed" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_import_missing_csr_404():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [None]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 999, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_import_active_name_collision_rejected_with_hint():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 9, "is_active": True}]
+
+    with _patched():
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "already exists" in exc_info.value.detail
+    assert "name" in exc_info.value.detail  # points at the override escape hatch
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_name_override_is_used_for_the_cert_row():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        result = await import_signed_certificate(
+            conn, 5, _import_payload(name="renamed-cert"), user_id=7
+        )
+
+    assert result["certificate_name"] == "renamed-cert"
+    _, *insert_args = conn.fetchval.call_args.args
+    assert "renamed-cert" in insert_args
+    # And the collision check must have run against the override, not csr.name.
+    name_lookup = conn.fetchrow.call_args_list[1]
+    assert name_lookup.args[1] == "renamed-cert"
+
+
+@pytest.mark.asyncio
+async def test_import_reactivates_soft_deleted_name_and_warns():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 77, "is_active": False}]
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 77
+    assert result["reactivated"] is True
+    assert any("reactivated" in w for w in result["warnings"])
+    assert not conn.fetchval.await_count, "reactivation must UPDATE, not INSERT"
+    update_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_certificates" in c.args[0]
+    ]
+    assert len(update_calls) == 1
+    update_sql = update_calls[0].args[0]
+    assert "source = 'csr'" in update_sql
+    # The reactivated row must come back to life invisible to agents until
+    # Apply, with the row itself active again.
+    assert "last_config_status = 'PENDING'" in update_sql
+    assert "is_active = TRUE" in update_sql
+    # Old cluster bindings must be wiped before re-binding to the new scope.
+    junction_deletes = [
+        c for c in conn.execute.call_args_list
+        if c.args and "DELETE FROM ssl_certificate_clusters" in c.args[0]
+    ]
+    assert len(junction_deletes) == 1
+    assert junction_deletes[0].args[1] == 77
+    # …and the importer's requested clusters re-bound via the junction.
+    junction_inserts = [
+        c for c in conn.execute.call_args_list
+        if c.args and "INSERT INTO ssl_certificate_clusters" in c.args[0]
+    ]
+    assert {c.args[2] for c in junction_inserts} == {1, 2}
+
+
+@pytest.mark.asyncio
+async def test_import_san_drift_warns_but_succeeds():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [
+        _csr_row(sans=json.dumps(["www.example.com", "api.example.com"])),
+        None,
+    ]
+    conn.fetchval.return_value = 42
+
+    drifted = dict(_VALID_PARSE)
+    drifted["all_domains"] = ["www.example.com", "cdn.example.com"]
+    with _patched(parse=drifted):
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert any("added" in w and "cdn.example.com" in w for w in result["warnings"])
+    assert any("dropped" in w and "api.example.com" in w for w in result["warnings"])
+
+
+# ----------------------------------------------------------------------------
+# insert_csr_row / assert_csr_name_available
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_active_cert_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [11]  # active cert with the name exists
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "certificate" in exc_info.value.detail.lower()
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_pending_csr_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [None, 12]  # no cert, but a pending CSR
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "pending CSR" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_insert_csr_row_translates_unique_violation_to_400():
+    """The uq_ssl_csrs_name_pending partial index closes the create/create
+    race — the loser must get a clean 400, not a 500."""
+    import asyncpg as _asyncpg
+
+    conn = _mk_conn()
+    # availability checks pass, INSERT hits the unique index
+    conn.fetchval.side_effect = [
+        None, None, _asyncpg.exceptions.UniqueViolationError("dup"),
+    ]
+    payload = SimpleNamespace(
+        name="raced", common_name="www.example.com", key_algorithm="rsa-2048"
+    )
+    bundle = {"subject": {}, "sans": ["www.example.com"], "csr_pem": "PEM", "private_key_pem": "KEY"}
+
+    with pytest.raises(HTTPException) as exc_info:
+        await insert_csr_row(conn, payload, bundle, user_id=1)
+    assert exc_info.value.status_code == 400
+    assert "concurrent" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# router-level guards
+# ----------------------------------------------------------------------------
+
+
+def test_cluster_id_int32_guard_rejects_out_of_range_with_404():
+    """Body-supplied cluster ids must never reach asyncpg out of int4 range
+    (DataError → raw 500) — same Bulgu #96 hygiene as the csr_id path param."""
+    from routers.csr import _assert_valid_cluster_id
+
+    _assert_valid_cluster_id(1)
+    _assert_valid_cluster_id(2_147_483_647)
+    for bad in (0, -1, 2_147_483_648, 99_999_999_999):
+        with pytest.raises(HTTPException) as exc_info:
+            _assert_valid_cluster_id(bad)
+        assert exc_info.value.status_code == 404
+        assert "Cluster not found" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# stage_ssl_config_versions
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_stage_creates_one_pending_version_per_cluster_with_ssl_naming():
+    import re
+
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001  # config_versions INSERT RETURNING id
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(return_value="# cfg"),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert all(r["success"] for r in results)
+    assert [r["cluster_id"] for r in results] == [1, 2]
+
+    insert_calls = [
+        c for c in conn.fetchval.call_args_list
+        if c.args and "INSERT INTO config_versions" in c.args[0]
+    ]
+    assert len(insert_calls) == 2
+    for call in insert_calls:
+        sql = call.args[0]
+        assert "FALSE, 'PENDING'" in sql, "staged versions must be inactive + PENDING"
+        version_name = call.args[2]
+        # EXACT manual-flow scheme: Apply Management + has_pending_config
+        # LIKE-filters key off 'ssl-{id}-...'.
+        assert re.match(r"^ssl-42-create-\d+$", version_name), version_name
+        assert call.args[5] == 7  # created_by honours the importing user
+
+
+@pytest.mark.asyncio
+async def test_stage_reports_per_cluster_failure_without_raising():
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001
+
+    async def _gen(cluster_id):
+        if cluster_id == 2:
+            raise RuntimeError("config generation exploded")
+        return "# cfg"
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(side_effect=_gen),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert results[0]["success"] is True
+    assert results[1]["success"] is False
+    assert "exploded" in results[1]["error"]

backend/tests/test_csr_key_encryption.py

@@ -0,0 +1,129 @@
+"""Issue #53 (v1.10.1) — at-rest encryption for the pending CSR private key.
+
+Pure logic: no DB, no network. Covers the round-trip, the backward-compatible read of rows
+written before this release, the unrecoverable-key path after a key rotation, and a static
+assertion that the write path can no longer store a raw PEM.
+"""
+import os
+import re
+from pathlib import Path
+
+import pytest
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-csr-encryption-unit-tests")
+
+from cryptography.fernet import Fernet
+
+from utils.csr_key_crypto import (
+    decrypt_csr_private_key,
+    encrypt_csr_private_key,
+    is_encrypted,
+    reset_fernet_for_tests,
+)
+
+_SAMPLE_PEM = (
+    "-----BEGIN PRIVATE KEY-----\n"
+    "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj\n"
+    "-----END PRIVATE KEY-----\n"
+)
+
+
+def test_roundtrip_and_ciphertext_does_not_contain_the_key():
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+    # The stored form must not be the PEM, and must not leak any recognisable fragment of it.
+    assert token != _SAMPLE_PEM
+    assert "-----BEGIN" not in token
+    assert "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj" not in token
+    assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+
+
+def test_is_encrypted_discriminates_token_from_legacy_pem():
+    reset_fernet_for_tests()
+    assert is_encrypted(encrypt_csr_private_key(_SAMPLE_PEM)) is True
+    assert is_encrypted(_SAMPLE_PEM) is False
+    assert is_encrypted("") is False
+    assert is_encrypted(None) is False
+
+
+def test_legacy_plaintext_row_is_read_unchanged():
+    # Rows written before v1.10.1 hold a raw PEM. They must keep working with NO data migration,
+    # otherwise upgrading would strand every CSR that is out for signature.
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(_SAMPLE_PEM) == _SAMPLE_PEM
+
+
+def test_empty_or_missing_value_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(None) is None
+    assert decrypt_csr_private_key("") is None
+
+
+def test_key_rotation_makes_the_stored_key_unrecoverable_rather_than_wrong():
+    """After a rotation the caller must get None, never a silently wrong key."""
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+
+    # Rotate: an explicit, different CSR_ENCRYPTION_KEY takes precedence over the derived one.
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    os.environ["CSR_ENCRYPTION_KEY"] = Fernet.generate_key().decode()
+    try:
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) is None
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_explicit_env_key_is_used_and_survives_reset():
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    key = Fernet.generate_key().decode()
+    os.environ["CSR_ENCRYPTION_KEY"] = key
+    try:
+        reset_fernet_for_tests()
+        token = encrypt_csr_private_key(_SAMPLE_PEM)
+        # Decryptable with the same explicit key from a fresh instance...
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+        # ...and independently verifiable with the raw Fernet key.
+        assert Fernet(key.encode()).decrypt(token.encode()).decode() == _SAMPLE_PEM
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_derivation_uses_its_own_hkdf_info_string():
+    """Each secret class derives an independent key, so rotating one never affects another."""
+    src = (Path(__file__).resolve().parent.parent / "utils" / "csr_key_crypto.py").read_text()
+    assert b"csr-private-key-v1".decode() in src
+    # Must NOT reuse another class's info string.
+    for foreign in ("dns-provider-creds-v1", "vip-vrrp-secret-v1", "mfa-totp-secret-v1"):
+        assert foreign not in src, f"CSR key derivation must not reuse the {foreign} info string"
+
+
+def test_write_path_stores_the_encrypted_form_not_the_pem():
+    """Static pin: insert_csr_row must encrypt before the INSERT.
+
+    A future refactor that passed bundle['private_key_pem'] straight through would silently
+    reintroduce plaintext storage, and no unit test with a mocked connection would notice.
+    """
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    insert_fn = src[src.index("async def insert_csr_row("):]
+    insert_fn = insert_fn[: insert_fn.index("\nasync def ")]
+    assert "encrypt_csr_private_key(bundle['private_key_pem'])" in insert_fn
+    # The raw PEM must not be a bind parameter of the INSERT itself.
+    assert not re.search(r"^\s*bundle\['private_key_pem'\],\s*$", insert_fn, re.M)
+
+
+def test_import_path_decrypts_and_fails_closed_on_unrecoverable_key():
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    fn = src[src.index("async def import_signed_certificate("):]
+    assert "decrypt_csr_private_key(row['private_key_pem'])" in fn
+    # A None decrypt must raise rather than fall through to the key-match comparison.
+    assert "cannot be decrypted" in fn

backend/tests/test_csr_migration.py

@@ -0,0 +1,104 @@
+"""
+v1.9.0 CSR creation — static source assertions (pattern: test_vip_purge.py).
+
+Guards the migration wiring that a unit test cannot exercise without a real
+database: the SCHEMA_VERSION bump (without it, deployed installs skip the
+whole migration run and the ssl_csrs table never appears), the migration
+registration, the security-relevant DDL, and the router registration.
+"""
+import os
+import re
+
+_BACKEND_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel_path: str) -> str:
+    with open(os.path.join(_BACKEND_DIR, rel_path), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_schema_version_bumped_to_10():
+    src = _read(os.path.join("database", "migrations.py"))
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert m, "SCHEMA_VERSION constant not found in migrations.py"
+    assert int(m.group(1)) >= 10, (
+        "SCHEMA_VERSION must be >= 10 for the v1.9.0 ssl_csrs table — "
+        "without the bump, existing installs (version >= 9) skip the whole "
+        "migration run and never gain the table."
+    )
+
+
+def test_ssl_csrs_migration_defined_and_registered():
+    src = _read(os.path.join("database", "migrations.py"))
+    assert "async def ensure_ssl_csrs_table" in src
+
+    inner = src.split("async def _run_all_migrations_inner", 1)[1]
+    inner = inner.split("\nasync def ", 1)[0]  # body of the runner only
+    assert "await ensure_ssl_csrs_table()" in inner, (
+        "ensure_ssl_csrs_table must be invoked from _run_all_migrations_inner"
+    )
+
+
+def test_ssl_csrs_ddl_essentials():
+    src = _read(os.path.join("database", "migrations.py"))
+    ddl_start = src.index("CREATE TABLE IF NOT EXISTS ssl_csrs")
+    ddl = src[ddl_start:ddl_start + 2500]
+
+    assert "private_key_pem TEXT" in ddl
+    assert "name VARCHAR(100) NOT NULL" in ddl, (
+        "ssl_csrs.name must align with ssl_certificates.name VARCHAR(100)"
+    )
+    assert "ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL" in ddl, (
+        "deleting the imported cert must not cascade into CSR history"
+    )
+    # Partial unique index: only PENDING CSRs reserve their target cert name.
+    assert "uq_ssl_csrs_name_pending" in src
+    assert re.search(
+        r"uq_ssl_csrs_name_pending\s+ON\s+ssl_csrs\(name\)\s+WHERE\s+status\s*=\s*'pending'",
+        src,
+    ), "name uniqueness must be scoped to pending CSRs (partial index)"
+
+
+def test_csr_router_registered_in_main():
+    src = _read("main.py")
+    assert "from routers.csr import router as csr_router" in src
+    assert "app.include_router(csr_router)" in src
+
+
+def test_csr_endpoint_permission_mapping():
+    """Pin which ssl.<action> permission each endpoint enforces: a regression
+    that dropped or weakened a _require() call would otherwise pass the
+    auth-rejection tests (they only assert 401/403 for unauthenticated calls)."""
+    src = _read(os.path.join("routers", "csr.py"))
+
+    def _handler_body(decorator):
+        start = src.index(decorator)
+        nxt = src.find("@router.", start + 1)
+        return src[start:nxt if nxt != -1 else len(src)]
+
+    expectations = [
+        ('@router.post("")', '"create"'),
+        ('@router.get("")', '"read"'),
+        ('@router.get("/{csr_id}")', '"read"'),
+        ('@router.post("/{csr_id}/import")', '"create"'),
+        ('@router.delete("/{csr_id}")', '"delete"'),
+    ]
+    for decorator, action in expectations:
+        body = _handler_body(decorator)
+        assert f"_require(authorization, {action})" in body, (
+            f"endpoint {decorator} must enforce ssl.{action.strip(chr(34))}"
+        )
+
+
+def test_csr_router_never_selects_private_key():
+    """The CSR endpoints must use the explicit column list — a bare
+    `SELECT *` into an API response is how the key would leak. The one place
+    SELECT * is allowed is the service-layer FOR UPDATE row (it needs the key
+    to pair with the cert); the router itself must not touch the column."""
+    src = _read(os.path.join("routers", "csr.py"))
+    code_only = re.sub(r"#.*", "", src)  # strip comments; the column name may
+    # legitimately appear there as documentation
+    assert "private_key_pem" not in code_only, (
+        "routers/csr.py must never reference private_key_pem in code"
+    )
+    assert "SELECT *" not in code_only, "routers/csr.py must use explicit column lists"

backend/tests/test_csr_models.py

@@ -0,0 +1,172 @@
+"""
+v1.9.0 CSR creation — Pydantic model validation tests (models/csr.py).
+
+The CSR name shares the SSL certificate name's path-traversal contract
+(Bulgu #21) with one deliberate tightening: max 100 chars, matching the
+ssl_certificates.name VARCHAR(100) column.
+"""
+import pytest
+from pydantic import ValidationError
+
+from models.csr import SSLCSRCreate, SSLCSRImport
+
+_CERT_PEM = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+
+
+def _create(**overrides):
+    base = dict(name="my-csr", common_name="www.example.com")
+    base.update(overrides)
+    return SSLCSRCreate(**base)
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRCreate
+# ----------------------------------------------------------------------------
+
+
+def test_minimal_valid_create():
+    m = _create()
+    assert m.name == "my-csr"
+    assert m.common_name == "www.example.com"
+    assert m.key_algorithm == "rsa-2048"
+    assert m.sans == []
+
+
+@pytest.mark.parametrize("bad_name", [
+    "../../etc/cron.d/evil",   # path traversal
+    "a..b",                    # embedded ..
+    ".hidden",                 # hidden filename
+    "-flag",                   # CLI flag confusion
+    "has space",
+    "wild*card",
+    "",
+    "x" * 101,                 # VARCHAR(100) alignment — 200 is NOT allowed here
+])
+def test_name_rejects_unsafe_values(bad_name):
+    with pytest.raises(ValidationError):
+        _create(name=bad_name)
+
+
+def test_name_accepts_100_chars():
+    assert _create(name="x" * 100).name == "x" * 100
+
+
+def test_common_name_wildcard_accepted_and_lowercased():
+    m = _create(common_name="*.Example.COM")
+    assert m.common_name == "*.example.com"
+
+
+@pytest.mark.parametrize("bad_cn", [
+    "",
+    "under_score.example.com",      # _ is not LDH
+    "*.*.example.com",              # wildcard only as leftmost single label
+    "-leading.example.com",
+    "a" * 70 + ".example.com",      # label > 63
+    "cn-longer-than-64-chars-" + "x" * 45 + ".example.com",  # CN > 64 total
+])
+def test_common_name_rejects_invalid(bad_cn):
+    with pytest.raises(ValidationError):
+        _create(common_name=bad_cn)
+
+
+def test_sans_normalised_deduped_and_capped():
+    m = _create(sans=["API.example.com", "api.example.com", "cdn.example.com"])
+    assert m.sans == ["api.example.com", "cdn.example.com"]
+
+    with pytest.raises(ValidationError):
+        _create(sans=[f"h{i}.example.com" for i in range(101)])
+
+
+def test_country_normalised_or_rejected():
+    assert _create(country="tr").country == "TR"
+    assert _create(country=None).country is None
+    for bad in ("TUR", "T", "1A"):
+        with pytest.raises(ValidationError):
+            _create(country=bad)
+
+
+def test_subject_fields_reject_control_characters():
+    with pytest.raises(ValidationError):
+        _create(organization="Evil\x00Corp")
+    with pytest.raises(ValidationError):
+        _create(locality="line\nbreak")
+
+
+def test_subject_fields_reject_overlength():
+    with pytest.raises(ValidationError):
+        _create(organization="x" * 65)
+
+
+def test_key_algorithm_strict_enum():
+    for good in ("rsa-2048", "rsa-4096", "ecdsa-p256", "ecdsa-p384"):
+        assert _create(key_algorithm=good).key_algorithm == good
+    for bad in ("rsa-1024", "rsa-8192", "ed25519", "2048", ""):
+        with pytest.raises(ValidationError):
+            _create(key_algorithm=bad)
+
+
+def test_email_basic_validation():
+    assert _create(email="ops@example.com").email == "ops@example.com"
+    with pytest.raises(ValidationError):
+        _create(email="not-an-email")
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRImport
+# ----------------------------------------------------------------------------
+
+
+def test_import_minimal_global():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True)
+    assert m.usage_type == "frontend"
+    assert m.name is None
+
+
+def test_import_requires_clusters_when_not_global():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[])
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[1])
+    assert m.cluster_ids == [1]
+
+
+def test_import_certificate_must_be_pem():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="not a pem", is_global=True)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="", is_global=True)
+
+
+def test_import_certificate_size_capped():
+    huge = _CERT_PEM + "A" * (64 * 1024 + 1)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=huge, is_global=True)
+
+
+def test_import_chain_optional_but_validated():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, chain_content="  ")
+    assert m.chain_content is None
+    with pytest.raises(ValidationError):
+        SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, chain_content="garbage"
+        )
+
+
+def test_import_name_override_shares_the_name_contract():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="renamed")
+    assert m.name == "renamed"
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="../evil")
+    # Empty override collapses to None (falls back to the CSR's own name).
+    m2 = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="  ")
+    assert m2.name is None
+
+
+def test_import_usage_type_enum():
+    for good in ("frontend", "server"):
+        assert SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, usage_type=good
+        ).usage_type == good
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, usage_type="both")

backend/tests/test_csr_router_auth.py

@@ -0,0 +1,66 @@
+"""
+v1.9.0 CSR creation — behavioral auth tests for /api/ssl/csrs endpoints
+(pattern: test_ssl_list_endpoint_auth.py).
+
+Every CSR endpoint must refuse unauthenticated / garbage-token requests.
+The CSR detail route additionally must never 200 without auth because it
+returns the CSR PEM; no endpoint ever returns the private key, but auth is
+the first line regardless.
+"""
+import pytest
+
+_VALID_CREATE_BODY = {
+    "name": "auth-test-csr",
+    "common_name": "www.example.com",
+}
+
+_VALID_IMPORT_BODY = {
+    "certificate_content": (
+        "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+    ),
+    "is_global": True,
+}
+
+_ENDPOINTS = [
+    ("get", "/api/ssl/csrs", None),
+    ("get", "/api/ssl/csrs/1", None),
+    ("post", "/api/ssl/csrs", _VALID_CREATE_BODY),
+    ("post", "/api/ssl/csrs/1/import", _VALID_IMPORT_BODY),
+    ("delete", "/api/ssl/csrs/1", None),
+]
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_unauthenticated_rejected(client, method, path, body):
+    """No Authorization header → endpoint must refuse the request."""
+    res = getattr(client, method)(path, json=body) if body is not None else getattr(client, method)(path)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} without Authorization returned "
+        f"{res.status_code} — anonymous access to CSR data must not be "
+        f"possible. Body: {res.text[:200]}"
+    )
+    if res.status_code == 200:  # defensive, mirrors the R18 test style
+        data = res.json()
+        assert not data, "CSR endpoint returned data without auth"
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_invalid_token_rejected(client, method, path, body):
+    """Garbage token → endpoint must refuse the request."""
+    headers = {"Authorization": "Bearer not-a-valid-jwt"}
+    if body is not None:
+        res = getattr(client, method)(path, json=body, headers=headers)
+    else:
+        res = getattr(client, method)(path, headers=headers)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} with an invalid token returned {res.status_code}"
+    )
+
+
+def test_csr_routes_are_registered(client):
+    """The router must actually be mounted — a 404 would make the auth tests
+    above pass vacuously."""
+    res = client.get("/api/ssl/csrs")
+    assert res.status_code != 404, (
+        "GET /api/ssl/csrs returned 404 — csr_router is not registered in main.py"
+    )

backend/tests/test_csr_service_crypto.py

@@ -0,0 +1,171 @@
+"""
+v1.9.0 CSR creation — pure-crypto tests for services/csr_service.py.
+
+No mocks: every algorithm's output must parse with `cryptography` and the
+CSR's public key must match the generated private key (the property the
+whole import flow depends on).
+"""
+from types import SimpleNamespace
+
+import pytest
+
+from cryptography import x509
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import ExtensionOID, NameOID
+
+from services.csr_service import csr_row_to_dict, diff_domains, generate_csr_bundle
+
+
+def _payload(**overrides):
+    base = dict(
+        name="test-csr",
+        common_name="www.example.com",
+        organization=None,
+        organizational_unit=None,
+        locality=None,
+        state=None,
+        country=None,
+        email=None,
+        sans=[],
+        key_algorithm="rsa-2048",
+    )
+    base.update(overrides)
+    return SimpleNamespace(**base)
+
+
+def _spki(key):
+    return key.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+
+
+@pytest.mark.parametrize(
+    "algo,key_cls,key_check",
+    [
+        ("rsa-2048", rsa.RSAPrivateKey, lambda k: k.key_size == 2048),
+        ("rsa-4096", rsa.RSAPrivateKey, lambda k: k.key_size == 4096),
+        ("ecdsa-p256", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp256r1"),
+        ("ecdsa-p384", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp384r1"),
+    ],
+)
+def test_generate_bundle_all_algorithms(algo, key_cls, key_check):
+    bundle = generate_csr_bundle(_payload(key_algorithm=algo))
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    key = serialization.load_pem_private_key(
+        bundle["private_key_pem"].encode(), password=None
+    )
+
+    assert isinstance(key, key_cls)
+    assert key_check(key)
+    # The CSR must be signed by exactly this key.
+    csr_spki = csr.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+    assert csr_spki == _spki(key)
+    assert csr.is_signature_valid
+    # PKCS8, unencrypted — the agent concatenates cert+key into one PEM and
+    # HAProxy cannot read passphrase-protected keys.
+    assert bundle["private_key_pem"].startswith("-----BEGIN PRIVATE KEY-----")
+
+
+def test_subject_contains_all_provided_fields():
+    bundle = generate_csr_bundle(_payload(
+        organization="Example Corp",
+        organizational_unit="IT",
+        locality="Istanbul",
+        state="Marmara",
+        country="TR",
+        email="ops@example.com",
+    ))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+
+    def _one(oid):
+        attrs = csr.subject.get_attributes_for_oid(oid)
+        return attrs[0].value if attrs else None
+
+    assert _one(NameOID.COMMON_NAME) == "www.example.com"
+    assert _one(NameOID.ORGANIZATION_NAME) == "Example Corp"
+    assert _one(NameOID.ORGANIZATIONAL_UNIT_NAME) == "IT"
+    assert _one(NameOID.LOCALITY_NAME) == "Istanbul"
+    assert _one(NameOID.STATE_OR_PROVINCE_NAME) == "Marmara"
+    assert _one(NameOID.COUNTRY_NAME) == "TR"
+    assert _one(NameOID.EMAIL_ADDRESS) == "ops@example.com"
+    assert bundle["subject"] == {
+        "O": "Example Corp", "OU": "IT", "L": "Istanbul",
+        "ST": "Marmara", "C": "TR", "emailAddress": "ops@example.com",
+    }
+
+
+def test_subject_omits_empty_fields():
+    bundle = generate_csr_bundle(_payload())
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    assert not csr.subject.get_attributes_for_oid(NameOID.ORGANIZATION_NAME)
+    assert bundle["subject"] == {}
+
+
+def test_sans_cn_first_and_deduped():
+    bundle = generate_csr_bundle(_payload(
+        common_name="www.example.com",
+        sans=["api.example.com", "www.example.com", "api.example.com", "cdn.example.com"],
+    ))
+    assert bundle["sans"] == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    dns_names = san_ext.value.get_values_for_type(x509.DNSName)
+    assert dns_names == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+
+def test_wildcard_common_name_flows_into_san():
+    bundle = generate_csr_bundle(_payload(common_name="*.example.com"))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    assert san_ext.value.get_values_for_type(x509.DNSName) == ["*.example.com"]
+
+
+def test_diff_domains_reports_added_and_dropped():
+    warnings = diff_domains(
+        ["www.example.com", "api.example.com"],
+        ["WWW.example.com", "cdn.example.com"],
+    )
+    assert len(warnings) == 2
+    added = next(w for w in warnings if "added" in w)
+    dropped = next(w for w in warnings if "dropped" in w)
+    assert "cdn.example.com" in added
+    assert "api.example.com" in dropped
+    # Case-insensitive: www must NOT be reported in either direction.
+    assert "www.example.com" not in added
+    assert "www.example.com" not in dropped
+
+
+def test_diff_domains_identical_sets_yield_no_warnings():
+    assert diff_domains(["a.example.com"], ["A.EXAMPLE.COM"]) == []
+    assert diff_domains([], []) == []
+
+
+def test_csr_row_to_dict_never_exposes_private_key():
+    row = {
+        "id": 1,
+        "name": "x",
+        "private_key_pem": "-----BEGIN PRIVATE KEY-----\nSECRET\n-----END PRIVATE KEY-----",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nX\n-----END CERTIFICATE REQUEST-----",
+        "subject": '{"O": "Example"}',
+        "sans": '["a.example.com"]',
+    }
+    out = csr_row_to_dict(row)
+    assert "private_key_pem" not in out
+    assert "csr_pem" not in out          # lists exclude the PEM
+    assert out["subject"] == {"O": "Example"}
+    assert out["sans"] == ["a.example.com"]
+
+    detail = csr_row_to_dict(row, include_pem=True)
+    assert "private_key_pem" not in detail  # NEVER, even on detail
+    assert detail["csr_pem"].startswith("-----BEGIN CERTIFICATE REQUEST-----")

backend/tests/test_dns01.py

@@ -2,7 +2,8 @@
 
 Covers the TXT-value math (RFC 8555 §8.4 — raw SHA-256 digest, base64url, NOT hex),
 the _acme-challenge record-name derivation (wildcard stripping), credential encryption
-round-trip + tamper handling, and the DNS provider registry/allow-list.
+round-trip + tamper handling, the DNS provider registry/allow-list, and (v1.10.0) the
+GoDaddy provider's zone-relative name derivation and additive RRset merge math.
 """
 import base64
 import hashlib
@@ -53,9 +54,9 @@ def test_decrypt_invalid_token_returns_none():
 
 def test_provider_registry_and_allow_list():
     names = {p["name"] for p in list_providers()}
-    assert {"manual", "cloudflare"} <= names
-    assert is_supported("manual") and is_supported("cloudflare")
-    assert not is_supported("route53")  # not in MVP allow-list
+    assert {"manual", "cloudflare", "godaddy"} <= names
+    assert is_supported("manual") and is_supported("cloudflare") and is_supported("godaddy")
+    assert not is_supported("route53")  # not in the allow-list
 
     assert get_provider("manual").automated is False
     cf = get_provider("cloudflare", {"api_token": "x"})
@@ -89,3 +90,526 @@ def test_cloudflare_token_sanitize():
     p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
     assert p._token == 'my-token_123'
     assert p._raw_token == '"my-token_123"'
+
+
+# --- v1.10.0: GoDaddy provider (pure logic only — no network, no DB) ---
+
+
+def test_godaddy_credential_fields_schema():
+    # Re-assert DnsCredentialsUpsert's validator rules directly against the declared schema, so the
+    # UI can never render a field whose submission the API would reject with a 422.
+    import re
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    fields = GoDaddyDNSProvider.credential_fields
+    assert [f["key"] for f in fields] == ["api_key", "api_secret"]
+    for f in fields:
+        assert re.match(r"^[a-zA-Z0-9_]{1,50}$", f["key"])         # DnsCredentialsUpsert key regex
+        assert f["type"] == "password"                              # renders Input.Password, not Input
+        assert isinstance(f["max_length"], int) and 0 < f["max_length"] <= 4000  # validator value cap
+        assert f["help"] and isinstance(f["help"], str)             # shown in the Form.Item `extra` slot
+    # api_secret is optional on purpose: leaving it blank is how a Personal Access Token is used
+    # (Bearer), which is the migration path off the sso-key scheme GoDaddy is retiring.
+    assert fields[0]["required"] is True and fields[1]["required"] is False
+    # Must not reuse Cloudflare's field name: the register modal's credential Form.Items are named
+    # cred_<key> in a SHARED form and are not cleared when the provider dropdown changes.
+    assert "api_token" not in {f["key"] for f in fields}
+
+
+def test_godaddy_provider_is_automated():
+    p = get_provider("godaddy", {"api_key": "k", "api_secret": "s"})
+    assert p.automated is True      # else the orchestrator takes the manual-confirm branch
+    assert p.name == "godaddy" and 1 <= len(p.name) <= 50  # dns_provider Field(min_length=1, max_length=50)
+    assert p.label == "GoDaddy"
+
+
+def test_godaddy_missing_credentials_returns_not_ok():
+    # verify_credentials must RETURN {"ok": False}, never raise: the router turns any non-
+    # DnsProviderError into the information-free generic 422 and the user never sees the reason.
+    import asyncio
+
+    for creds in ({}, {"api_secret": "s"}):  # blank UI fields arrive as MISSING keys, not ""
+        r = asyncio.run(get_provider("godaddy", creds).verify_credentials())
+        assert r["ok"] is False and r["detail"]
+    # Short-circuits before any request, so this touches no network.
+
+
+def test_godaddy_auth_header_formats_and_secret_never_leaks():
+    from services.dns_providers.godaddy import GoDaddyDNSProvider, _scrub
+
+    sentinel = "SENTINEL-SECRET-DO-NOT-LEAK"
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": sentinel})
+    # Literal prefix, one space, a single colon — no base64, no quoting.
+    assert p._auth_header() == f"sso-key KEY123:{sentinel}"
+    # No secret -> Personal Access Token. This one branch is the whole sso-key-sunset migration.
+    assert GoDaddyDNSProvider({"api_key": "PAT"})._auth_header() == "Bearer PAT"
+    # _scrub removes credential substrings from anything bound for a log or an order event.
+    assert sentinel not in _scrub(f"boom {sentinel} boom", "KEY123", sentinel)
+    assert "KEY123" not in _scrub("boom KEY123", "KEY123", sentinel)
+    assert _scrub("x" * 500, "KEY123") == "x" * 300  # bounded, so a huge body can't flood an event
+
+    # The channel that actually persists text: _http_error composes the message an order event and
+    # letsencrypt_orders.error_detail will carry, so it must scrub its own inputs — a caller that
+    # forgets to pre-scrub must not be able to leak. (Regression guard: scrubbing used to live at
+    # the single call site in _request instead of here.)
+    exc = p._http_error(403, f"DENIED_{sentinel}", f"token {sentinel} rejected", None)
+    assert sentinel not in str(exc) and "***" in str(exc)
+
+    # This module must not log at all — logging is the one channel _scrub cannot reach, since the
+    # arguments would be formatted by the logging framework rather than passed through it.
+    import inspect
+    import re as _re
+    from services.dns_providers import godaddy as gd_mod
+
+    assert not _re.search(r"\blogger\.\w+\(", inspect.getsource(gd_mod)), \
+        "godaddy.py must not log; surface everything through DnsProviderError so it is scrubbed"
+
+
+def test_godaddy_relative_record_name():
+    # GoDaddy names are RELATIVE to the zone with no trailing dot; the apex is the literal "@".
+    from services.dns_providers.godaddy import _relative_name
+
+    assert _relative_name("_acme-challenge.example.com", "example.com") == "_acme-challenge"
+    assert _relative_name("_acme-challenge.foo.bar.example.com", "example.com") == "_acme-challenge.foo.bar"
+    assert _relative_name("example.com", "example.com") == "@"          # never "" — see _rrset_path
+    assert _relative_name("_acme-challenge.example.com.", "example.com") == "_acme-challenge"
+    assert _relative_name("_ACME-Challenge.Example.COM", "example.com") == "_acme-challenge"
+    # Apex and wildcard produce the SAME relative name — which is exactly why the merge below
+    # has to be additive.
+    apex = ACMEService._challenge_dns_name("example.com")
+    wild = ACMEService._challenge_dns_name("*.example.com")
+    assert _relative_name(apex, "example.com") == _relative_name(wild, "example.com") == "_acme-challenge"
+
+
+def test_godaddy_rrset_merge_is_additive():
+    # THE critical test: GoDaddy's PUT REPLACES an entire RRset, so the merge math is the only thing
+    # keeping a wildcard+apex certificate's two coexisting TXT values alive.
+    from services.dns_providers.godaddy import _live_values, _merge_add, _merge_remove
+
+    def vals(body):
+        return sorted(r["data"] for r in body)
+
+    assert vals(_merge_add([{"data": "valueA", "ttl": 600}], "valueB")) == ["valueA", "valueB"]
+    assert _merge_add([{"data": "valueA"}], "valueA") is None      # idempotent; ACME retries land here
+    # Total, not an all()-over-a-computed-list (which passes vacuously on an empty result): the
+    # first publish at a fresh name must emit exactly one element, carrying the 600s TTL floor.
+    assert _merge_add([], "v") == [{"data": "v", "ttl": 600}]      # below 600 GoDaddy answers 422
+    # Tombstone rows ({"data": ""}) must never be echoed back — GoDaddy answers 422 INVALID_BODY.
+    assert _live_values([{"data": ""}, {"data": "x"}, {}]) == ["x"]
+    assert vals(_merge_add([{"data": ""}, {"data": "valueA"}], "valueB")) == ["valueA", "valueB"]
+
+    assert vals(_merge_remove([{"data": "valueA"}, {"data": "valueB"}], "valueB")) == ["valueA"]
+    assert _merge_remove([{"data": "valueA"}], "valueZ") is None    # already gone — tolerate
+    assert _merge_remove([], "valueZ") is None
+    # [] means "use DELETE": PUT with an empty array is rejected (422, "Records must be specified").
+    assert _merge_remove([{"data": "valueA"}], "valueA") == []
+    assert _merge_remove([{"data": ""}, {"data": "valueA"}], "valueA") == []
+
+
+def test_godaddy_never_builds_a_zone_wide_txt_path():
+    # A 3-segment path (.../records/TXT) is the endpoint that wipes EVERY TXT in the zone — SPF,
+    # DKIM, DMARC, domain verifications. An empty relative name must never be able to produce it.
+    from services.dns_providers.godaddy import _rrset_path
+
+    p = _rrset_path("example.com", "_acme-challenge")
+    assert p == "/domains/example.com/records/TXT/_acme-challenge"
+    assert p.count("/") == 5 and not p.endswith("/TXT")
+    assert _rrset_path("example.com", "@").endswith("/%40")  # apex percent-encoded for proxy safety
+    # "." and ".." survive quote() and are then normalized away by yarl when the URL is built, so
+    # ".../records/TXT/.." would resolve to the whole-zone endpoint. They must be refused too.
+    for bad in [("example.com", ""), ("", "_acme-challenge"), ("example.com", "."),
+                ("example.com", ".."), ("example.com", "...")]:
+        raised = False
+        try:
+            _rrset_path(*bad)
+        except DnsProviderError:
+            raised = True
+        assert raised, f"_rrset_path{bad} must refuse to build a zone-wide TXT path"
+    # And the only way to reach those inputs — a malformed domain — really does produce them.
+    from services.dns_providers.godaddy import _relative_name as _rel
+    assert _rel("..example.com", "example.com") == "."
+
+
+def test_godaddy_credential_encryption_roundtrip():
+    # The two-field credential dict rides the same Fernet blob as Cloudflare's single token.
+    reset_fernet_for_tests()
+    creds = {"api_key": "gd-key-plaintext", "api_secret": "gd-secret-plaintext"}
+    token = encrypt_dns_credentials(creds)
+    assert "gd-key-plaintext" not in token and "gd-secret-plaintext" not in token  # ciphertext
+    assert decrypt_dns_credentials(token) == creds
+    # This sorted key list is exactly what GET /dns-credentials exposes as credential_fields_present
+    # — names only, never values.
+    assert sorted(decrypt_dns_credentials(token).keys()) == ["api_key", "api_secret"]
+
+
+_GD_NS = "/domains/example.com/records/NS"
+_GD_TXT = "/domains/example.com/records/TXT/_acme-challenge"
+
+
+def _gd_provider(responses):
+    """A GoDaddy provider whose _request is replaced by a recorder.
+
+    The pure-merge tests above prove the MATH; this proves the WRITE PATH actually uses it. Without
+    it, replacing the merge with a single-value PUT — the mutation that silently destroys the
+    sibling value of every wildcard+apex certificate — leaves the whole suite green.
+
+    `responses` maps (method, path) -> value to return, or an Exception to raise. Unmapped calls
+    return None, which is how the zone suffix-walk's failed probes are modelled.
+    """
+    import types
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    calls = []
+
+    async def _fake_request(self, session, method, path, **kwargs):
+        calls.append((method, path, kwargs.get("json")))
+        result = responses.get((method, path))
+        if isinstance(result, Exception):
+            raise result
+        return result
+
+    p = GoDaddyDNSProvider({"api_key": "k", "api_secret": "s"})
+    p._request = types.MethodType(_fake_request, p)
+    return p, calls
+
+
+def _assert_never_zone_wide(calls):
+    # A write to .../records or .../records/TXT replaces every TXT (or every record) in the zone.
+    for method, path, _json in calls:
+        if method in ("PUT", "DELETE"):
+            assert not path.endswith("/records"), f"zone-wide write: {method} {path}"
+            assert not path.endswith("/records/TXT"), f"type-wide write: {method} {path}"
+
+
+def test_godaddy_add_write_path_merges_siblings():
+    import asyncio
+
+    # An existing sibling value at the same name — the apex half of an apex+wildcard certificate.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+
+    writes = [c for c in calls if c[0] in ("PUT", "PATCH", "DELETE")]
+    assert len(writes) == 1 and writes[0][0] == "PUT" and writes[0][1] == _GD_TXT
+    # BOTH values must be in the body: GoDaddy's PUT replaces the whole RRset.
+    assert sorted(r["data"] for r in writes[0][2]) == ["valueA", "valueB"]
+    _assert_never_zone_wide(calls)
+
+
+def test_godaddy_add_write_path_is_idempotent_and_fails_closed():
+    import asyncio
+
+    # Already published -> no write at all (this is where an ACME retry cycle lands).
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueB", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    assert [c for c in calls if c[0] != "GET"] == []
+
+    # Unreadable RRset read (2xx whose body did not parse as a list) must FAIL, never be treated as
+    # an empty RRset — the PUT that follows would replace the sibling values with only ours.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): None,
+    })
+    raised = False
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    except DnsProviderError:
+        raised = True
+    assert raised, "an unreadable RRset read must not be coerced into an empty RRset"
+    assert [c for c in calls if c[0] != "GET"] == []
+
+
+def test_godaddy_remove_write_path_uses_delete_for_the_last_value():
+    import asyncio
+
+    # Two values -> PUT back the survivor only.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}, {"data": "valueB"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueB"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0][0] == "PUT"
+    assert [r["data"] for r in writes[0][2]] == ["valueA"]
+
+    # Last value -> DELETE. `PUT []` is rejected by GoDaddy (422 INVALID_BODY), so an empty PUT
+    # body would make every cleanup fail forever.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueA"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0] == ("DELETE", _GD_TXT, None)
+    assert not any(c[0] == "PUT" and c[2] == [] for c in calls)
+
+    # Value already gone -> no write, no error.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueZ"))
+    assert [c for c in calls if c[0] != "GET"] == []
+    _assert_never_zone_wide(calls)
+
+
+class _FakeGDResponse:
+    """Minimal stand-in for aiohttp's ClientResponse: status, headers, and json()."""
+
+    _NO_BODY = object()
+
+    def __init__(self, status, body=_NO_BODY, headers=None):
+        self.status = status
+        self._body = body
+        self.headers = headers or {}
+
+    async def json(self, content_type=None):
+        if self._body is _FakeGDResponse._NO_BODY:
+            raise ValueError("no body to decode")  # what an empty 204 does
+        return self._body
+
+
+class _FakeGDSession:
+    def __init__(self, response):
+        self._response = response
+        self.calls = []
+
+    def request(self, method, url, **kwargs):
+        self.calls.append((method, url, kwargs))
+        response = self._response
+
+        class _Ctx:
+            async def __aenter__(self_inner):
+                return response
+
+            async def __aexit__(self_inner, *exc):
+                return False
+
+        return _Ctx()
+
+
+def test_godaddy_request_status_handling():
+    import asyncio
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": "SEC456"})
+
+    def call(response):
+        session = _FakeGDSession(response)
+        try:
+            return asyncio.run(p._request(session, "PUT", "/domains/example.com/records/TXT/x",
+                                          json=[{"data": "v", "ttl": 600}])), None, session
+        except DnsProviderError as exc:
+            return None, str(exc), session
+
+    # 204 with an EMPTY body is the normal answer to every GoDaddy write — it must not raise.
+    body, err, session = call(_FakeGDResponse(204))
+    assert body is None and err is None
+    # Redirects are deliberately not followed (aiohttp would forward the Authorization header), so
+    # a 3xx is a FAILED call. Treating it as success would report a redirected write as a no-op.
+    _kw = session.calls[0][2]
+    assert _kw["allow_redirects"] is False
+    assert _kw["headers"]["Authorization"] == "sso-key KEY123:SEC456"
+    assert _kw["headers"]["Accept"] == "application/json"
+    for status in (301, 302, 307):
+        body, err, _ = call(_FakeGDResponse(status))
+        assert body is None and err and str(status) in err, f"HTTP {status} must not read as success"
+
+    # 200 with a list is passed through verbatim.
+    body, err, _ = call(_FakeGDResponse(200, [{"data": "v"}]))
+    assert err is None and body == [{"data": "v"}]
+
+    # Error mapping: each message must name what the operator has to fix.
+    _, err, _ = call(_FakeGDResponse(401, {"code": "UNABLE_TO_AUTHENTICATE", "message": "nope"}))
+    assert "PRODUCTION" in err and "UNABLE_TO_AUTHENTICATE" in err
+    _, err, _ = call(_FakeGDResponse(403, {"code": "ACCESS_DENIED", "message": "not allowed"}))
+    assert "domains.dns:update" in err
+    # 429: Retry-After wins; the legacy body field is the fallback; absent both -> 60s default.
+    _, err, _ = call(_FakeGDResponse(429, None, {"Retry-After": "17"}))
+    assert "~17s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"retryAfterSec": 42}))
+    assert "~42s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"Retry-After": "not-a-number"}))
+    assert "~60s" in err
+    # A non-dict error body must not crash the error mapper.
+    _, err, _ = call(_FakeGDResponse(500, "<html>gateway</html>"))
+    assert "500" in err
+
+    # A transport failure MID-READ must not be mistaken for "empty body". Only a decode error may
+    # be swallowed: a caller reading an RRset would otherwise see None and could take it for an
+    # empty record set, and the full-RRset PUT that follows would destroy the sibling values.
+    import aiohttp
+
+    class _TruncatedResponse(_FakeGDResponse):
+        async def json(self, content_type=None):
+            raise aiohttp.ClientPayloadError("connection closed mid-body")
+
+    body, err, _ = call(_TruncatedResponse(200))
+    assert body is None and err and "GoDaddy" in err
+
+
+def test_godaddy_zone_resolution_walks_suffixes_and_caches():
+    import asyncio
+    from services.dns_providers.godaddy import _GoDaddyHTTPError
+
+    # The deepest candidate is not a zone (404 = "not this zone"); the walk must continue to the
+    # registrable domain and then reuse it, so the second challenge at the same name costs no probe.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("nope", status=404, code="UNKNOWN_DOMAIN"),
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueA"))
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    probes = [c for c in calls if c[1].endswith("/records/NS")]
+    assert len(probes) == 2, "the resolved zone must be cached for the life of the provider"
+    # Relative name derived from the RESOLVED zone, never from the deepest candidate.
+    assert all(c[1] == _GD_TXT for c in calls if "/records/TXT/" in c[1])
+
+    # A credential/eligibility failure during the walk must surface, not be swallowed as
+    # "no managed domain" — otherwise the operator chases a DNS problem that is really a bad key.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("denied", status=403, code="ACCESS_DENIED"),
+    })
+    raised = ""
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "v"))
+    except DnsProviderError as exc:
+        raised = str(exc)
+    assert "denied" in raised and "No managed GoDaddy domain" not in raised
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/tests/test_haproxy_validator_bulgu12.py

@@ -206,41 +206,38 @@ def test_module_level_validate_haproxy_config_forwards_partial_fragment():
 
 
 # ──────────────────────────────────────────────────────────────────────
-# Wizard Pydantic gate: ACL `-f` flag must be REJECTED at submit.
+# Issue #38 follow-up: ACL `-f <file>` pattern-file references are
+# ACCEPTED (the Bulgu #12 hard reject was removed — pattern files are
+# operator-managed host files, the agent's pre-reload `haproxy -c`
+# makes a missing file fail safely, and bulk import always accepted
+# `-f`). These tests pin the ACCEPT behaviour.
 # ──────────────────────────────────────────────────────────────────────
 
 
-def test_wizard_pydantic_rejects_acl_with_file_flag():
-    """Pre-fix the wizard's ACL string validator passed
-    `acl name path -i -m reg -f /path` straight through. Apply-time
-    HAProxy `-c` then failed with "failed to open pattern file".
-    Pin that the validator now rejects `-f` at submit.
+def test_wizard_pydantic_accepts_acl_with_file_flag():
+    """Issue #38 follow-up — the wizard's ACL string validator must
+    ACCEPT `-f <file>` pattern-file references (Bulgu #12 reject
+    removed). Operators with large host-managed IP blacklists rely
+    on this in production.
     """
     from models.site_wizard import FrontendStep
 
-    # Minimal valid wizard frontend kwargs — only the offending
-    # acl_rules entry should trigger the failure.
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=["acl1 path -i -m reg -f /path"],
     )
-    from pydantic import ValidationError
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(**fe_kwargs)
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Bulgu #12 regression: ACL -f flag must be rejected with a "
-        f"clear pattern-file error. Got: {msg}"
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"], (
+        "ACL `-f` rule must round-trip verbatim through the wizard model"
     )
 
 
 @pytest.mark.parametrize(
     "rule",
     [
-        # Various spacing / position variants the regex must catch.
+        # Various spacing / position variants must all be accepted.
         "acl1 path -f /etc/haproxy/list",
         "acl1 path -i -f /tmp/x.lst",
         "acl1 src -f /etc/haproxy/admins.lst",
@@ -249,23 +246,19 @@ def test_wizard_pydantic_rejects_acl_with_file_flag():
         "acl1 path -f",
     ],
 )
-def test_wizard_pydantic_rejects_acl_with_file_flag_variants(rule):
-    """Every spacing / position variant the operator might type must
-    be rejected. Pinned defensively so the regex never accidentally
-    relaxes to "only matches trailing -f".
-    """
+def test_wizard_pydantic_accepts_acl_with_file_flag_variants(rule):
+    """Every spacing / position variant must be accepted verbatim
+    (Issue #38 follow-up — no `-f` shape may be rejected)."""
     from models.site_wizard import FrontendStep
-    from pydantic import ValidationError
 
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=[rule],
     )
-    with pytest.raises(ValidationError):
-        FrontendStep(**fe_kwargs)
+    assert fe.acl_rules == [rule]
 
 
 def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
@@ -291,42 +284,29 @@ def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
     assert len(fe.acl_rules) == 3
 
 
-def test_manual_frontend_validator_rejects_acl_with_file_flag():
-    """Parity check: the manual Frontend API
-    (`models/frontend.py::validate_acl_rules`) must apply the same
-    `-f` rejection. Operators see consistent behaviour from both the
-    wizard and the per-entity frontend page.
+def test_manual_frontend_validator_accepts_acl_with_file_flag():
+    """Parity check (Issue #38 follow-up): the manual Frontend API
+    (`models/frontend.py::validate_acl_rules`) must ACCEPT `-f`
+    pattern-file references, same as the wizard and bulk import.
     """
     from models.frontend import FrontendConfig
-    from pydantic import ValidationError
 
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendConfig(
-            name="fe1",
-            bind_port=80,
-            mode="http",
-            acl_rules=["acl1 path -i -m reg -f /path"],
-        )
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Manual frontend API parity regression: ACL -f flag must be "
-        f"rejected. Got: {msg}"
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        acl_rules=["acl1 path -i -m reg -f /path"],
     )
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"]
 
 
-def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
-    """Round-3 audit extension — structured redirect dicts (the
-    alternative shape that `models/site_wizard.py::_validate_redirect_rules`
-    accepts alongside legacy strings) also flow through to
-    `services/haproxy_config.py::_format_redirect_rule` and emit
-    their `condition` / `target` verbatim into the rendered HAProxy
-    directive. Without the dict-aware reject the visual builder's
-    `-f` block could be bypassed by hand-crafting a dict payload
-    against the API — recreating the same `failed to open pattern
-    file` failure at apply time.
+def test_wizard_pydantic_accepts_structured_redirect_dict_with_file_flag():
+    """Issue #38 follow-up — structured redirect dicts carrying `-f`
+    pattern-file references in `condition`/`target` are ACCEPTED
+    (the Bulgu #12 dict-aware reject was removed together with the
+    string-rule reject).
     """
-    from models.site_wizard import FrontendStep, BackendStep
-    from pydantic import ValidationError
+    from models.site_wizard import FrontendStep
 
     fe_kwargs = dict(
         name="fe1",
@@ -334,37 +314,32 @@ def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
         mode="http",
     )
 
-    # `condition` carrying `-f` must be rejected.
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "scheme",
-                    "target": "https",
-                    "condition": "if { src -f /etc/haproxy/admins.lst }",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `condition` carrying `-f` is accepted.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "scheme",
+                "target": "https",
+                "condition": "if { src -f /etc/haproxy/admins.lst }",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/admins.lst }"
 
-    # `target` carrying `-f` must also be rejected (defence-in-depth
-    # for hand-crafted payloads).
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "location",
-                    "target": "/foo -f /tmp/x.lst",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `target` carrying `-f` is accepted too.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "location",
+                "target": "/foo -f /tmp/x.lst",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["target"] == "/foo -f /tmp/x.lst"
 
-    # Clean structured dict still passes — no false positive.
+    # Clean structured dict still passes.
     FrontendStep(
         **fe_kwargs,
         redirect_rules=[
@@ -667,8 +642,8 @@ def test_user_reported_wizard_config_emits_no_false_warnings():
     zero WARNINGs from the directives we expanded.
     """
     # Distilled from the user's bulk-site-create snapshot, minus the
-    # `-f` ACL (which the new Pydantic gate rejects before this
-    # validator ever runs).
+    # `-f` ACL (accepted since the Issue #38 follow-up, but irrelevant
+    # to the directive-expansion warnings this test pins).
     config = """# ─── Wizard candidate fragment (dry-run preview) ───
 frontend fe-site1
     bind *:80

backend/tests/test_keepalived_parser.py

@@ -0,0 +1,341 @@
+"""Issue #27 follow-up (v1.10.4) — unit tests for parsing an EXISTING keepalived.conf so a
+hand-maintained VIP can be adopted.
+
+Pure-function tests; no DB, no network. The parser exists because the heartbeat carries only
+the VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+fields — and because adoption REPLACES the operator's file, so anything our renderer cannot
+reproduce has to be reported as a blocker rather than silently dropped.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-keepalived-parser-tests")
+
+from services import keepalived_config as kc  # noqa: E402
+from services.keepalived_parser import (  # noqa: E402
+    KeepalivedParseError, analyse_keepalived_conf, build_adoption_candidate,
+    parse_keepalived_conf,
+)
+
+
+# A realistic hand-maintained config: two nodes, unicast VRRP, password auth, HAProxy check.
+HANDWRITTEN = """\
+! Configuration File for keepalived
+global_defs {
+    enable_script_security
+    script_user root
+}
+
+vrrp_script chk_haproxy {
+    script "/etc/keepalived/check_haproxy.sh"
+    interval 2
+    weight -21
+}
+
+vrrp_instance VI_1 {
+    state MASTER
+    interface eth0        # public leg
+    virtual_router_id 51
+    priority 150
+    advert_int 1
+    authentication {
+        auth_type PASS
+        auth_pass s3cr3t
+    }
+    unicast_src_ip 10.0.0.11
+    unicast_peer {
+        10.0.0.12
+    }
+    virtual_ipaddress {
+        10.0.0.100/24 dev eth0
+    }
+    track_script {
+        chk_haproxy
+    }
+}
+"""
+
+
+def _only_candidate(text):
+    parsed = parse_keepalived_conf(text)
+    assert len(parsed["instances"]) == 1
+    return build_adoption_candidate(parsed, parsed["instances"][0])
+
+
+def test_parses_a_handwritten_config_into_model_fields():
+    cand = _only_candidate(HANDWRITTEN)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["blockers"] == []
+    assert cand["vip"] == {
+        "virtual_ip": "10.0.0.100",
+        "prefix_length": 24,
+        "virtual_router_id": 51,
+        "advert_int": 1,
+        "use_unicast": True,
+        "track_haproxy": True,
+        "auth_pass": "s3cr3t",
+    }
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["peers"] == ["10.0.0.12"] and cand["unicast_src_ip"] == "10.0.0.11"
+    assert cand["defaulted"] == []          # every value came from the file, nothing assumed
+
+
+def test_comment_and_layout_variants():
+    # `!` and `#` both start comments; a block may open and close on one line; a quoted
+    # script path keeps its spaces. None of this may change the parse.
+    text = """\
+#!/not/a/shebang — this whole line is a comment
+vrrp_script chk { script "/opt/my scripts/chk.sh" }
+vrrp_instance VI_1 { state BACKUP
+    interface eth1   ! trailing bang comment
+    virtual_router_id 7
+    priority 90
+    virtual_ipaddress { 192.168.5.9/32 dev eth1 }
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert parsed["scripts"]["chk"]["script"] == "/opt/my scripts/chk.sh"
+    inst = parsed["instances"][0]
+    assert inst["state"] == "BACKUP" and inst["interface"] == "eth1"
+    assert inst["virtual_router_id"] == 7 and inst["priority"] == 90
+    assert inst["virtual_ips"] == [
+        {"address": "192.168.5.9", "prefix_length": 32, "dev": "eth1", "extra": []}
+    ]
+
+
+def test_documented_defaults_are_applied_and_flagged():
+    # keepalived's own defaults for absent directives. Applying them re-renders the same
+    # behaviour, so they are allowed — but the UI must be able to say they were assumed.
+    text = """\
+vrrp_instance VI_1 {
+    interface eth0
+    virtual_router_id 12
+    virtual_ipaddress { 10.1.1.5/24 dev eth0 }
+}
+"""
+    cand = _only_candidate(text)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["member"]["role"] == "BACKUP" and cand["member"]["priority"] == 100
+    assert cand["vip"]["advert_int"] == 1
+    assert sorted(cand["defaulted"]) == ["advert_int", "priority", "state"]
+    # No authentication block and no track_script — both legal, both faithfully represented.
+    assert cand["vip"]["auth_pass"] is None and cand["vip"]["track_haproxy"] is False
+    assert cand["vip"]["use_unicast"] is False
+
+
+def _blockers_for(text):
+    return " | ".join(_only_candidate(text)["blockers"])
+
+
+def test_directives_we_cannot_render_block_adoption():
+    # THE central safety property: adoption overwrites the file, so a failover hook we do not
+    # render would be destroyed. It must stop the flow, not warn.
+    text = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _blockers_for(text)
+    assert "notify_master" in blockers and "would delete it" in blockers
+    assert _only_candidate(text)["adoptable"] is False
+
+
+def test_multiple_addresses_in_one_instance_block_adoption():
+    text = HANDWRITTEN.replace("        10.0.0.100/24 dev eth0",
+                               "        10.0.0.100/24 dev eth0\n        10.0.0.101/24 dev eth0")
+    blockers = _blockers_for(text)
+    assert "2 addresses" in blockers and "10.0.0.101" in blockers
+
+
+def test_missing_vrid_blocks_adoption_with_the_split_brain_reason():
+    text = HANDWRITTEN.replace("    virtual_router_id 51\n", "")
+    blockers = _blockers_for(text)
+    assert "no virtual_router_id" in blockers and "separate VRRP domains" in blockers
+
+
+def test_missing_prefix_blocks_adoption():
+    # Our renderer always writes an explicit prefix; guessing one would change the netmask of a
+    # live VIP, so the operator has to state it.
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _blockers_for(text)
+    assert "no explicit prefix length" in blockers
+
+
+def test_address_on_a_different_dev_blocks_adoption():
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100/24 dev eth1")
+    blockers = _blockers_for(text)
+    assert "dev eth1" in blockers and "interface eth0" in blockers
+
+
+def test_foreign_track_script_blocks_adoption():
+    text = HANDWRITTEN.replace("        chk_haproxy", "        chk_custom")
+    blockers = _blockers_for(text)
+    assert "chk_custom" in blockers and "replaced by OpenManager" in blockers
+
+
+def test_unsupported_auth_type_blocks_adoption():
+    text = HANDWRITTEN.replace("auth_type PASS", "auth_type AH")
+    assert "auth_type AH" in _blockers_for(text)
+
+
+def test_fractional_advert_int_blocks_adoption():
+    # Rounding 0.5s to 1s changes VRRP timing, so adopt-and-alter is not acceptable.
+    text = HANDWRITTEN.replace("advert_int 1", "advert_int 0.5")
+    blockers = _blockers_for(text)
+    assert "advert_int 0.5" in blockers and "fractional" in blockers
+
+
+def test_half_configured_unicast_blocks_adoption():
+    text = HANDWRITTEN.replace("    unicast_peer {\n        10.0.0.12\n    }\n", "")
+    assert "unicast_src_ip without unicast_peer" in _blockers_for(text)
+
+
+def test_sync_group_and_lvs_sections_block_adoption():
+    text = HANDWRITTEN + """
+vrrp_sync_group VG1 {
+    group {
+        VI_1
+    }
+}
+virtual_server 10.0.0.100 80 {
+    lb_algo rr
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert [g["name"] for g in parsed["sync_groups"]] == ["VG1"]
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "vrrp_sync_group VG1" in directives and "virtual_server" in directives
+    # Both are top-level, so EVERY candidate in the file is blocked — a sync group changes
+    # failover semantics for the instances it groups.
+    cand = build_adoption_candidate(parsed, parsed["instances"][0])
+    assert cand["adoptable"] is False
+
+
+def test_extra_global_defs_are_reported_as_losses():
+    text = HANDWRITTEN.replace("    script_user root",
+                               "    script_user root\n    router_id LVS_DEVEL")
+    parsed = parse_keepalived_conf(text)
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "global_defs/router_id LVS_DEVEL" in directives
+    assert parsed["global_defs"]["router_id"] == "LVS_DEVEL"
+
+
+def test_multiple_instances_yield_one_candidate_each():
+    text = HANDWRITTEN + """
+vrrp_instance VI_2 {
+    state BACKUP
+    interface eth0
+    virtual_router_id 52
+    priority 100
+    advert_int 1
+    virtual_ipaddress { 10.0.0.200/24 dev eth0 }
+}
+"""
+    analysed = analyse_keepalived_conf(text)
+    assert analysed["instance_count"] == 2
+    names = [c["instance_name"] for c in analysed["candidates"]]
+    assert names == ["VI_1", "VI_2"]
+    assert [c["vip"]["virtual_ip"] for c in analysed["candidates"]] == ["10.0.0.100", "10.0.0.200"]
+    assert all(c["adoptable"] for c in analysed["candidates"])
+
+
+def test_unbalanced_braces_raise():
+    for bad in ("vrrp_instance VI_1 {\n    state MASTER\n", "}\n"):
+        raised = False
+        try:
+            parse_keepalived_conf(bad)
+        except KeepalivedParseError:
+            raised = True
+        assert raised, f"should have raised for {bad!r}"
+
+
+def test_our_own_render_round_trips_with_zero_blockers():
+    """The invariant that keeps the parser honest: a config WE generated must parse back into
+    the same model with nothing unsupported. If a future change to render_keepalived_conf emits
+    a directive the parser does not know, this fails — instead of adoption silently reporting
+    that OpenManager's own output is unadoptable."""
+    vip = {"id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+           "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True}
+    members = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                "agent_id": 1, "ip_address": "10.0.0.11"},
+               {"role": "BACKUP", "priority": 100, "network_interface": "eth0",
+                "agent_id": 2, "ip_address": "10.0.0.12"}]
+    rendered = kc.render_keepalived_conf(
+        vip=vip, members=members, this_agent=members[0],
+        peer_ips=["10.0.0.12"], auth_pass_plain="s3cr3t")
+
+    cand = _only_candidate(rendered)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["vip"]["virtual_ip"] == vip["virtual_ip"]
+    assert cand["vip"]["prefix_length"] == vip["prefix_length"]
+    assert cand["vip"]["virtual_router_id"] == vip["virtual_router_id"]
+    assert cand["vip"]["track_haproxy"] is True and cand["vip"]["use_unicast"] is True
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["vip"]["auth_pass"] == "s3cr3t"
+
+    # And the same for the no-auth / multicast / untracked shape, which renders fewer blocks.
+    plain = kc.render_keepalived_conf(
+        vip={**vip, "use_unicast": False, "track_haproxy": False},
+        members=members, this_agent=members[1], peer_ips=[], auth_pass_plain=None)
+    cand2 = _only_candidate(plain)
+    assert cand2["adoptable"] is True, cand2["blockers"]
+    assert cand2["vip"]["use_unicast"] is False and cand2["vip"]["track_haproxy"] is False
+    assert cand2["vip"]["auth_pass"] is None
+
+
+# --- v1.10.4 adoption gate: which blockers an operator may resolve --------------------------
+
+
+def test_only_prefix_and_data_loss_are_waivable():
+    from services.keepalived_parser import remaining_blockers
+
+    loss = "line 9: `notify_master \"/x.sh\"` — OpenManager's renderer cannot reproduce this, so adopting would delete it"
+    prefix = "`10.0.0.5` has no explicit prefix length; state it during adoption so the netmask cannot change on takeover"
+    hard_vrid = "no virtual_router_id — it cannot be guessed: a wrong VRID puts the nodes in separate VRRP domains"
+    hard_auth = "auth_type AH is not supported (only PASS is rendered)"
+    all_four = [loss, prefix, hard_vrid, hard_auth]
+
+    # Nothing waived: everything survives.
+    assert remaining_blockers(all_four) == all_four
+    # A supplied prefix resolves ONLY the prefix blocker.
+    assert remaining_blockers(all_four, prefix_supplied=True) == [loss, hard_vrid, hard_auth]
+    # Accepting data loss resolves ONLY the loss blocker.
+    assert remaining_blockers(all_four, accept_data_loss=True) == [prefix, hard_vrid, hard_auth]
+    # Both together still cannot wave through an impossibility — this is the property that stops
+    # a UI flag from destroying a VIP whose VRID or auth_type we could not reproduce.
+    assert remaining_blockers(all_four, prefix_supplied=True, accept_data_loss=True) == \
+        [hard_vrid, hard_auth]
+    # And an adoptable candidate stays adoptable.
+    assert remaining_blockers([]) == []
+
+
+def test_waiver_markers_match_the_messages_the_parser_actually_emits():
+    # The gate matches on substrings of the blocker prose, so a reworded message would silently
+    # stop being waivable. Pin both directions against real parser output.
+    from services.keepalived_parser import remaining_blockers
+
+    no_prefix = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _only_candidate(no_prefix)["blockers"]
+    assert blockers, "expected a prefix blocker"
+    assert remaining_blockers(blockers, prefix_supplied=True) == []
+
+    with_hook = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _only_candidate(with_hook)["blockers"]
+    assert blockers, "expected a data-loss blocker"
+    assert remaining_blockers(blockers, accept_data_loss=True) == []
+
+
+def test_auth_pass_masking_leaves_no_trace_of_the_secret():
+    # The discovered config is stored and served to the UI, so the ingest endpoint masks the VRRP
+    # password. Reuse the router's own regex so the test breaks if it is loosened.
+    from routers.agent import _AUTH_PASS_MASK_RE
+
+    secret = "s3cr3t with spaces"
+    text = HANDWRITTEN.replace("auth_pass s3cr3t", f"auth_pass {secret}")
+    masked = _AUTH_PASS_MASK_RE.sub(r"\1********", text)
+    assert secret not in masked and "s3cr3t" not in masked
+    assert "auth_pass ********" in masked
+    # Everything else survives, so the preview is still useful.
+    assert "virtual_router_id 51" in masked and "10.0.0.100/24 dev eth0" in masked

backend/tests/test_outbound_instrumentation.py

@@ -0,0 +1,327 @@
+"""v1.11.0: every outbound HTTP call is recorded, and instrumentation can never
+become the failure.
+
+Two independent risks:
+
+**Secrets.** The outbound calls carry the most sensitive material in the
+system: the ACME JWS (a replayable signed capability for the lifetime of its
+nonce) and the DNS provider API credentials. Those call sites must opt out of
+request-body capture and out of verbatim error text — the tests below assert
+that at the call site, not just in the helper.
+
+**Availability.** Both DNS provider funnels end in
+`except Exception: raise DnsProviderError("Unexpected ... failure")`, and in
+GoDaddy's publish path that reverts `dns_record_published` and stalls the ACME
+order. So an exception escaping `outbound_span` would be reported to the
+operator as a provider outage. It must never raise — and it must never swallow.
+"""
+import asyncio
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+from unittest.mock import patch  # noqa: E402
+
+from utils import http_instrumentation  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.http_instrumentation import (  # noqa: E402
+    TARGET_ACME,
+    TARGET_DNS_CLOUDFLARE,
+    TARGET_DNS_GODADDY,
+    outbound_span,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(*parts):
+    with open(os.path.join(_BACKEND, *parts), encoding="utf-8") as f:
+        return f.read()
+
+
+def _function_body(src, signature):
+    start = src.index(signature)
+    rest = src[start:]
+    # Next def at the same or lower indentation ends the body.
+    end = rest.find("\n    async def ", 1)
+    alt = rest.find("\n    def ", 1)
+    if alt != -1 and (end == -1 or alt < end):
+        end = alt
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    rows = []
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", rows.append)
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(http_instrumentation, "get_config", lambda: request_log_settings._CACHE)
+    return rows
+
+
+# --------------------------------------------------------------------------
+# outbound_span behaviour
+# --------------------------------------------------------------------------
+
+def test_records_a_successful_call(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="POST",
+                                 url="https://acme-v02.api.letsencrypt.org/acme/new-order") as span:
+            span.set_response(201, {"content-type": "application/json"}, {"status": "pending"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.direction == "outbound"
+    assert row.target == TARGET_ACME
+    assert row.method == "POST"
+    assert row.status_code == 201
+    assert row.status_class == 2
+    assert row.response_body_value == {"status": "pending"}
+
+
+def test_exception_is_recorded_and_reraised_unchanged(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com/x"):
+            raise ValueError("connection reset")
+
+    with pytest.raises(ValueError, match="connection reset"):
+        asyncio.run(run())
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a call that never got a response must be status_class 0 — the sentinel the "
+        "error-retention window keys off"
+    )
+    assert row.error.startswith("ValueError")
+
+
+def test_safe_error_only_records_the_type_not_the_message(captured):
+    async def run():
+        async with outbound_span(target=TARGET_DNS_GODADDY, method="PUT",
+                                 url="https://api.godaddy.com/v1/domains/example.com/records/TXT/_acme-challenge",
+                                 safe_error_only=True):
+            raise RuntimeError("failed talking to https://api.godaddy.com/v1/domains/secret-zone")
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(run())
+
+    assert captured[0].error == "RuntimeError"
+    assert "secret-zone" not in (captured[0].error or "")
+
+
+def test_instrumentation_failure_never_becomes_a_provider_failure(captured, monkeypatch):
+    """A bug in row construction must not surface to the operator as
+    'Unexpected GoDaddy API failure' and stall an ACME order."""
+    def explode(row):
+        raise RuntimeError("sink is broken")
+
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", explode)
+
+    async def run():
+        async with outbound_span(target=TARGET_DNS_CLOUDFLARE, method="GET",
+                                 url="https://api.cloudflare.com/client/v4/zones") as span:
+            span.set_response(200, {}, {"success": True})
+            return "provider-result"
+
+    assert asyncio.run(run()) == "provider-result", (
+        "a broken sink propagated out of outbound_span; both DNS funnels would convert "
+        "that into DnsProviderError('Unexpected ... failure'), and in GoDaddy's publish "
+        "path that reverts dns_record_published and stalls the ACME order"
+    )
+
+
+def test_block_exception_still_propagates_when_the_sink_is_broken(monkeypatch):
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer",
+                        lambda row: (_ for _ in ()).throw(RuntimeError("sink is broken")))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com"):
+            raise KeyError("original")
+
+    with pytest.raises(KeyError, match="original"):
+        asyncio.run(run())
+
+
+def test_capture_body_false_stores_the_summary_not_the_payload(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_ACME, method="POST", url="https://acme/new-order",
+            request_body={"jws": True, "kid_present": True, "payload_empty": False},
+            capture_body=False,
+        ) as span:
+            span.set_response(200, {}, {"status": "valid"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.request_body_value == {"jws": True, "kid_present": True, "payload_empty": False}
+    assert row.request_body_raw is None
+    # The CA's RESPONSE is still captured — that is the half operators need.
+    assert row.response_body_value == {"status": "valid"}
+
+
+def test_urls_are_scrubbed_before_storage(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_DNS_CLOUDFLARE, method="GET",
+            url="https://user:hunter2@api.cloudflare.com/client/v4/zones?api_key=abc&page=1",
+        ) as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+
+    url = captured[0].url
+    assert "hunter2" not in url
+    assert "abc" not in url
+    assert "page=1" in url
+
+
+def test_outbound_rows_inherit_the_inbound_request_id(captured):
+    from utils.request_log_sink import request_id_context
+
+    async def run():
+        token = request_id_context.set("abc123def456")
+        try:
+            async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+                span.set_response(200, {}, {})
+        finally:
+            request_id_context.reset(token)
+
+    asyncio.run(run())
+
+    assert captured[0].request_id == "abc123def456", (
+        "an outbound call must carry the inbound request's id, otherwise the detail view "
+        "cannot show which API call triggered which CA/DNS call"
+    )
+
+
+def test_background_calls_get_a_task_scoped_id(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured[0].request_id.startswith("bg:")
+
+
+def test_disabled_outbound_capture_produces_no_row(captured, monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE",
+                        replace(DEFAULT_CONFIG, capture_outbound=False))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            # The call site keeps working — set_response must still be callable.
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured == []
+
+
+def test_set_response_tolerates_a_response_without_headers(captured):
+    """Some call sites are driven in tests by minimal fakes exposing only
+    `.status`."""
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="HEAD", url="https://acme/nonce") as span:
+            span.set_response(200, None)
+
+    asyncio.run(run())
+    assert captured[0].status_code == 200
+
+
+# --------------------------------------------------------------------------
+# Call-site coverage
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path,target", [
+    (("services", "acme_service.py"), "TARGET_ACME"),
+    (("services", "acme_diagnostics.py"), "TARGET_ACME_DIAG"),
+    (("services", "dns_providers", "cloudflare.py"), "TARGET_DNS_CLOUDFLARE"),
+    (("services", "dns_providers", "godaddy.py"), "TARGET_DNS_GODADDY"),
+    (("routers", "letsencrypt.py"), "TARGET_LETSENCRYPT_CA"),
+    (("routers", "settings.py"), "TARGET_SETTINGS_PROBE"),
+    (("haproxy_client.py",), "TARGET_HAPROXY_STATS"),
+    (("agent_notifications.py",), "TARGET_AGENT"),
+])
+def test_every_outbound_module_is_instrumented(path, target):
+    src = _read(*path)
+    assert "outbound_span(" in src, f"{'/'.join(path)} makes HTTP calls but records nothing"
+    assert target in src, f"{'/'.join(path)} does not tag its rows with {target}"
+
+
+def test_acme_signed_request_never_captures_the_jws_body():
+    """The JWS body is {protected, payload, signature}: `protected` carries the
+    nonce and account kid, `signature` is made with the account private key. A
+    stored (protected, signature) pair is a replayable ACME credential."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    assert "capture_body=False" in body, (
+        "the ACME JWS request body would be written to request_logs verbatim — that is a "
+        "replayable signed credential sitting in an audit table"
+    )
+    assert '"jws": True' in body, "no synthetic summary replaces the suppressed JWS body"
+
+
+def test_acme_span_is_inside_the_retry_loop():
+    """The session is built outside `for attempt in range(3)`; the span must be
+    inside it, so a badNonce retry is its own row rather than being folded into
+    the successful attempt."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    loop_at = body.index("for attempt in range(3):")
+    span_at = body.index("async with outbound_span(")
+    assert loop_at < span_at, (
+        "outbound_span wraps the retry loop instead of sitting inside it, so three "
+        "attempts collapse into one log row and a nonce retry becomes invisible"
+    )
+
+
+@pytest.mark.parametrize("path", [
+    ("services", "dns_providers", "cloudflare.py"),
+    ("services", "dns_providers", "godaddy.py"),
+])
+def test_dns_providers_record_error_types_only(path):
+    src = _read(*path)
+    body = _function_body(src, "    async def _request(")
+    assert "safe_error_only=True" in body, (
+        f"{'/'.join(path)} would record the full exception text, which can carry the "
+        f"request URL and through it the tenant/zone identifier"
+    )
+
+
+def test_godaddy_narrow_value_error_handling_is_preserved():
+    """R-round hardening: only a JSON decode failure may be swallowed. Widening
+    it would make a mid-read transport failure look like an empty RRset, and the
+    follow-up full-RRset PUT would then destroy coexisting TXT values."""
+    src = _read("services", "dns_providers", "godaddy.py")
+    body = _function_body(src, "    async def _request(")
+    assert "except ValueError:" in body
+    assert "except Exception:\n                        body = None" not in body
+
+
+def test_acme_diagnostics_keeps_its_ipv4_pinned_connector():
+    """Duplicates an existing assertion on purpose: instrumenting this module
+    must not have refactored the SSRF-guard connector away."""
+    src = _read("services", "acme_diagnostics.py")
+    assert "TCPConnector(family=socket.AF_INET" in src, (
+        "the IPv4 pin was removed from the port-80 probe — that reopens the dual-stack "
+        "AAAA bypass the SSRF guard closes"
+    )
+
+
+def test_haproxy_stats_never_logs_basic_auth_or_the_csv():
+    """aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+    password, and a full stats CSV has no audit value."""
+    src = _read("haproxy_client.py")
+    body = _function_body(src, "    async def _get_stats_via_http(")
+    assert "capture_body=False" in body
+    assert "capture_response_body=False" in body
+    assert "auth=auth" in body and "request_body=auth" not in body

backend/tests/test_request_log_fleet_scale.py

@@ -0,0 +1,327 @@
+"""v1.11.0: the log's cost must follow operator activity, not fleet size.
+
+Every property here was a real defect measured on the feature branch, and each
+one only shows up at scale or at the edge of a setting's documented range, which
+is why none of them were caught by the rule-level tests.
+
+  * one row per API call becomes millions per day once the fleet is a few
+    hundred nodes, and the row cap then evicts the forensic history the feature
+    exists for;
+  * the operator role could not see the rows its grant was written for;
+  * every background call ever made shared one correlation id;
+  * queue memory was a function of an operator-editable setting, not a limit.
+"""
+import asyncio
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils.http_instrumentation import _correlation_id, begin_background_trace  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    get_config,
+    set_config,
+)
+from utils.request_log_sink import (  # noqa: E402
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    RequestLogSink,
+    request_id_context,
+)
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MIDDLEWARE = os.path.join(_BACKEND, "middleware", "request_logger.py")
+
+
+@pytest.fixture(autouse=True)
+def _restore_config():
+    """These tests mutate the module-global snapshot; put it back."""
+    before = get_config()
+    yield
+    set_config(before)
+
+
+def _row(**kw):
+    kw.setdefault("request_id", "a" * 32)
+    kw.setdefault("direction", "inbound")
+    kw.setdefault("method", "GET")
+    kw.setdefault("url", "/api/agents/prod-lb-1/config")
+    kw.setdefault("status_code", 200)
+    return RequestLogRow(**kw)
+
+
+class _CountingSink(RequestLogSink):
+    """Counts what survives `offer()` without needing an event loop."""
+
+    def __init__(self, **kw):
+        super().__init__(kw.pop("maxsize", 10000), 100, 500, **kw)
+        self.accepted = []
+
+    def _ensure_queue(self):
+        sink = self
+
+        class _Q:
+            def put_nowait(self, row):
+                sink.accepted.append(row)
+
+            def qsize(self):
+                return len(sink.accepted)
+
+        return _Q()
+
+
+# --------------------------------------------------------------------------
+# Volume: successful agent polls are not rows
+# --------------------------------------------------------------------------
+
+def test_successful_agent_polls_are_dropped_by_default():
+    """~9 800 rows/day PER AGENT, all of them 200s meaning "nothing changed".
+
+    At 200 nodes that is ~2M rows/day and the 500 000 row cap is reached in
+    about six hours, so the configured "7 days of successes, 30 days of
+    failures" silently becomes about six hours of each — for everything in the
+    table, not just for the agent rows.
+    """
+    assert DEFAULT_CONFIG.capture_agent_success is False, (
+        "the default must be off; on, the table's size is a function of node "
+        "count rather than of anything anyone did"
+    )
+    sink = _CountingSink()
+    for _ in range(100):
+        sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert sink.accepted == []
+
+
+@pytest.mark.parametrize("status", [401, 422, 500, None])
+def test_failed_agent_calls_are_always_kept(status):
+    """The half an operator actually needs, and rare enough to be free.
+
+    `None` is a transport error with no HTTP response at all, which
+    status_class reports as 0.
+    """
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=status))
+    assert len(sink.accepted) == 1, f"a {status} agent call must be recorded"
+
+
+def test_operator_traffic_is_unaffected_by_the_agent_gate():
+    sink = _CountingSink()
+    sink.offer(_row(target=None, status_code=200, user_id=7))
+    assert len(sink.accepted) == 1
+
+
+def test_the_gate_can_be_turned_on_for_debugging():
+    set_config(replace(get_config(), capture_agent_success=True))
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert len(sink.accepted) == 1
+
+
+def test_agent_traffic_is_identified_by_headers_not_by_a_database_lookup():
+    """The hot path runs on every request; a lookup per call is not affordable.
+
+    The installed agent sends `X-API-Key` and never `Authorization`; the UI
+    sends a JWT and never an agent key.
+    """
+    from middleware.request_logger import _is_agent_call
+
+    def scope(headers):
+        return {"type": "http", "headers": [(k.encode(), v.encode()) for k, v in headers.items()]}
+
+    assert _is_agent_call(scope({"x-api-key": "agt_x"})) is True
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z"})) is False
+    # generate-install-script accepts either; self-upgrade sends only the key.
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z", "x-api-key": "agt_x"})) is False
+    assert _is_agent_call(scope({})) is False
+
+
+def test_agent_gate_does_not_reach_for_a_connection():
+    """`offer()` is called from the request coroutine and must stay pure."""
+    src = open(os.path.join(_BACKEND, "utils", "request_log_sink.py"), encoding="utf-8").read()
+    body = src.split("def offer(", 1)[1].split("\n    # -- consumer", 1)[0]
+    for forbidden in ("await ", "get_database_connection", "fetch"):
+        assert forbidden not in body, f"offer() must not {forbidden.strip()!r} — it runs on the hot path"
+
+
+# --------------------------------------------------------------------------
+# Visibility: the operator grant has to mean something
+# --------------------------------------------------------------------------
+
+def test_read_only_scoping_admits_agent_rows_but_not_other_users():
+    """`operator` holds requestlog.read to "debug failing applies" — but an
+    apply fails on the NODE, and the node reports over its own API key, so that
+    row has user_id NULL and own-rows-only scoping hid it.
+
+    Keyed on `target`, NOT on `user_id IS NULL`: anonymous traffic (failed
+    logins and their usernames, unauthenticated probes) is not agent traffic
+    and must stay admin-only.
+    """
+    src = open(_ROUTER, encoding="utf-8").read()
+    clause = re.search(r"if not can_manage:(.*?)where_sql =", src, re.S)
+    assert clause, "the self-scoping block moved; re-check this test"
+    # Code only: the comment above the clause explains what it deliberately
+    # does NOT do, and would otherwise match the negative assertion below.
+    body = "\n".join(
+        line for line in clause.group(1).splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "TARGET_INBOUND_AGENT" in body, "agent rows are still hidden from requestlog.read"
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL would also expose anonymous traffic, including failed "
+        "logins and the usernames they carry"
+    )
+
+
+def test_detail_endpoint_uses_the_same_scoping_rule_as_the_list():
+    src = open(_ROUTER, encoding="utf-8").read()
+    detail = src.split('@router.get("/{log_id}")', 1)[1]
+    assert "TARGET_INBOUND_AGENT" in detail, (
+        "the detail endpoint would 404 on the very rows the list now shows"
+    )
+
+
+@pytest.mark.parametrize("decorator", [
+    '@router.get("/settings")', '@router.put("/settings")',
+    '@router.get("/stats")', '@router.post("/purge")',
+    '@router.get("")', '@router.get("/{log_id}")',
+])
+def test_permission_is_enforced_before_the_try_block(decorator):
+    """The repo's GHSA-3p5c pattern: a permission check inside `try` gets
+    swallowed by the handler's own `except Exception -> 500`, turning a 403
+    into a server error and, worse, hiding that the check ran at all."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    body = src.split(decorator, 1)[1]
+    body = body.split("\n@router.")[0]
+    require_at = body.find("_require(authorization")
+    try_at = body.find("\n    try:")
+    assert require_at != -1, f"{decorator} does not call _require at all"
+    assert try_at == -1 or require_at < try_at, (
+        f"{decorator} checks permissions INSIDE its try block"
+    )
+
+
+# --------------------------------------------------------------------------
+# Correlation: a trace that groups the wrong rows is worse than no trace
+# --------------------------------------------------------------------------
+
+def test_each_background_pass_gets_its_own_correlation_id():
+    """Nothing in main.py names its tasks, so the old `bg:<task name>` fallback
+    gave one long-lived loop a single id for its entire life — measured, 15
+    ACME calls across 5 ticks came out as 1 id. `related` (LIMIT 100) then
+    presents up to a hundred unrelated calls as this request's trace.
+    """
+    async def loop():
+        per_tick = []
+        for _ in range(5):
+            begin_background_trace("acme_renewals")
+            per_tick.append([_correlation_id() for _ in range(3)])
+            await asyncio.sleep(0)
+        return per_tick
+
+    ticks = asyncio.run(loop())
+    for tick in ticks:
+        assert len(set(tick)) == 1, "calls within one pass must share an id"
+    ids = [t[0] for t in ticks]
+    assert len(set(ids)) == 5, f"passes must not share an id, got {ids}"
+
+
+def test_unwrapped_background_code_does_not_collapse_onto_one_id():
+    """Erring toward too little grouping: a row that stands alone is honest, a
+    row falsely grouped with a hundred others is not."""
+    async def unwrapped():
+        request_id_context.set(None)
+        return [_correlation_id() for _ in range(4)]
+
+    ids = asyncio.run(unwrapped())
+    assert len(set(ids)) == 4
+
+
+def test_the_background_loops_that_make_outbound_calls_open_a_trace():
+    src = open(os.path.join(_BACKEND, "main.py"), encoding="utf-8").read()
+    for loop_name in ("complete_pending_acme_orders", "check_letsencrypt_renewals",
+                      "monitor_agent_status"):
+        body = src.split(f"async def {loop_name}", 1)[1].split("\nasync def ")[0]
+        assert "begin_background_trace(" in body, (
+            f"{loop_name} makes outbound calls but never opens a per-pass trace"
+        )
+
+
+# --------------------------------------------------------------------------
+# Memory: a limit, not a setting
+# --------------------------------------------------------------------------
+
+def test_queue_memory_is_bounded_even_at_the_max_body_size_ceiling():
+    """`max_body_bytes` is editable from Settings and its documented ceiling is
+    256 KB, which a row carries twice. Against the default 2 000-row queue that
+    is ~1 GiB — the entire pod limit — reachable from in-range values.
+    """
+    set_config(replace(get_config(), max_body_bytes=262144, capture_agent_success=True))
+    budget = 8 * 1024 * 1024
+    sink = _CountingSink(maxsize=2000, max_bytes=budget)
+    blob = b"x" * 262144
+    for _ in range(2000):
+        sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+
+    held = sum(r.queue_weight() for r in sink.accepted)
+    assert held <= budget, f"queue held {held} bytes against a {budget} byte budget"
+    assert sink.stats["dropped"] > 0, "over-budget rows must be dropped, and counted"
+    unbounded = 2000 * (2 * 262144 + 1400)
+    assert held < unbounded / 10, (
+        f"without the byte budget this queue would hold {unbounded // 1024 // 1024} MiB"
+    )
+
+
+_DRAIN_ROWS = 50
+
+
+def test_the_byte_budget_is_released_as_rows_drain():
+    """A budget that only ever counts up is a slow leak, not a limit.
+
+    Deliberately time-INDEPENDENT. `_collect()` stops at whichever comes first,
+    `batch_size` rows or the `flush_ms` deadline, so a batch size larger than
+    the row count makes the result a function of how fast the runner happens to
+    be. The first version of this test used batch=100/flush=10ms for 50 rows and
+    passed on a native build while failing in CI, which builds
+    linux/amd64 + linux/arm64 and therefore runs one of them under qemu
+    emulation: 25 iterations of `asyncio.wait_for` were enough to exhaust 10 ms
+    there, `_collect()` returned half a batch, and the assertion read
+    `239800 == 0` - measuring the scheduler, not the sink.
+
+    So: batch size EQUAL to the row count, so the loop exits on the count and
+    never consults the deadline; a generous flush window in case it somehow
+    does; and a drain loop rather than a single call. Nothing here depends on
+    wall-clock speed.
+    """
+    async def drain():
+        sink = RequestLogSink(2000, _DRAIN_ROWS, 5000, max_bytes=8 * 1024 * 1024)
+        blob = b"x" * 4096
+        set_config(replace(get_config(), capture_agent_success=True))
+        for _ in range(_DRAIN_ROWS):
+            sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+        assert sink.stats["queued_bytes"] > 0, "nothing was queued, so nothing is being measured"
+        assert sink._queue.qsize() == _DRAIN_ROWS, "the queue did not take every row"
+
+        drained = 0
+        while not sink._queue.empty():
+            drained += len(await sink._collect())
+        assert drained == _DRAIN_ROWS, f"drained {drained} of {_DRAIN_ROWS} rows"
+        return sink.stats["queued_bytes"]
+
+    assert asyncio.run(drain()) == 0
+
+
+def test_stats_say_the_sink_counters_are_per_worker():
+    """The sink is a module global; with UVICORN_WORKERS > 1 each process keeps
+    its own. A number that looks fleet-wide but is not understates drops by
+    exactly the worker count."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    assert '"scope"' in src.split('"sink"', 1)[1][:400], (
+        "the stats response must label the sink counters as this-worker-only"
+    )

backend/tests/test_request_log_middleware.py

@@ -0,0 +1,432 @@
+"""v1.11.0: the request/response logger must be invisible to everything below it.
+
+This is the highest-risk piece of the feature. A logging middleware that reads
+the request body the naive way DRAINS the ASGI receive channel, and the handler
+underneath then sees an empty body — `POST /api/agents/heartbeat` reads the raw
+stream itself, so every agent in the fleet would start failing its heartbeat
+because someone wanted nicer logs.
+
+The implementation therefore TEES rather than consumes. These tests drive the
+middleware over a stub ASGI app and assert that property directly: the
+downstream app sees the full body, the client sees the full response, and only
+a capped copy is kept.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from middleware.request_logger import RequestResponseLogMiddleware  # noqa: E402
+from utils.logging_config import correlation_id_context  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+
+
+_MAIN = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "main.py")
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    """Collect the rows the middleware hands to the sink, instead of writing them."""
+    rows = []
+    monkeypatch.setattr(sink_module.request_log_sink, "offer", rows.append)
+    # The middleware imports `request_log_sink` by value, so patch there too.
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl.request_log_sink, "offer", rows.append)
+    return rows
+
+
+@pytest.fixture(autouse=True)
+def default_config(monkeypatch):
+    """Every test starts from the shipped defaults, with a small body cap so the
+    truncation paths are exercised without megabyte fixtures."""
+    cfg = replace(DEFAULT_CONFIG, max_body_bytes=1024)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl, "get_config", lambda: request_log_settings._CACHE)
+    return cfg
+
+
+def set_config(monkeypatch, **overrides):
+    cfg = replace(request_log_settings._CACHE, **overrides)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    return cfg
+
+
+# --------------------------------------------------------------------------
+# A minimal ASGI harness — no TestClient, no HTTP stack, just the protocol.
+# --------------------------------------------------------------------------
+
+async def drive(app, *, method="POST", path="/api/backends", body=b"", query=b"",
+                headers=None, content_type="application/json"):
+    """Run one request through `app` and return (status, headers, body)."""
+    raw_headers = [(b"host", b"testserver")]
+    if content_type:
+        raw_headers.append((b"content-type", content_type.encode()))
+    for k, v in (headers or {}).items():
+        raw_headers.append((k.encode().lower(), v.encode()))
+
+    scope = {
+        "type": "http",
+        "asgi": {"version": "3.0"},
+        "http_version": "1.1",
+        "method": method,
+        "scheme": "http",
+        "path": path,
+        "raw_path": path.encode(),
+        "query_string": query,
+        "root_path": "",
+        "headers": raw_headers,
+        "client": ("10.1.2.3", 51234),
+        "server": ("testserver", 80),
+    }
+
+    # Deliver the body in three chunks so the tee is exercised across messages.
+    chunks = [body[i:i + max(1, len(body) // 3 or 1)] for i in range(0, len(body), max(1, len(body) // 3 or 1))] or [b""]
+    pending = list(chunks)
+
+    async def receive():
+        if pending:
+            chunk = pending.pop(0)
+            return {"type": "http.request", "body": chunk, "more_body": bool(pending)}
+        return {"type": "http.request", "body": b"", "more_body": False}
+
+    sent = {"status": None, "headers": [], "body": b""}
+
+    async def send(message):
+        if message["type"] == "http.response.start":
+            sent["status"] = message["status"]
+            sent["headers"] = message.get("headers", [])
+        elif message["type"] == "http.response.body":
+            sent["body"] += message.get("body", b"") or b""
+
+    await app(scope, receive, send)
+    return sent
+
+
+def echo_length_app(status=200, content_type=b"application/json"):
+    """Stub app that CONSUMES the whole request body and reports its length.
+
+    This is the regression shape: if the middleware drained the stream, the app
+    below it would see 0 bytes.
+    """
+    async def app(scope, receive, send):
+        total = 0
+        while True:
+            message = await receive()
+            total += len(message.get("body", b"") or b"")
+            if not message.get("more_body"):
+                break
+        payload = json.dumps({"received_bytes": total}).encode()
+        await send({"type": "http.response.start", "status": status,
+                    "headers": [(b"content-type", content_type)]})
+        await send({"type": "http.response.body", "body": payload})
+    return app
+
+
+def chunked_app(chunks, content_type=b"application/json"):
+    async def app(scope, receive, send):
+        await send({"type": "http.response.start", "status": 200,
+                    "headers": [(b"content-type", content_type)]})
+        for i, chunk in enumerate(chunks):
+            await send({"type": "http.response.body", "body": chunk,
+                        "more_body": i < len(chunks) - 1})
+    return app
+
+
+# --------------------------------------------------------------------------
+# The transparency guarantees
+# --------------------------------------------------------------------------
+
+def test_request_body_reaches_downstream_intact(captured):
+    """THE regression guard: draining the receive channel would break the raw-body
+    agent heartbeat handler."""
+    body = b"x" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=body))
+
+    assert json.loads(sent["body"])["received_bytes"] == 100_000, (
+        "the handler below the logger saw a different body length than the client sent — "
+        "the middleware consumed the receive channel instead of teeing it"
+    )
+
+
+def test_response_body_reaches_client_intact(captured):
+    chunks = [b'{"part":', b'"one",', b'"n":2}']
+    app = RequestResponseLogMiddleware(chunked_app(chunks))
+
+    sent = asyncio.run(drive(app, method="GET", body=b""))
+
+    assert sent["body"] == b"".join(chunks), "a response chunk was swallowed by the logger"
+    assert sent["status"] == 200
+
+
+def test_only_the_capped_prefix_is_captured(captured):
+    body = b"y" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=body))
+
+    row = captured[0]
+    assert row.request_body_bytes == 100_000, "the on-the-wire size must be recorded in full"
+    assert len(row.request_body_raw) <= 1024, (
+        "the middleware buffered more than max_body_bytes — memory is unbounded per request"
+    )
+
+
+def test_non_capturable_content_type_is_counted_but_not_buffered(captured):
+    app = RequestResponseLogMiddleware(chunked_app([b"\x00\x01\x02" * 500],
+                                                   content_type=b"application/octet-stream"))
+
+    asyncio.run(drive(app, method="GET", content_type=None))
+
+    row = captured[0]
+    assert row.response_body_bytes == 1500
+    assert row.response_body_raw is None, (
+        "a binary response body was buffered — this is what keeps streaming/file "
+        "responses safe"
+    )
+
+
+# --------------------------------------------------------------------------
+# What gets logged, and what does not
+# --------------------------------------------------------------------------
+
+def test_basic_row_fields(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="POST", path="/api/backends",
+                      body=b'{"name":"web"}', query=b"cluster_id=2&token=secret"))
+
+    row = captured[0]
+    assert row.direction == "inbound"
+    assert row.method == "POST"
+    assert row.path == "/api/backends"
+    assert row.status_code == 200
+    assert row.status_class == 2
+    assert row.client_ip == "10.1.2.3"
+    assert row.duration_ms >= 0
+    # The query string is scrubbed before it is stored, in the URL and the dict.
+    assert "secret" not in row.url
+    assert row.query_params["token"] == "***REDACTED***"
+    assert row.query_params["cluster_id"] == "2"
+
+
+@pytest.mark.parametrize("path", [
+    "/api/health",
+    "/api/health/deep",
+    "/api/docs",
+    "/api/openapi.json",
+    "/.well-known/acme-challenge/abc123",
+    "/api/agents/heartbeat",
+    "/favicon.ico",
+])
+def test_excluded_paths_produce_no_row(captured, path):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path=path))
+    assert captured == [], f"{path} must not be logged by default"
+
+
+def test_log_viewer_path_cannot_be_un_excluded(captured, monkeypatch):
+    """`exclude_paths` is operator-editable, so the viewer's own endpoints have a
+    hard floor — otherwise reading the log generates log entries about reading
+    the log."""
+    set_config(monkeypatch, exclude_paths=())
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path="/api/request-logs?limit=50"))
+
+    assert captured == [], (
+        "clearing exclude_paths re-enabled logging of the log viewer itself"
+    )
+
+
+def test_options_preflight_is_skipped(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="OPTIONS", path="/api/backends"))
+    assert captured == []
+
+
+def test_get_can_be_turned_off(captured, monkeypatch):
+    set_config(monkeypatch, capture_get=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="GET", path="/api/backends"))
+    assert captured == []
+
+    asyncio.run(drive(app, method="POST", path="/api/backends", body=b"{}"))
+    assert len(captured) == 1, "turning GETs off must not silence writes"
+
+
+def test_disabled_config_short_circuits_but_still_serves(captured, monkeypatch):
+    set_config(monkeypatch, enabled=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=b"hello"))
+
+    assert captured == []
+    assert sent["status"] == 200, "the kill-switch must not break request serving"
+
+
+def test_capture_bodies_off_keeps_sizes_but_drops_content(captured, monkeypatch):
+    set_config(monkeypatch, capture_bodies=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=b'{"secret":"x"}'))
+
+    row = captured[0]
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 14, "size accounting must survive with bodies off"
+
+
+# --------------------------------------------------------------------------
+# Errors and correlation
+# --------------------------------------------------------------------------
+
+def test_exception_is_recorded_as_status_class_zero_and_reraised(captured):
+    async def boom(scope, receive, send):
+        raise RuntimeError("handler exploded")
+
+    app = RequestResponseLogMiddleware(boom)
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(drive(app, method="GET"))
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a request that produced no HTTP response must be status_class 0 — that is the "
+        "sentinel the error-retention prune keys off"
+    )
+    assert row.error.startswith("RuntimeError")
+
+
+def test_correlation_id_is_seeded_before_downstream_and_reset_after(captured):
+    seen = {}
+
+    async def app(scope, receive, send):
+        seen["cid"] = correlation_id_context.get()
+        await send({"type": "http.response.start", "status": 204, "headers": []})
+        await send({"type": "http.response.body", "body": b""})
+
+    wrapped = RequestResponseLogMiddleware(app)
+    asyncio.run(drive(wrapped, method="GET"))
+
+    row = captured[0]
+    assert seen["cid"] == row.request_id[:8], (
+        "the downstream error handler would mint its own id, so X-Correlation-ID would "
+        "not match request_logs.request_id"
+    )
+    assert correlation_id_context.get() is None, (
+        "the ContextVar token was not reset — the next request on this task would inherit "
+        "a stale correlation id"
+    )
+
+
+def test_x_request_id_header_is_returned_to_the_client(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    sent = asyncio.run(drive(app, method="GET"))
+
+    names = {k.decode().lower() for k, _ in sent["headers"]}
+    assert "x-request-id" in names, (
+        "without this header a user reporting a problem has no id to quote"
+    )
+
+
+def test_error_responses_are_logged_with_their_status(captured):
+    app = RequestResponseLogMiddleware(echo_length_app(status=422))
+    asyncio.run(drive(app, body=b'{"bad":true}'))
+
+    row = captured[0]
+    assert row.status_code == 422
+    assert row.status_class == 4, "4xx must be classed as an error for retention purposes"
+
+
+def test_jwt_identity_is_resolved_without_a_database(captured):
+    """The middleware runs on every request; a DB lookup per call is not
+    acceptable, so the user is read straight out of the token claims."""
+    from datetime import datetime, timedelta
+
+    from jose import jwt
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+
+    token = jwt.encode(
+        {"user_id": 42, "username": "ops", "exp": datetime.utcnow() + timedelta(minutes=10)},
+        JWT_SECRET_KEY, algorithm=JWT_ALGORITHM,
+    )
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": f"Bearer {token}"}))
+
+    row = captured[0]
+    assert row.user_id == 42
+    assert row.username == "ops"
+
+
+def test_malformed_token_yields_an_anonymous_row(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer not.a.jwt"}))
+
+    row = captured[0]
+    assert row.user_id is None
+    assert row.username is None
+    # Logging is not an auth path — a bad token must not turn into an exception.
+
+
+def test_authorization_header_is_never_stored_verbatim(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer super-secret"}))
+
+    params = captured[0].to_params()
+    assert "super-secret" not in json.dumps(params, default=str)
+
+
+# --------------------------------------------------------------------------
+# Registration order in main.py
+# --------------------------------------------------------------------------
+
+def test_middleware_is_registered_last_so_it_is_outermost():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    log_at = src.index("app.add_middleware(RequestResponseLogMiddleware)")
+    cors_at = src.index("    CORSMiddleware,")
+
+    assert log_at > cors_at, (
+        "Starlette's add_middleware inserts at index 0, so the LAST registration is the "
+        "OUTERMOST middleware. Registering the request logger before CORS would put it "
+        "inside the stack, where it can no longer see the final client-visible response "
+        "and can no longer seed the correlation id before the error handler reads it."
+    )
+
+
+def test_env_kill_switch_guards_the_registration():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert re.search(
+        r"if REQUEST_LOG_ENABLED:\s*\n\s*app\.add_middleware\(RequestResponseLogMiddleware\)",
+        src,
+    ), (
+        "REQUEST_LOG_ENABLED must gate the add_middleware call itself, not a branch inside "
+        "the middleware — the whole point is that a disabled log costs nothing"
+    )
+
+
+def test_cors_exposes_the_request_id_header():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert "expose_headers=" in src and "X-Request-ID" in src, (
+        "without expose_headers the browser cannot read X-Request-ID on a cross-origin "
+        "deployment, so the id is unusable from the app"
+    )

backend/tests/test_request_log_migration.py

@@ -0,0 +1,251 @@
+"""v1.11.0: the request_logs migration actually runs on existing installs.
+
+Source-scan tests (the sanctioned pattern here — there is no database in this
+suite). The failure mode being pinned is specific and silent: migrations are
+gated on `applied_version >= SCHEMA_VERSION`, so forgetting the bump means the
+whole sequence is skipped on every already-deployed database and neither the
+table nor the new permissions ever appear — while a fresh install works fine,
+so it looks correct in development.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_MIGRATIONS = os.path.join(_BACKEND, "database", "migrations.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        return f.read()
+
+
+@pytest.fixture(scope="module")
+def runner_body(src):
+    """The body of _run_all_migrations_inner, where steps are registered."""
+    assert "async def _run_all_migrations_inner" in src
+    return src.split("async def _run_all_migrations_inner", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.fixture(scope="module")
+def rbac_body(src):
+    return src.split("async def update_system_roles_to_enterprise_rbac", 1)[1].split("\nasync def ", 1)[0]
+
+
+def _role_block(rbac_body, role):
+    """Slice one role's permission list out of the enterprise_roles literal."""
+    start = rbac_body.index(f"'{role}': {{")
+    end = rbac_body.index("]", rbac_body.index("'permissions': [", start))
+    return rbac_body[start:end]
+
+
+# --------------------------------------------------------------------------
+# The version gate
+# --------------------------------------------------------------------------
+
+def test_schema_version_bumped_to_at_least_11(src):
+    match = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert match, "SCHEMA_VERSION assignment not found in migrations.py"
+    assert int(match.group(1)) >= 11, (
+        "SCHEMA_VERSION was not bumped for the request_logs table. run_all_migrations() "
+        "returns early when the recorded version is already >= SCHEMA_VERSION, so every "
+        "existing deployment would skip the whole run: no request_logs table, no "
+        "requestlog.* permissions, and the feature would silently do nothing in production "
+        "while working perfectly on a fresh database."
+    )
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_both_migration_steps_are_registered(runner_body):
+    assert "await ensure_request_logs_table()" in runner_body, (
+        "ensure_request_logs_table is defined but never called from the migration runner"
+    )
+    assert "await ensure_request_log_settings()" in runner_body, (
+        "the retention defaults are never seeded, so an upgraded install has no "
+        "requestlog.* rows and Settings shows blanks"
+    )
+
+
+def test_table_is_created_before_its_settings_are_seeded(runner_body):
+    table_at = runner_body.index("await ensure_request_logs_table()")
+    seed_at = runner_body.index("await ensure_request_log_settings()")
+    assert table_at < seed_at, (
+        "the settings seed runs before the table step; if the table step then raises, the "
+        "run aborts with settings but no table"
+    )
+
+
+# --------------------------------------------------------------------------
+# The DDL itself
+# --------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def ddl_body(src):
+    return src.split("async def ensure_request_logs_table", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.mark.parametrize("fragment", [
+    "CREATE TABLE IF NOT EXISTS request_logs",
+    "id                  BIGSERIAL PRIMARY KEY",
+    "request_id          VARCHAR(64) NOT NULL",
+    "direction           VARCHAR(8)  NOT NULL",
+    "status_class        SMALLINT",
+    "created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()",
+    "request_logs_direction_check",
+    "client_ip           INET",
+])
+def test_ddl_essentials(ddl_body, fragment):
+    assert fragment in ddl_body, f"request_logs DDL is missing {fragment!r}"
+
+
+def test_ddl_is_idempotent(ddl_body):
+    assert "CREATE TABLE IF NOT EXISTS" in ddl_body
+    creates = re.findall(r"CREATE INDEX(?: IF NOT EXISTS)?", ddl_body)
+    assert creates, "no indexes are created for request_logs"
+    assert all(c == "CREATE INDEX IF NOT EXISTS" for c in creates), (
+        "an index is created without IF NOT EXISTS — the second startup would raise and "
+        "abort the whole migration run"
+    )
+
+
+def test_prune_partial_indexes_are_present(ddl_body):
+    """The retention delete is split by outcome, so a plain
+    (status_class, created_at) index would still range-scan the half it does not
+    want."""
+    assert "idx_request_logs_prune_ok" in ddl_body
+    assert "idx_request_logs_prune_err" in ddl_body
+    assert "WHERE status_class BETWEEN 1 AND 3" in ddl_body
+    assert "WHERE status_class = 0 OR status_class >= 4" in ddl_body
+
+
+def test_request_id_index_exists_for_the_trace_view(ddl_body):
+    assert "idx_request_logs_request_id" in ddl_body, (
+        "without this index, opening one request to see the outbound calls it triggered "
+        "is a sequential scan"
+    )
+
+
+def test_no_foreign_key_on_user_id(ddl_body):
+    """Deliberate deviation from the house style — see the docstring in
+    migrations.py. Pinned so it is not 'fixed' back into an FK later."""
+    user_id_line = [line for line in ddl_body.splitlines() if "user_id " in line and "INTEGER" in line]
+    assert user_id_line, "user_id column not found"
+    assert "REFERENCES" not in user_id_line[0], (
+        "an FK was added to request_logs.user_id — per-insert FK validation on the "
+        "highest-volume table in the system, and audit rows must outlive the account"
+    )
+
+
+def test_migration_step_reraises_on_failure(ddl_body):
+    """The version marker is written only after the inner sequence completes, so
+    swallowing here would stamp version 11 with no table and the gate would then
+    skip every retry, permanently."""
+    assert re.search(r"\n\s+raise\n", ddl_body), (
+        "ensure_request_logs_table swallows its exception instead of re-raising"
+    )
+
+
+def test_settings_seed_does_not_overwrite_operator_tuning(src):
+    seed_body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    assert "ON CONFLICT (key) DO NOTHING" in seed_body, (
+        "the seed uses DO UPDATE, so every upgrade would reset the operator's retention "
+        "settings back to the defaults"
+    )
+
+
+# --------------------------------------------------------------------------
+# Permission seeding
+# --------------------------------------------------------------------------
+
+def test_super_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "super_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_security_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "security_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_operator_gets_read_only(rbac_body):
+    block = _role_block(rbac_body, "operator")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" not in block, (
+        "operators should be able to read the log to debug an apply or an ACME order, but "
+        "retention policy and purge belong to the admins"
+    )
+
+
+def test_viewer_gets_neither(rbac_body):
+    block = _role_block(rbac_body, "viewer")
+    assert "requestlog" not in block, (
+        "viewer was granted a requestlog permission. Even redacted, captured request and "
+        "response bodies are a far broader disclosure surface than the read-only config "
+        "views a viewer is meant to have."
+    )
+
+
+def test_permission_strings_have_exactly_one_dot(rbac_body):
+    """get_user_permissions splits on the FIRST dot and silently drops any
+    string without one."""
+    for perm in re.findall(r"'(requestlog[^']*)'", rbac_body):
+        assert perm.count(".") == 1, f"{perm!r} is not a <resource>.<action> pair"
+
+
+def test_initial_seed_lists_stay_in_sync(src):
+    """create_initial_system_data() is overwritten by the enterprise seeder on
+    every run, but that seeder swallows all exceptions — keeping the two in sync
+    is the safety net."""
+    initial = src.split("system_roles = [", 1)[1].split("\n        ]", 1)[0]
+    assert '"requestlog.read"' in initial
+    assert '"requestlog.manage"' in initial
+
+
+# --------------------------------------------------------------------------
+# Runtime wiring
+# --------------------------------------------------------------------------
+
+def test_prune_loop_is_started_and_independent_of_the_acme_loop():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "async def prune_request_logs_loop" in main_src
+    assert "asyncio.create_task(prune_request_logs_loop())" in main_src, (
+        "the retention prune task is defined but never started, so request_logs grows "
+        "without bound"
+    )
+    loop_body = main_src.split("async def prune_request_logs_loop", 1)[1].split("\n# Production middleware", 1)[0]
+    assert "table_name = 'request_logs'" in loop_body, (
+        "the prune loop does not check for its own table, so it would log an error every "
+        "tick on a database where the migration has not run yet"
+    )
+    assert "table_name = 'letsencrypt_orders'" not in loop_body, (
+        "the prune loop was gated on the ACME table, which would disable retention "
+        "entirely on an install that never uses ACME"
+    )
+
+
+def test_sink_is_flushed_before_the_pool_closes():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    body = main_src.split("async def shutdown_event", 1)[1]
+    flush_at = body.find("request_log_sink.flush")
+    close_at = body.find("close_database_pool()")
+    assert flush_at != -1, "queued request-log rows are never flushed on shutdown"
+    assert flush_at < close_at, (
+        "the sink is flushed after the pool is closed, so the queued rows are lost. The "
+        "sink's writer is a `while True` loop and can never satisfy the generic "
+        "asyncio.wait drain, so it needs its own explicit flush first."
+    )

backend/tests/test_request_log_prune.py

@@ -0,0 +1,261 @@
+"""v1.11.0: retention actually reclaims space, and cannot be turned into an
+injection point or a 60-second lock.
+
+`request_logs` is the highest-volume table in the system, so the prune has
+three properties that are easy to get wrong and expensive to get wrong:
+
+  * the operator-supplied retention day counts are BIND PARAMETERS, never
+    string-interpolated into the SQL;
+  * deletes are BATCHED, because the pool's command_timeout is 60s and an
+    unbounded DELETE over millions of rows raises and then nothing is ever
+    pruned;
+  * the watermark is stamped only after a COMPLETE pass, so a pass that dies
+    half-way is retried instead of being recorded as done.
+
+The fake connection dispatches on the SQL text rather than on call order — an
+ordered side_effect list silently passes tests for the wrong reason as soon as
+the number of statements changes.
+"""
+import asyncio
+import json
+import os
+import sys
+from datetime import datetime, timedelta
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_prune  # noqa: E402
+from utils.request_log_prune import (  # noqa: E402
+    BATCH_SIZE,
+    MAX_BATCHES,
+    PRUNE_LOCK_KEY,
+    prune_request_logs_if_due,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_SUCCESS_MARKER = "status_class BETWEEN 1 AND 3"
+_ERROR_MARKER = "status_class = 0 OR status_class >= 4"
+_CAP_MARKER = "id <= $1"
+
+
+def _conn(*, lock=True, watermark_age_minutes=None, cutoff_id=None,
+          success_batches=None, error_batches=None, cap_batches=None,
+          fail_on=None):
+    """A fake asyncpg connection that answers by SQL shape."""
+    conn = AsyncMock()
+
+    def fetchval(sql, *args):
+        text = str(sql)
+        if "pg_try_advisory_lock" in text:
+            return lock
+        if "ORDER BY id DESC OFFSET" in text:
+            return cutoff_id
+        return None
+
+    conn.fetchval = AsyncMock(side_effect=fetchval)
+
+    if watermark_age_minutes is None:
+        conn.fetchrow = AsyncMock(return_value=None)
+    else:
+        stamp = (datetime.utcnow() - timedelta(minutes=watermark_age_minutes)).isoformat() + "Z"
+        conn.fetchrow = AsyncMock(return_value={"value": json.dumps(stamp)})
+
+    queues = {
+        _SUCCESS_MARKER: list(success_batches or ["DELETE 0"]),
+        _ERROR_MARKER: list(error_batches or ["DELETE 0"]),
+        _CAP_MARKER: list(cap_batches or ["DELETE 0"]),
+    }
+
+    def execute(sql, *args):
+        text = str(sql)
+        if fail_on and fail_on in text:
+            raise RuntimeError("statement timeout")
+        for marker, queue in queues.items():
+            if marker in text:
+                return queue.pop(0) if queue else "DELETE 0"
+        return ""
+
+    conn.execute = AsyncMock(side_effect=execute)
+    return conn
+
+
+def _run(conn, *, force=False, **cfg_overrides):
+    cfg = replace(DEFAULT_CONFIG, **cfg_overrides)
+    with patch.object(request_log_prune, "get_config", lambda: cfg), \
+         patch.object(request_log_prune, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        return asyncio.run(prune_request_logs_if_due(force=force))
+
+
+def _delete_sql(conn):
+    return [str(c.args[0]) for c in conn.execute.call_args_list
+            if "DELETE FROM request_logs" in str(c.args[0])]
+
+
+def test_skips_entirely_when_another_replica_holds_the_lock():
+    conn = _conn(lock=False)
+    counts = _run(conn)
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+    assert _delete_sql(conn) == [], (
+        "a second replica ran the prune concurrently — pg_try_advisory_lock is what keeps "
+        "N pods from all scanning the same table at once"
+    )
+
+
+def test_skips_when_the_watermark_is_still_fresh():
+    conn = _conn(watermark_age_minutes=10)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["ran"] == 0
+    assert _delete_sql(conn) == []
+
+
+def test_runs_all_three_limits_when_due():
+    conn = _conn(
+        watermark_age_minutes=120, cutoff_id=999,
+        success_batches=["DELETE 3"], error_batches=["DELETE 4"], cap_batches=["DELETE 5"],
+    )
+    counts = _run(conn, prune_interval_minutes=60, success_retention_days=7,
+                  error_retention_days=30, max_rows=500000)
+
+    sqls = _delete_sql(conn)
+    assert len(sqls) == 3, f"expected success TTL + error TTL + row cap, got {len(sqls)}"
+    assert _SUCCESS_MARKER in sqls[0]
+    assert _ERROR_MARKER in sqls[1]
+    assert _CAP_MARKER in sqls[2]
+
+    assert counts["success"] == 3
+    assert counts["error"] == 4
+    assert counts["overflow"] == 5
+    assert counts["ran"] == 1
+
+
+def test_retention_days_travel_as_bind_parameters():
+    """Injection guard: the day counts come straight from an operator-editable
+    setting, so they must never be formatted into the SQL text."""
+    conn = _conn(watermark_age_minutes=120)
+    _run(conn, prune_interval_minutes=60, success_retention_days=7, error_retention_days=30)
+
+    ttl_calls = [c for c in conn.execute.call_args_list
+                 if "created_at < NOW()" in str(c.args[0])]
+    assert len(ttl_calls) == 2
+
+    for call in ttl_calls:
+        assert "($1 || ' days')::INTERVAL" in str(call.args[0]), (
+            "the retention window is interpolated into the SQL string instead of bound — "
+            "an operator-supplied value reaching the parser is an injection point"
+        )
+
+    assert ttl_calls[0].args[1] == "7"
+    assert ttl_calls[1].args[1] == "30"
+    assert ttl_calls[0].args[2] == BATCH_SIZE
+
+
+def test_deletes_are_batched_until_a_short_batch():
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}", f"DELETE {BATCH_SIZE}", "DELETE 12"],
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * 2 + 12, (
+        "the batch loop stopped early or double-counted"
+    )
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == 3, "the loop must stop on the first short batch"
+
+
+def test_batch_loop_respects_the_ceiling():
+    """A table so far behind that every batch comes back full must still hand the
+    connection back rather than looping forever."""
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}"] * (MAX_BATCHES * 3),
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * MAX_BATCHES
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == MAX_BATCHES
+
+
+def test_watermark_is_not_stamped_when_a_step_fails():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=42, fail_on=_CAP_MARKER)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert stamps == [], (
+        "a partially-completed pass stamped the watermark, so the remainder would not be "
+        "retried until the next interval"
+    )
+    assert counts["ran"] == 0
+
+
+def test_watermark_is_stamped_after_a_complete_pass():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert len(stamps) == 1
+    # args = (sql, key, json_value)
+    assert stamps[0].args[1] == "requestlog.last_pruned_at"
+    assert stamps[0].args[2].startswith('"'), (
+        "the watermark must be stored as a JSON string — the ::jsonb cast rejects a bare "
+        "timestamp, and the reader json.loads() it back"
+    )
+    assert counts["ran"] == 1
+
+
+def test_advisory_lock_is_released_even_on_failure():
+    conn = _conn(watermark_age_minutes=120, fail_on=_SUCCESS_MARKER)
+    _run(conn, prune_interval_minutes=60)
+
+    unlocks = [c for c in conn.execute.call_args_list if "pg_advisory_unlock" in str(c.args[0])]
+    assert unlocks, "the advisory lock was leaked — every later pass on any replica would skip"
+    assert unlocks[0].args[1] == PRUNE_LOCK_KEY
+
+
+def test_never_raises_when_the_pool_is_exhausted():
+    with patch.object(request_log_prune, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        counts = asyncio.run(prune_request_logs_if_due())
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+
+
+def test_row_cap_is_a_noop_when_the_table_is_smaller_than_the_cap():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None, cap_batches=["DELETE 77"])
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["overflow"] == 0, (
+        "the cap deleted rows even though OFFSET max_rows found no cutoff — that would "
+        "truncate a table that is under the limit"
+    )
+    assert not any(_CAP_MARKER in s for s in _delete_sql(conn))
+
+
+def test_force_bypasses_the_watermark():
+    """The manual purge button must not be a no-op just because the scheduled
+    pass ran a minute ago."""
+    conn = _conn(watermark_age_minutes=1, cutoff_id=None,
+                 success_batches=["DELETE 1"], error_batches=["DELETE 2"])
+    counts = _run(conn, force=True, prune_interval_minutes=1440)
+
+    assert counts["ran"] == 1
+    assert counts["success"] == 1
+    assert counts["error"] == 2
+
+
+def test_lock_key_does_not_collide_with_the_existing_ones():
+    # 18181818 drafts cap, 18181819 wizard create, 18181820 apply,
+    # 0x41434D45 per-ACME-order, 1836016242 migration.
+    assert PRUNE_LOCK_KEY not in (18181818, 18181819, 18181820, 0x41434D45, 1836016242)

backend/tests/test_request_log_real_payloads.py

@@ -0,0 +1,289 @@
+"""v1.11.0: redaction pinned against THIS codebase's real payloads.
+
+test_request_log_redaction.py pins the RULES — which key names match, which
+value shapes fire. It passed 72/72 while six real endpoints of this application
+still wrote secrets to `request_logs`, because a rule test proves the rule, not
+the coverage. Every case here is built from an actual handler's request or
+response shape, with the field names taken from the source and named in the
+docstring, so a future change to redaction is measured against what this system
+actually sends rather than against what someone remembered to imagine.
+
+Method note: the payloads go through `decode_body()`, the same entry point the
+writer task uses, rather than calling `redact()` directly. That is deliberate —
+two of the findings below only appear on the way in (an oversized body never
+reaches `redact()` as a dict at all, it arrives as one `_raw` string), so a test
+that starts from a dict would report a pass on a payload that leaks in
+production.
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    decode_body,
+    is_secret_key,
+    redact_headers,
+)
+
+VRRP_SECRET = "S3cr3tVrrpPass!"
+TOTP_SECRET = "JBSWY3DPEHPK3PXP"
+STATS_PASSWORD = "StatsPa55word"
+USERLIST_HASH = "$6$rounds=5000$abcdefgh$XyZ"
+JWT = (
+    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
+    ".eyJzdWIiOiIxIiwidXNlcm5hbWUiOiJhZG1pbiJ9"
+    ".dQw4w9WgXcQdQw4w9WgXcQdQw4w9WgXcQ"
+)
+PEM_KEY = (
+    "-----BEGIN RSA PRIVATE KEY-----\n"
+    + "MIIEowIBAAKCAQEA" + "A" * 200 + "\n"
+    + "-----END RSA PRIVATE KEY-----\n"
+)
+
+# The rendered file `GET /api/agents/{n}/keepalived-config` hands to an agent,
+# and the one `POST /api/agents/{n}/keepalived-discovery` sends back.
+KEEPALIVED_CONF = f"""! Managed by HAProxy OpenManager
+vrrp_instance VI_1 {{
+    state MASTER
+    interface eth0
+    virtual_router_id 51
+    priority 200
+    advert_int 1
+    authentication {{
+        auth_type PASS
+        auth_pass {VRRP_SECRET}
+    }}
+    virtual_ipaddress {{
+        10.20.30.40/24
+    }}
+}}
+"""
+
+# A production haproxy.cfg as the agent uploads it verbatim from the node
+# (`config_content=$(cat "$config_path")` -> POST .../config-response).
+HAPROXY_CFG = f"""global
+    log stdout local0
+    stats socket /var/run/haproxy.sock mode 660
+
+userlist admins
+    user ops password {USERLIST_HASH}
+    user dev insecure-password Hunter2Plain
+
+listen stats
+    bind *:8404
+    stats enable
+    stats auth admin:{STATS_PASSWORD}
+    stats uri /stats
+
+backend web
+    server web1 10.0.0.1:80 check
+"""
+
+
+def _capture(payload, *, cap=8192, content_type="application/json"):
+    """Run a payload through the capture path exactly as the writer does.
+
+    `cap` is `requestlog.max_body_bytes`. Bodies larger than it arrive
+    truncated, do not parse as JSON, and land in the `{"_raw": ...}` fallback —
+    which is the common case for config uploads and the case a dict-based test
+    never exercises.
+    """
+    body = json.dumps(payload).encode()
+    value, truncated = decode_body(body[:cap], content_type, len(body))
+    return json.dumps(value, default=str), truncated
+
+
+def _assert_absent(rendered, *secrets):
+    for secret in secrets:
+        assert secret not in rendered, (
+            f"{secret!r} reached request_logs. Rendered row: {rendered[:400]}"
+        )
+
+
+# --------------------------------------------------------------------------
+# The VRRP password. routers/vip.py: "the secret never leaves the server in
+# cleartext ... only the at-rest Fernet token and the agent-delivery endpoint
+# ever see the real value."
+# --------------------------------------------------------------------------
+
+def test_vip_create_body_does_not_store_auth_pass():
+    """POST/PUT /api/vip — `payload.auth_pass`, routers/vip.py:585,679."""
+    rendered, _ = _capture({
+        "name": "vip-prod", "virtual_ip": "10.20.30.40", "interface": "eth0",
+        "virtual_router_id": 51, "auth_pass": VRRP_SECRET,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_keepalived_config_delivery_does_not_store_the_rendered_secret():
+    """GET /api/agents/{n}/keepalived-config — `keepalived.config_content`.
+
+    Polled on the SSL cadence, so an unmasked capture rewrites the secret to the
+    audit table roughly 576 times a day per member node.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "status": "available",
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "keepalived": {
+            "vip_id": 3, "vip_name": "vip-prod",
+            "config_content": KEEPALIVED_CONF, "config_hash": "abc123",
+        },
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "auth_pass" in rendered, "the directive should stay visible, only its value masked"
+    assert "vrrp_instance VI_1" in rendered, "masking must not destroy the rest of the config"
+
+
+def test_keepalived_discovery_body_does_not_store_the_found_secret():
+    """POST /api/agents/{n}/keepalived-discovery — `config_content`.
+
+    routers/agent.py already pops auth_pass out of the parsed analysis,
+    Fernet-encrypts it into its own column and stores only
+    `vip_discoveries.raw_config_masked`. Capturing the request that produced all
+    that, unmasked, would put the plaintext straight back next to it.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "exists": True, "is_managed": False,
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "config_content": KEEPALIVED_CONF,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_auth_pass_is_masked_when_the_body_is_too_large_to_parse():
+    """The truncated `_raw` path, where line breaks are the escape `\\n`.
+
+    A value pattern that stops only at a REAL newline runs to the end of the
+    string here: no leak, but the whole remainder of the config is masked and
+    the row is useless. Both properties are asserted.
+    """
+    payload = {"config_content": KEEPALIVED_CONF + "backend b\n    server s1 10.0.0.1:80 check\n" * 400}
+    rendered, truncated = _capture(payload)
+    assert truncated, "this fixture must exercise the truncated path"
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "server s1 10.0.0.1:80" in rendered, (
+        "masking ran past the end of the auth_pass line and ate the rest of the config"
+    )
+
+
+# --------------------------------------------------------------------------
+# TOTP. routers/mfa.py logs `{"secret_len": ...}` with the comment
+# "NEVER log the secret itself".
+# --------------------------------------------------------------------------
+
+def test_mfa_enroll_response_does_not_store_the_totp_secret_in_either_field():
+    """POST /api/mfa/enroll — returns `secret` AND `otpauth_uri`.
+
+    Redacting one while the same value sits in the other is not redaction.
+    """
+    rendered, _ = _capture({
+        "secret": TOTP_SECRET,
+        "otpauth_uri": f"otpauth://totp/OpenManager:admin?secret={TOTP_SECRET}&issuer=OpenManager",
+        "qr_size": 256,
+    })
+    _assert_absent(rendered, TOTP_SECRET)
+
+
+def test_userinfo_credentials_in_a_url_are_dropped():
+    rendered, _ = _capture({"webhook": "https://svc:Sup3rSecret@hooks.example.com/notify?api_key=abc123"})
+    _assert_absent(rendered, "Sup3rSecret", "abc123")
+    assert "hooks.example.com" in rendered, "the host is the diagnostic value; keep it"
+
+
+# --------------------------------------------------------------------------
+# HAProxy config. We never RENDER credentials into one, but the agent uploads
+# the node's real file and the operator can paste one.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("cap,label", [(8192, "truncated _raw path"), (10 ** 6, "parsed path")])
+def test_uploaded_haproxy_config_masks_credentials_on_both_paths(cap, label):
+    """POST /api/configuration/agents/{n}/config-response, and
+    POST /api/config/validate."""
+    rendered, _ = _capture({"config_content": HAPROXY_CFG, "config_path": "/etc/haproxy/haproxy.cfg"}, cap=cap)
+    _assert_absent(rendered, STATS_PASSWORD, USERLIST_HASH, "Hunter2Plain")
+    assert "stats auth admin:" in rendered, f"[{label}] the account name is diagnostic; keep it"
+    assert "server web1 10.0.0.1:80" in rendered, f"[{label}] the rest of the config must survive"
+
+
+@pytest.mark.parametrize("prose", [
+    "invalid password format",
+    "the password must be at least 8 characters",
+    "authentication failed for user admin",
+])
+def test_ordinary_prose_is_not_mangled(prose):
+    """Over-matching would blank the messages the log exists to show."""
+    rendered, _ = _capture({"detail": prose})
+    assert prose in rendered, f"redaction damaged an ordinary message: {rendered}"
+
+
+# --------------------------------------------------------------------------
+# Regressions guarding what already worked, so a later rule change cannot
+# quietly trade one of these away for one of the above.
+# --------------------------------------------------------------------------
+
+def test_login_exchange_stores_neither_the_password_nor_the_token():
+    req, _ = _capture({"username": "admin", "password": "hunter2hunter2"})
+    _assert_absent(req, "hunter2hunter2")
+    res, _ = _capture({"access_token": JWT, "token_type": "bearer", "user": {"id": 1}})
+    _assert_absent(res, JWT)
+
+
+def test_private_key_is_redacted_even_under_an_innocent_key_name():
+    """The value-shape guard is the net under the key-name rules."""
+    rendered, _ = _capture({"blob": PEM_KEY, "note": "backup"})
+    _assert_absent(rendered, "MIIEowIBAAKCAQEA")
+
+
+def test_dns_provider_credentials_are_redacted():
+    cf, _ = _capture({"provider": "cloudflare", "api_token": "cf_live_abcdefghijklmnop", "zone_id": "z1"})
+    _assert_absent(cf, "cf_live_abcdefghijklmnop")
+    gd, _ = _capture({"provider": "godaddy", "api_key": "gd_key_1234567890", "api_secret": "gd_secret_098"})
+    _assert_absent(gd, "gd_key_1234567890", "gd_secret_098")
+
+
+def test_innocent_urls_survive_untouched():
+    """Scrubbing must not rewrite the ACME URLs an operator reads back."""
+    for url in (
+        "https://acme-v02.api.letsencrypt.org/directory",
+        "https://acme-v02.api.letsencrypt.org/acme/acct/12345",
+    ):
+        rendered, _ = _capture({"directory_url": url})
+        assert url in rendered, f"an innocent URL was rewritten: {rendered}"
+
+
+def test_credential_headers_are_presence_only_and_the_rest_are_dropped():
+    out = redact_headers({
+        "authorization": f"Bearer {JWT}",
+        "x-api-key": "agt_" + "a" * 32,
+        "cookie": "session=abc123",
+        "user-agent": "curl/8.4.0",
+        "x-forwarded-for": "10.20.30.5",
+        "x-internal-secret": "not-on-the-allowlist",
+    })
+    rendered = json.dumps(out)
+    _assert_absent(rendered, JWT, "agt_" + "a" * 32, "abc123", "not-on-the-allowlist")
+    assert out["user-agent"] == "curl/8.4.0"
+    assert out["x-forwarded-for"] == "10.20.30.5"
+    assert "x-internal-secret" not in out, "an unlisted header must be dropped, not kept"
+
+
+@pytest.mark.parametrize("key", ["auth_pass", "authPass", "auth-pass", "AUTH_PASS"])
+def test_auth_pass_key_matches_in_every_spelling(key):
+    assert is_secret_key(key), (
+        f"{key!r} normalizes to something no rule matches. 'password' is not a "
+        f"substring of 'authpass' and the bare 'auth' entry is an exact match."
+    )
+
+
+@pytest.mark.parametrize("key", [
+    "monkey", "key_suffix", "payload_size", "nonce_count", "keyboard_layout",
+    "config_path", "authenticated", "author",
+])
+def test_innocent_field_names_are_still_kept(key):
+    """The other half of the trade: over-redaction blanks the fields the
+    feature exists to show."""
+    assert not is_secret_key(key)

backend/tests/test_request_log_redaction.py

@@ -0,0 +1,262 @@
+"""v1.11.0: nothing secret reaches request_logs.
+
+The request/response log stores bodies and headers, so redaction is the single
+control standing between "operators can debug a failing ACME order" and "the
+audit table is a credential store". These tests pin both halves of that: the
+things that MUST be redacted, and the innocent field names that must NOT be
+(over-matching would silently blank out the fields the feature exists to show).
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    REDACTED,
+    decode_body,
+    is_capturable_content_type,
+    is_secret_key,
+    redact,
+    redact_headers,
+    safe_error_text,
+    scrub_query_string,
+    scrub_url,
+)
+
+
+# --------------------------------------------------------------------------
+# Key matching
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("key", [
+    "password", "PASSWORD", "Pass_Word", "passwd", "pwd",
+    # api_token is the literal field name of the Cloudflare provider credential
+    # (services/dns_providers/cloudflare.py) — it must never survive a round trip.
+    "token", "access_token", "refreshToken", "MFA_TOKEN",
+    "api_token", "agent_token", "csrf_token", "session_token",
+    "api_key", "API-KEY", "apiKey", "x-api-key",
+    "secret", "client_secret", "eab_hmac_key",
+    "private_key", "cert_private_key", "csr_private_key", "jwk_private_key",
+    "authorization", "cookie", "set-cookie",
+    "signature", "protected", "payload", "nonce", "replay-nonce",
+    "key_authorization", "backup_codes", "totp_secret",
+    "stats_password", "credentials", "encryption_key",
+])
+def test_secret_keys_are_detected(key):
+    assert is_secret_key(key), f"{key!r} must be treated as a secret field name"
+
+
+@pytest.mark.parametrize("key", [
+    # Every one of these has a secret-looking substring but is innocent. If any
+    # starts redacting, the log stops being useful for the exact debugging it
+    # was built for.
+    "key_suffix", "monkey", "keyboard", "turkey",
+    "payload_size", "nonce_count",
+    "public_key_id", "keys_total",
+    "name", "status_code", "duration_ms", "domain", "directory_url",
+])
+def test_innocent_keys_are_not_redacted(key):
+    assert not is_secret_key(key), (
+        f"{key!r} was redacted by over-matching — the log would blank out a field "
+        f"operators need"
+    )
+
+
+# --------------------------------------------------------------------------
+# Recursive body redaction
+# --------------------------------------------------------------------------
+
+def test_nested_dicts_and_lists_are_redacted_recursively():
+    body = {
+        "user": {"username": "admin", "password": "hunter2"},
+        "accounts": [
+            {"email": "a@example.com", "eab_hmac_key": "s3cr3t"},
+            {"email": "b@example.com", "api_token": "cf-token"},
+        ],
+        "cluster_id": 3,
+    }
+    out = redact(body)
+
+    assert out["user"]["username"] == "admin"
+    assert out["user"]["password"] == REDACTED
+    assert out["accounts"][0]["email"] == "a@example.com"
+    assert out["accounts"][0]["eab_hmac_key"] == REDACTED
+    assert out["accounts"][1]["api_token"] == REDACTED
+    assert out["cluster_id"] == 3
+
+
+def test_depth_limit_stops_runaway_nesting():
+    deep = current = {}
+    for _ in range(20):
+        current["child"] = {}
+        current = current["child"]
+    current["password"] = "leak"
+
+    out = redact(deep)
+    flattened = json.dumps(out)
+    assert "***DEPTH_LIMIT***" in flattened
+    assert "leak" not in flattened
+
+
+def test_node_budget_bounds_a_very_wide_body():
+    wide = {f"field_{i}": i for i in range(5000)}
+    out = redact(wide)
+    assert out.get("_node_limit") is True
+    assert len(out) < 5000, "node budget did not bound a pathologically wide body"
+
+
+def test_pem_private_key_is_redacted_by_value_shape():
+    body = {"blob": "-----BEGIN RSA PRIVATE KEY-----\n" + "A" * 200 + "\n-----END RSA PRIVATE KEY-----"}
+    out = redact(body)
+    assert out["blob"] == REDACTED, (
+        "a PEM private key under an innocent key name was stored verbatim"
+    )
+
+
+def test_jwt_shaped_string_is_redacted_by_value_shape():
+    jwt_like = "eyJhbGciOiJIUzI1NiJ9." + "a" * 40 + "." + "b" * 40
+    out = redact({"data": jwt_like})
+    assert out["data"] == REDACTED
+
+
+def test_long_strings_are_truncated_with_a_marker():
+    out = redact({"note": "x" * 9000})
+    assert out["note"].endswith("chars]")
+    assert len(out["note"]) < 9000
+
+
+def test_redact_never_raises_on_odd_input():
+    class Weird:
+        def __repr__(self):
+            raise RuntimeError("boom")
+
+    # Non-serializable leaf values must pass straight through, not explode.
+    assert redact({"x": Weird()}) is not None
+
+
+# --------------------------------------------------------------------------
+# Headers (allowlist)
+# --------------------------------------------------------------------------
+
+def test_headers_use_an_allowlist_with_presence_markers():
+    out = redact_headers({
+        "Content-Type": "application/json",
+        "User-Agent": "curl/8.0",
+        "Authorization": "Bearer super-secret-token",
+        "Cookie": "session=abc",
+        "X-Custom-Internal": "some value",
+    })
+
+    assert out["content-type"] == "application/json"
+    assert out["user-agent"] == "curl/8.0"
+    # Presence is useful when debugging a 401; the value is not.
+    assert out["authorization"] == REDACTED
+    assert out["cookie"] == REDACTED
+    # Not on the allowlist and not a known credential header -> dropped entirely.
+    assert "x-custom-internal" not in out
+
+
+def test_redact_headers_handles_none():
+    assert redact_headers(None) is None
+    assert redact_headers({}) is None
+
+
+# --------------------------------------------------------------------------
+# URLs and query strings
+# --------------------------------------------------------------------------
+
+def test_query_string_secrets_are_scrubbed():
+    scrubbed, as_dict = scrub_query_string("token=abc123&page=2&api_key=xyz")
+    assert "abc123" not in scrubbed
+    assert "xyz" not in scrubbed
+    assert "page=2" in scrubbed
+    assert as_dict["token"] == REDACTED
+    assert as_dict["page"] == "2"
+
+
+def test_scrub_url_strips_userinfo_and_query_secrets():
+    out = scrub_url("https://user:hunter2@api.example.com:8443/v1/zones?api_key=abc&page=1")
+    assert "hunter2" not in out
+    assert "user" not in out.split("/v1")[0].replace("api.example.com", "")
+    assert "abc" not in out
+    assert "api.example.com:8443" in out
+    assert "page=1" in out
+
+
+def test_scrub_url_drops_the_fragment():
+    # Fragments never reach a server, and they are a classic token carrier.
+    assert "#" not in scrub_url("https://example.com/x?a=1#access_token=leak")
+
+
+# --------------------------------------------------------------------------
+# Body decoding, capping, truncation marker
+# --------------------------------------------------------------------------
+
+def test_decode_body_parses_and_redacts_json():
+    raw = json.dumps({"username": "admin", "password": "hunter2"}).encode()
+    value, truncated = decode_body(raw, "application/json", len(raw))
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+    assert truncated is False
+
+
+def test_decode_body_marks_truncation_with_the_original_size():
+    full = b"x" * 20000
+    captured = full[:1024]
+    value, truncated = decode_body(captured, "text/plain", len(full))
+    assert truncated is True
+    assert value["_truncated"] is True
+    assert value["_original_bytes"] == 20000
+
+
+def test_decode_body_wraps_non_json_as_raw_object():
+    value, _ = decode_body(b"plain text response", "text/plain", 19)
+    assert value == {"_raw": "plain text response"}
+
+
+def test_decode_body_survives_truncated_json():
+    # A JSON body cut off at the cap will not parse — keep the prefix rather
+    # than losing the field entirely.
+    value, truncated = decode_body(b'{"a": "bb', "application/json", 500)
+    assert truncated is True
+    assert "_raw" in value
+
+
+def test_decode_body_parses_form_encoded():
+    value, _ = decode_body(b"username=admin&password=hunter2",
+                           "application/x-www-form-urlencoded", 30)
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+
+
+def test_decode_body_returns_none_for_empty():
+    assert decode_body(b"", "application/json", 0) == (None, False)
+    assert decode_body(None, "application/json", 0) == (None, False)
+
+
+def test_binary_content_types_are_not_capturable():
+    assert is_capturable_content_type("application/json")
+    assert is_capturable_content_type("application/json; charset=utf-8")
+    assert is_capturable_content_type("text/plain")
+    assert not is_capturable_content_type("application/octet-stream")
+    assert not is_capturable_content_type("image/png")
+    assert not is_capturable_content_type("text/event-stream")
+
+
+# --------------------------------------------------------------------------
+# Error rendering
+# --------------------------------------------------------------------------
+
+def test_safe_error_text_type_only_hides_the_message():
+    exc = ValueError("https://api.godaddy.com/v1/domains/secret-zone/records failed")
+    assert safe_error_text(exc, type_only=True) == "ValueError"
+    assert "godaddy" not in safe_error_text(exc, type_only=True)
+
+
+def test_safe_error_text_includes_the_message_when_allowed():
+    text = safe_error_text(RuntimeError("connection refused"))
+    assert text.startswith("RuntimeError")
+    assert "connection refused" in text

backend/tests/test_request_log_router_auth.py

@@ -0,0 +1,205 @@
+"""v1.11.0: the request log API is gated, and its routes resolve.
+
+Two distinct failure modes are pinned here.
+
+**Auth.** The table holds redacted-but-real request and response bodies for
+every user, so an unauthenticated or under-privileged caller must never get a
+row. There is no database in this suite, so the behavioural checks assert only
+that an anonymous call is rejected before any DB work — which is exactly the
+property that matters — and a source scan covers the per-endpoint permission.
+
+**Route order.** `/{log_id}` is a single-segment path and FastAPI matches in
+declaration order, so declaring it before `/settings`, `/stats` or `/purge`
+makes those three unreachable (they parse as a log id and 422). This is the
+mirror image of the shadowing trap already present in routers/settings.py.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+REJECT = (401, 403, 422)
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_ROUTER, encoding="utf-8") as f:
+        return f.read()
+
+
+# --------------------------------------------------------------------------
+# Behavioural: nothing is readable without credentials
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("method,path", [
+    ("get", "/api/request-logs"),
+    ("get", "/api/request-logs/1"),
+    ("get", "/api/request-logs/stats"),
+    ("get", "/api/request-logs/settings"),
+    ("put", "/api/request-logs/settings"),
+    ("post", "/api/request-logs/purge"),
+])
+def test_anonymous_access_is_rejected(client, method, path):
+    res = getattr(client, method)(path) if method != "put" else client.put(path, json={})
+    assert res.status_code in REJECT, (
+        f"{method.upper()} {path} returned {res.status_code} without an Authorization "
+        f"header — the request log contains captured bodies for every user"
+    )
+
+
+def test_a_garbage_token_is_rejected(client):
+    res = client.get("/api/request-logs", headers={"authorization": "Bearer not-a-token"})
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# Source scan: per-endpoint permission
+# --------------------------------------------------------------------------
+
+def _handler_body(src, decorator):
+    start = src.index(decorator)
+    rest = src[start + len(decorator):]
+    end = rest.find("\n@router.")
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.mark.parametrize("decorator,action", [
+    ('@router.get("/settings")', "manage"),
+    ('@router.put("/settings")', "manage"),
+    ('@router.get("/stats")', "read"),
+    ('@router.post("/purge")', "manage"),
+    ('@router.get("")', "read"),
+    ('@router.get("/{log_id}")', "read"),
+])
+def test_every_endpoint_enforces_its_permission(src, decorator, action):
+    body = _handler_body(src, decorator)
+    assert f'_require(authorization, "{action}")' in body, (
+        f"{decorator} does not enforce requestlog.{action}"
+    )
+
+
+def test_require_helper_raises_403_not_a_silent_pass(src):
+    helper = src.split("async def _require", 1)[1].split("\nasync def ", 1)[0]
+    assert "check_user_permission" in helper
+    assert "status_code=403" in helper
+    assert "current_user=current_user" in helper, (
+        "the admin bypass is skipped, so every call pays an extra SELECT on users"
+    )
+
+
+# --------------------------------------------------------------------------
+# Route declaration order
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("literal", ['@router.get("/settings")', '@router.put("/settings")',
+                                     '@router.get("/stats")', '@router.post("/purge")'])
+def test_literal_routes_are_declared_before_the_catch_all(src, literal):
+    catch_all = src.index('@router.get("/{log_id}")')
+    assert src.index(literal) < catch_all, (
+        f"{literal} is declared after GET /{{log_id}}. FastAPI matches in declaration "
+        f"order and /{{log_id}} is a single-segment path, so it would swallow this route "
+        f"and the request would fail parsing 'settings' as an int."
+    )
+
+
+def test_list_route_is_declared_before_the_catch_all(src):
+    assert src.index('@router.get("")') < src.index('@router.get("/{log_id}")')
+
+
+# --------------------------------------------------------------------------
+# Query construction
+# --------------------------------------------------------------------------
+
+def test_filters_are_bound_never_interpolated(src):
+    """User-supplied filters reach the WHERE clause; they must arrive as $n
+    parameters."""
+    body = _handler_body(src, '@router.get("")')
+    # The only f-string interpolation allowed into SQL is the placeholder index
+    # and the assembled clause list, never a raw value.
+    for match in re.findall(r'add\("([^"]+)"', body):
+        assert "{n}" in match, f"filter clause {match!r} does not use a bound placeholder"
+
+
+def test_list_endpoint_scopes_non_privileged_callers(src):
+    """A caller with only `requestlog.read` sees their own rows plus the fleet's.
+
+    Widened from own-rows-only during review, deliberately. The `operator` role
+    is granted requestlog.read to "debug failing applies and ACME orders", but
+    an apply fails on the NODE and the node reports it over its own API key, so
+    the row carrying the diagnosis has `user_id IS NULL` — own-rows-only hid it
+    from exactly the role the grant was written for.
+
+    What must NOT widen is the part this test was written to protect: another
+    USER's captured bodies. Both halves are asserted below.
+    """
+    # Comments explain what the clause deliberately does NOT do, so match on
+    # code only — otherwise the prose describing the rule fails the test for it.
+    body = "\n".join(
+        line for line in _handler_body(src, '@router.get("")').splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "if not can_manage:" in body
+    assert "direction = 'inbound'" in body, (
+        "outbound rows are not scoped at all, so a caller with only "
+        "requestlog.read would see every CA and DNS call the backend ever made"
+    )
+    assert "user_id = $" in body, (
+        "a caller with only requestlog.read can see every other user's captured "
+        "request bodies"
+    )
+    assert "TARGET_INBOUND_AGENT" in body, (
+        "agent rows are hidden from requestlog.read, which is the one thing the "
+        "operator grant exists for"
+    )
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL rather than on target would also expose anonymous "
+        "traffic — failed logins and the usernames they carry, unauthenticated "
+        "probes — to any requestlog.read holder"
+    )
+
+
+def test_detail_endpoint_applies_the_same_scoping(src):
+    body = _handler_body(src, '@router.get("/{log_id}")')
+    assert "can_manage" in body
+    assert "404" in body, (
+        "the detail endpoint should 404 rather than 403 for a row the caller may not see, "
+        "so it does not confirm which ids exist"
+    )
+
+
+def test_list_response_omits_bodies(src):
+    """A 200-row page carrying two 8 KB JSONB blobs per row is a multi-megabyte
+    response; bodies belong to the detail endpoint."""
+    columns = src.split("_LIST_COLUMNS = ", 1)[1].split('"""', 2)[1]
+    assert "request_body," not in columns
+    assert "response_body," not in columns
+    assert "request_body_bytes" in columns, "the size is still useful in the list"
+
+
+def test_count_is_bounded(src):
+    body = _handler_body(src, '@router.get("")')
+    assert "LIMIT {count_cap}" in body or "count_cap" in body, (
+        "an unbounded COUNT(*) over request_logs is a sequential scan on every page change"
+    )
+    assert "total_is_estimate" in body
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_router_is_registered_in_main():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "from routers.request_logs import router as request_logs_router" in main_src
+    assert "app.include_router(request_logs_router)" in main_src, (
+        "the router is imported but never mounted, so every endpoint 404s"
+    )

backend/tests/test_request_log_settings.py

@@ -0,0 +1,258 @@
+"""v1.11.0: the retention policy the operator sees is the policy that runs.
+
+Two things drift silently and are caught here:
+
+  1. The defaults live in TWO places — the seed SQL in migrations.py and the
+     dataclass in utils/request_log_settings.py. If they disagree, a fresh
+     install and an upgraded install behave differently, which is the worst
+     kind of bug to chase.
+  2. Values in `system_settings` are operator-editable and arrive from asyncpg
+     as raw JSON *strings*. Anything out of range, mistyped or hand-edited must
+     be clamped rather than crash the writer loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    RequestLogConfig,
+    config_from_mapping,
+    get_config,
+    normalize_exclude_paths,
+    refresh_config,
+    set_config,
+)
+
+_MIGRATIONS = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "database", "migrations.py"
+)
+
+
+def _seeded_defaults():
+    """Parse the ('requestlog.x', 'value', ...) tuples out of the seed SQL."""
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        src = f.read()
+
+    body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    out = {}
+    for key, raw in re.findall(r"\('requestlog\.(\w+)', '(.*?)', 'requestlog'", body):
+        try:
+            out[key] = json.loads(raw)
+        except json.JSONDecodeError:
+            out[key] = raw
+    return out
+
+
+# --------------------------------------------------------------------------
+# Defaults must not drift between the seed and the code
+# --------------------------------------------------------------------------
+
+def test_seed_and_dataclass_defaults_agree():
+    seeded = _seeded_defaults()
+    assert seeded, "could not parse the requestlog seed rows out of migrations.py"
+
+    code = DEFAULT_CONFIG.as_dict()
+    for key, seed_value in seeded.items():
+        assert key in code, f"migrations seeds requestlog.{key} but RequestLogConfig has no such field"
+        assert code[key] == seed_value, (
+            f"requestlog.{key} default drifted: migrations.py seeds {seed_value!r} but "
+            f"RequestLogConfig has {code[key]!r}. A fresh install and an upgraded install "
+            f"would then behave differently."
+        )
+
+    for key in code:
+        assert key in seeded, (
+            f"RequestLogConfig has {key!r} but migrations.py does not seed requestlog.{key} — "
+            f"existing installs would silently fall back to the in-code default"
+        )
+
+
+def test_log_viewer_is_excluded_by_default():
+    assert "/api/request-logs" in DEFAULT_EXCLUDE_PATHS
+    assert "/api/health" in DEFAULT_EXCLUDE_PATHS
+    assert "/.well-known/acme-challenge" in DEFAULT_EXCLUDE_PATHS, (
+        "the ACME challenge endpoint returns key_authorization — logging it would store "
+        "the challenge secret"
+    )
+    assert "/api/agents/heartbeat" in DEFAULT_EXCLUDE_PATHS, (
+        "the agent heartbeat is the highest-volume POST in the system; logging it by "
+        "default would dominate the table"
+    )
+
+
+def test_error_retention_defaults_longer_than_success_retention():
+    assert DEFAULT_CONFIG.error_retention_days > DEFAULT_CONFIG.success_retention_days, (
+        "the whole point of splitting the two is to keep failures around after the "
+        "ordinary traffic has aged out"
+    )
+
+
+# --------------------------------------------------------------------------
+# Coercion and clamping
+# --------------------------------------------------------------------------
+
+def test_raw_json_strings_from_asyncpg_are_parsed():
+    cfg = config_from_mapping({
+        "enabled": True,
+        "max_body_bytes": 4096,
+        "sample_rate": 0.25,
+        "success_retention_days": 3,
+        "exclude_paths": ["/api/health", "/metrics"],
+    })
+    assert cfg.enabled is True
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.25
+    assert cfg.success_retention_days == 3
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+
+@pytest.mark.parametrize("raw,expected", [
+    ("true", True), ("false", False), ("1", True), ("0", False),
+    ("on", True), ("off", False), (1, True), (0, False), (True, True),
+])
+def test_boolean_coercion_accepts_hand_written_values(raw, expected):
+    cfg = config_from_mapping({"enabled": raw})
+    assert cfg.enabled is expected
+
+
+@pytest.mark.parametrize("field,value,expected", [
+    ("max_body_bytes", 10_000_000, 262144),
+    ("max_body_bytes", -5, 0),
+    ("success_retention_days", 0, 1),
+    ("success_retention_days", 9999, 365),
+    ("error_retention_days", 0, 1),
+    ("max_rows", 10, 1000),
+    ("prune_interval_minutes", 1, 5),
+    ("prune_interval_minutes", 99999, 1440),
+])
+def test_out_of_range_values_are_clamped_not_rejected(field, value, expected):
+    """A bad value in the table must not disable logging or crash the writer —
+    it is clamped to the nearest sane bound."""
+    cfg = config_from_mapping({field: value})
+    assert getattr(cfg, field) == expected
+
+
+@pytest.mark.parametrize("value,expected", [(1.5, 1.0), (-0.2, 0.0), ("0.4", 0.4)])
+def test_sample_rate_is_clamped(value, expected):
+    assert config_from_mapping({"sample_rate": value}).sample_rate == expected
+
+
+def test_garbage_values_fall_back_to_the_default():
+    cfg = config_from_mapping({"max_body_bytes": "not-a-number", "sample_rate": "abc"})
+    assert cfg.max_body_bytes == DEFAULT_CONFIG.max_body_bytes
+    assert cfg.sample_rate == DEFAULT_CONFIG.sample_rate
+
+
+def test_exclude_paths_shape_is_enforced():
+    out = normalize_exclude_paths(
+        ["/good", "no-leading-slash", "/" + "x" * 500, 42, "/also-good"],
+        DEFAULT_EXCLUDE_PATHS,
+    )
+    assert out == ("/good", "/also-good")
+
+
+def test_exclude_paths_count_is_bounded():
+    out = normalize_exclude_paths([f"/p{i}" for i in range(500)], DEFAULT_EXCLUDE_PATHS)
+    assert len(out) <= 64
+
+
+def test_empty_exclude_paths_falls_back_rather_than_logging_everything():
+    """An empty list would re-enable logging of health checks and the docs, and
+    flood the table — treat it as 'not configured'."""
+    assert normalize_exclude_paths([], DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+    assert normalize_exclude_paths(None, DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+
+
+def test_partial_mapping_keeps_the_other_defaults():
+    cfg = config_from_mapping({"sample_rate": 0.5})
+    assert cfg.sample_rate == 0.5
+    assert cfg.success_retention_days == DEFAULT_CONFIG.success_retention_days
+    assert cfg.enabled is DEFAULT_CONFIG.enabled
+
+
+# --------------------------------------------------------------------------
+# refresh_config
+# --------------------------------------------------------------------------
+
+def test_refresh_config_parses_the_raw_jsonb_strings_asyncpg_returns():
+    conn = AsyncMock()
+    conn.fetch = AsyncMock(return_value=[
+        {"key": "requestlog.enabled", "value": "false"},
+        {"key": "requestlog.max_body_bytes", "value": "4096"},
+        {"key": "requestlog.sample_rate", "value": "0.5"},
+        {"key": "requestlog.exclude_paths", "value": '["/api/health","/metrics"]'},
+    ])
+
+    with patch.object(request_log_settings, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.5
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+    set_config(DEFAULT_CONFIG)
+
+
+def test_refresh_config_keeps_the_previous_snapshot_on_db_failure():
+    """A transient pool error must not silently flip logging on or off."""
+    known = RequestLogConfig(enabled=False, sample_rate=0.1)
+    set_config(known)
+
+    with patch.object(request_log_settings, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.sample_rate == 0.1
+    set_config(DEFAULT_CONFIG)
+
+
+def test_get_config_is_synchronous_and_needs_no_database():
+    """The middleware calls this on every request; it must never await."""
+    assert not asyncio.iscoroutinefunction(get_config)
+    assert isinstance(get_config(), RequestLogConfig)
+
+
+# --------------------------------------------------------------------------
+# The Pydantic model the API exposes
+# --------------------------------------------------------------------------
+
+def test_api_model_defaults_match_the_dataclass():
+    from routers.request_logs import RequestLogSettings
+
+    model = RequestLogSettings().model_dump()
+    code = DEFAULT_CONFIG.as_dict()
+    for key, value in code.items():
+        assert model[key] == value, f"API model default for {key} disagrees with RequestLogConfig"
+
+
+@pytest.mark.parametrize("payload", [
+    {"max_body_bytes": 999999},
+    {"success_retention_days": 0},
+    {"error_retention_days": 400},
+    {"sample_rate": 1.5},
+    {"max_rows": 10},
+    {"prune_interval_minutes": 1},
+    {"exclude_paths": ["no-slash"]},
+    {"exclude_paths": ["/" + "x" * 300]},
+])
+def test_api_model_rejects_out_of_range_input(payload):
+    from pydantic import ValidationError
+
+    from routers.request_logs import RequestLogSettings
+
+    with pytest.raises(ValidationError):
+        RequestLogSettings(**payload)

backend/tests/test_request_log_sink.py

@@ -0,0 +1,276 @@
+"""v1.11.0: the batching writer must never slow down or break a request.
+
+One row per API call is the highest write volume in the system and the asyncpg
+pool (min=10/max=50) is shared with every handler and four background loops. So
+the hot path enqueues and returns; a single writer task batches and inserts.
+The properties pinned here:
+
+  * `offer()` never blocks and never raises — a full queue drops and counts;
+  * the parameter list stays aligned with the INSERT placeholders (a column
+    added to one and not the other would fail every write at runtime, in
+    production, with the migration already applied);
+  * a failed batch is dropped with a warning rather than killing the loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_settings  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+from utils.request_log_sink import (  # noqa: E402
+    RequestLogRow,
+    RequestLogSink,
+    _INSERT_SQL,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+
+def _row(**overrides):
+    base = dict(
+        request_id="abc123",
+        direction="inbound",
+        method="POST",
+        url="/api/backends",
+        path="/api/backends",
+        status_code=200,
+        duration_ms=12,
+        created_at=datetime(2026, 8, 11, 9, 0, tzinfo=timezone.utc),
+    )
+    base.update(overrides)
+    return RequestLogRow(**base)
+
+
+@pytest.fixture(autouse=True)
+def defaults(monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(sink_module, "get_config", lambda: request_log_settings._CACHE)
+
+
+def _set(monkeypatch, **overrides):
+    monkeypatch.setattr(request_log_settings, "_CACHE", replace(DEFAULT_CONFIG, **overrides))
+
+
+# --------------------------------------------------------------------------
+# SQL / parameter alignment
+# --------------------------------------------------------------------------
+
+def test_insert_placeholders_match_the_column_list():
+    columns = _INSERT_SQL.split("(", 1)[1].split(")", 1)[0]
+    n_columns = len([c for c in columns.split(",") if c.strip()])
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+
+    assert n_columns == n_placeholders, (
+        f"the INSERT names {n_columns} columns but binds {n_placeholders} placeholders — "
+        f"every write would fail at runtime, on a database where the migration has "
+        f"already succeeded"
+    )
+
+
+def test_row_produces_exactly_as_many_params_as_the_insert_binds():
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+    assert len(_row().to_params()) == n_placeholders, (
+        "RequestLogRow.to_params() drifted from _INSERT_SQL"
+    )
+
+
+def test_jsonb_params_are_serialized_strings_not_dicts():
+    """No JSONB codec is registered on this pool, so JSONB values travel as text
+    and are cast in SQL — handing asyncpg a dict raises."""
+    row = _row(
+        query_params={"page": "2"},
+        request_headers={"content-type": "application/json"},
+        request_body_value={"name": "web"},
+    )
+    params = row.to_params()
+
+    for value in params:
+        assert not isinstance(value, (dict, list)), (
+            f"{value!r} was passed as a Python container; asyncpg cannot bind it to a "
+            f"jsonb parameter"
+        )
+
+    assert json.loads(params[6]) == {"page": "2"}
+
+
+def test_client_ip_is_never_a_placeholder_string():
+    """client_ip is an INET column: 'unknown' or a comma-joined X-Forwarded-For
+    raises on INSERT."""
+    params = _row(client_ip=None).to_params()
+    assert params[12] is None
+
+
+def test_status_class_is_zero_when_there_was_no_response():
+    assert _row(status_code=None).status_class == 0
+    assert _row(status_code=204).status_class == 2
+    assert _row(status_code=503).status_class == 5
+
+
+# --------------------------------------------------------------------------
+# offer(): the hot path
+# --------------------------------------------------------------------------
+
+def test_offer_drops_and_counts_when_the_queue_is_full():
+    sink = RequestLogSink(maxsize=3, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(10):
+            sink.offer(_row())
+
+    asyncio.run(run())
+
+    assert sink.stats["queued"] == 3
+    assert sink.stats["dropped"] == 7, (
+        "a full queue must drop and count, never block the request or raise"
+    )
+
+
+def test_offer_never_raises_on_a_broken_row():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(None)  # not a RequestLogRow at all
+
+    asyncio.run(run())  # must not raise
+
+
+def test_offer_respects_the_kill_switch(monkeypatch):
+    _set(monkeypatch, enabled=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    asyncio.run(_offer(sink, _row()))
+    assert sink.stats["queued"] == 0
+
+
+def test_offer_respects_the_per_direction_switches(monkeypatch):
+    _set(monkeypatch, capture_outbound=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row(direction="outbound", target="acme"))
+        sink.offer(_row(direction="inbound"))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 1
+
+
+def test_sampling_never_drops_errors(monkeypatch):
+    """A sample rate of zero must still capture every failure — that is the whole
+    point of sampling successes only."""
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(20):
+            sink.offer(_row(status_code=200))
+        for _ in range(5):
+            sink.offer(_row(status_code=500))
+        for _ in range(5):
+            sink.offer(_row(status_code=None))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 10, (
+        "sampling removed error rows; only 1xx/2xx/3xx inbound traffic may be sampled out"
+    )
+
+
+def test_sampling_does_not_touch_outbound_rows(monkeypatch):
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row(direction="outbound", target="acme", status_code=200))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 5, (
+        "outbound calls are low-volume and high-value; sampling them away hides which CA "
+        "or DNS call was made"
+    )
+
+
+def test_capture_bodies_off_strips_the_payload_before_queueing(monkeypatch):
+    _set(monkeypatch, capture_bodies=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    row = _row(request_body_raw=b'{"a":1}', request_body_bytes=7)
+
+    asyncio.run(_offer(sink, row))
+
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 7, "the size must survive so growth is still measurable"
+
+
+async def _offer(sink, row):
+    sink.offer(row)
+
+
+# --------------------------------------------------------------------------
+# The writer
+# --------------------------------------------------------------------------
+
+def test_a_batch_is_written_with_one_executemany():
+    conn = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            return await sink.flush(timeout=1.0)
+
+    written = asyncio.run(run())
+
+    assert written == 5
+    assert conn.executemany.await_count == 1, (
+        "rows were inserted one at a time; that is one pool acquire per API call and the "
+        "pool has 50 connections"
+    )
+    sql, params = conn.executemany.await_args.args
+    assert "INSERT INTO request_logs" in sql
+    assert len(params) == 5
+
+
+def test_a_failed_batch_does_not_kill_the_writer():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("relation does not exist"))
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())  # must not raise
+    assert sink.stats["failed_batches"] == 1
+
+
+def test_the_connection_is_released_even_when_the_write_fails():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("boom"))
+    release = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", release):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())
+    assert release.await_count == 1, "a failed batch leaked a pooled connection"
+
+
+def test_flush_on_an_empty_queue_is_a_noop():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    assert asyncio.run(sink.flush(timeout=0.1)) == 0

backend/tests/test_router_path_shadowing.py

@@ -0,0 +1,141 @@
+"""
+v1.10.6 — literal API paths must never be declared after a parameterised one.
+
+Found in production on the v1.10.4 VIP-adoption feature: `@router.get("/discoveries")`
+sat at the bottom of routers/vip.py, below `@router.get("/{vip_id}")`. FastAPI matches
+routes in DECLARATION order, so every `GET /api/vip/discoveries` was answered by the
+`/{vip_id}` handler, which declares `vip_id: int` and therefore rejected the request with
+422 before `list_vip_discoveries` ever ran.
+
+Nothing about that failure was visible. The agents reported their discoveries correctly,
+the rows landed in `vip_discoveries`, and the HA/VIP page treats any non-OK response as
+"nothing to show" — so the adoption feature simply did not exist as far as the UI was
+concerned, with no error anywhere.
+
+These tests are a STATIC source scan on purpose: no imports, no app construction, no DB.
+They therefore also cover routers that cannot be imported in a bare test environment, and
+they keep covering routes added in the future.
+"""
+import pathlib
+import re
+
+import pytest
+
+ROUTERS_DIR = pathlib.Path(__file__).resolve().parents[1] / "routers"
+
+# `@router.get("/x")`, `@some_router.post("/x", ...)` — the path is the first string arg.
+_DECORATOR = re.compile(r'^@(?:\w+)\.(get|post|put|delete|patch)\(\s*[\'"]([^\'"]*)[\'"]')
+
+
+def _routes(source: str):
+    """[(line_no, verb, path)] in declaration order."""
+    out = []
+    for line_no, line in enumerate(source.splitlines(), 1):
+        match = _DECORATOR.match(line)
+        if match:
+            out.append((line_no, match.group(1), match.group(2)))
+    return out
+
+
+def _shadows(earlier: str, later: str) -> bool:
+    """True if `earlier` (declared first) swallows the literal path `later`.
+
+    Only literal paths can be silently swallowed, and only by a path that has the same
+    number of segments where every non-placeholder segment matches. The collection route
+    ("" or "/") is its own path and never collides.
+    """
+    if not later.strip("/") or not earlier.strip("/"):
+        return False
+    if "{" in later:
+        return False
+    if "{" not in earlier:
+        return False
+    earlier_segments = earlier.strip("/").split("/")
+    later_segments = later.strip("/").split("/")
+    if len(earlier_segments) != len(later_segments):
+        return False
+    return all(
+        e.startswith("{") or e == l
+        for e, l in zip(earlier_segments, later_segments)
+    )
+
+
+def _shadowed_routes(path: pathlib.Path):
+    routes = _routes(path.read_text())
+    found = []
+    for index, (line_no, verb, route_path) in enumerate(routes):
+        for prior_line, prior_verb, prior_path in routes[:index]:
+            if prior_verb == verb and _shadows(prior_path, route_path):
+                found.append(
+                    f"{path.name}:{line_no} {verb.upper()} {route_path} is swallowed by "
+                    f"{prior_path} declared at line {prior_line}"
+                )
+    return found
+
+
+# ----------------------------------------------------------------------------
+# 1. The specific regression: /api/vip/discoveries must outrank /{vip_id}
+# ----------------------------------------------------------------------------
+
+def test_vip_discoveries_declared_before_vip_id():
+    routes = _routes((ROUTERS_DIR / "vip.py").read_text())
+    get_paths = [path for _line, verb, path in routes if verb == "get"]
+
+    assert "/discoveries" in get_paths, "the discoveries endpoint disappeared"
+    assert "/{vip_id}" in get_paths, "the get-one endpoint disappeared"
+    assert get_paths.index("/discoveries") < get_paths.index("/{vip_id}"), (
+        "GET /discoveries is declared after GET /{vip_id}; FastAPI will route "
+        "/api/vip/discoveries into get_vip and answer 422, silently emptying the "
+        "adoption panel"
+    )
+
+
+# ----------------------------------------------------------------------------
+# 2. The general guard: no literal path anywhere is shadowed
+# ----------------------------------------------------------------------------
+
+def test_no_literal_route_is_shadowed_in_any_router():
+    problems = []
+    for router_file in sorted(ROUTERS_DIR.glob("*.py")):
+        problems.extend(_shadowed_routes(router_file))
+
+    assert not problems, (
+        "literal route(s) declared after a parameterised route that swallows them:\n  "
+        + "\n  ".join(problems)
+    )
+
+
+# ----------------------------------------------------------------------------
+# 3. The detector itself must actually detect (guards against a vacuous pass)
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize(
+    "earlier,later,expected",
+    [
+        ("/{vip_id}", "/discoveries", True),        # the v1.10.4 bug
+        ("/{vip_id}", "/{other}", False),           # two placeholders never shadow
+        ("/{vip_id}", "", False),                   # collection route is its own path
+        ("/{vip_id}/apply", "/adopt", False),       # different segment counts
+        ("/{vip_id}/apply", "/adopt/now", False),   # literal mismatch in segment 2
+        ("/{vip_id}/{action}", "/adopt/now", True), # both segments placeheld
+        ("/vips", "/discoveries", False),           # literal never shadows a literal
+    ],
+)
+def test_shadow_detector_semantics(earlier, later, expected):
+    assert _shadows(earlier, later) is expected
+
+
+def test_detector_flags_the_original_declaration_order():
+    """A synthetic file in the pre-fix order must be reported, so a future refactor that
+    breaks the detector cannot make the guard above pass vacuously."""
+    source = (
+        '@router.get("")\n'
+        "async def list_vips(): ...\n"
+        '@router.get("/{vip_id}")\n'
+        "async def get_vip(vip_id: int): ...\n"
+        '@router.get("/discoveries")\n'
+        "async def list_vip_discoveries(): ...\n"
+    )
+    routes = _routes(source)
+    assert [verb for _l, verb, _p in routes] == ["get", "get", "get"]
+    assert _shadows(routes[1][2], routes[2][2]) is True

backend/tests/test_security_advisories_auth.py

@@ -0,0 +1,223 @@
+"""Regression tests for the 2026-07 security advisories.
+
+Covers:
+- GHSA-7rhv-c5pc-69r8 (CRITICAL RCE): agent script-template management must
+  require the agents.version permission, not merely authentication.
+- GHSA-3p5c-m5m4-mjpx (missing auth): agent data-plane endpoints must require a
+  valid X-API-Key, and operator/UI endpoints must require a JWT. An anonymous
+  caller must never get a 200 with sensitive data.
+
+These are behavioral assertions via FastAPI's TestClient. The auth checks were
+deliberately moved ahead of any DB access, so an unauthenticated request is
+rejected without needing a database — the same approach as the existing
+test_ssl_list_endpoint_auth.py. Accepted rejection statuses are 401/403/422
+(never 200-with-data).
+"""
+import re
+import os
+import pytest
+
+REJECT = (401, 403, 422)
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: agent data-plane endpoints must reject a missing X-API-Key
+# --------------------------------------------------------------------------
+
+def test_agent_config_requires_api_key(client):
+    """GET /api/agents/{name}/config leaked the full haproxy.cfg without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/config")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: agent config served without X-API-Key ({res.status_code})"
+    )
+
+
+def test_agent_ssl_certificates_requires_api_key(client):
+    """GET /api/agents/{name}/ssl-certificates leaked SSL private keys without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/ssl-certificates")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: SSL certs (private keys!) served without X-API-Key ({res.status_code})"
+    )
+    if res.status_code == 200:
+        assert "private_key_content" not in res.text
+
+
+def test_agent_upgrade_status_requires_api_key(client):
+    res = client.get("/api/agents/prod-haproxy-1/upgrade-status")
+    assert res.status_code in REJECT
+
+
+def test_agent_pending_requests_requires_api_key(client):
+    res = client.get("/api/configuration/agents/prod-haproxy-1/pending-requests")
+    assert res.status_code in REJECT
+
+
+def test_agent_heartbeat_by_name_requires_api_key(client):
+    res = client.post("/api/agents/heartbeat", json={"name": "rogue-poc"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless heartbeat/auto-register accepted ({res.status_code})"
+    )
+
+
+def test_agent_heartbeat_by_id_requires_api_key(client):
+    res = client.post("/api/agents/1/heartbeat", json={"name": "spoofed"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless by-id heartbeat state-spoof accepted ({res.status_code})"
+    )
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: operator/UI endpoints must reject a missing JWT
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_requires_jwt(client):
+    """GET /api/agents (RCE read-back channel) was served without a JWT."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", ["/api/health/deep", "/api/health/agents", "/api/health/clusters"])
+def test_detailed_health_requires_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, f"{path} served without a JWT ({res.status_code})"
+
+
+def test_simple_health_stays_public(client):
+    """The liveness probe endpoint (/api/health) must remain UNAUTHENTICATED.
+
+    It reports 200 when healthy and 503 when the DB is unreachable (as in this
+    no-DB test env); what matters for the k8s probe is that it is never gated
+    behind auth (401/403). We only added auth to /api/health/{deep,agents,clusters}.
+    """
+    res = client.get("/api/health")
+    assert res.status_code not in (401, 403), (
+        f"Regression: /api/health liveness probe now requires auth ({res.status_code}) — "
+        f"this breaks k8s liveness/readiness"
+    )
+
+
+def test_dashboard_stats_requires_jwt(client):
+    res = client.get("/api/dashboard-stats/stats?cluster_id=1")
+    assert res.status_code in REJECT
+
+
+def test_ssl_config_versions_requires_jwt(client):
+    res = client.get("/api/ssl/certificates/1/config-versions")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-7rhv (CRITICAL RCE): script-template management
+# --------------------------------------------------------------------------
+
+def test_script_template_write_requires_auth(client):
+    """Anonymous POST must be rejected outright."""
+    res = client.post("/api/agents/script-templates/linux",
+                      json={"script_content": "#!/bin/bash\nid", "version": "9.9.9"})
+    assert res.status_code in REJECT
+
+
+def test_script_template_read_requires_auth(client):
+    res = client.get("/api/agents/script-templates/linux")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): sibling endpoints exposing the SAME class of data
+# (found during post-merge review — must also require a JWT)
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/haproxy-cluster-pools/1/agents",   # full agent inventory — same class as GET /api/agents
+    "/api/pools",
+    "/api/haproxy-cluster-pools",
+    "/api/dashboard/stats",
+    "/api/dashboard/overview",               # optional-auth pattern — leaked stats/names/alerts anonymously
+    "/api/haproxy/stats",
+    "/api/waf/rules",
+    "/api/health/errors",
+])
+def test_sibling_inventory_endpoints_require_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c (round 2) regression: {path} served without a JWT ({res.status_code}) — "
+        f"anonymous access to inventory/topology/WAF/error data"
+    )
+
+
+def test_pool_agents_no_anonymous_inventory_leak(client):
+    """The richest bypass: /api/haproxy-cluster-pools/{id}/agents must not leak inventory."""
+    res = client.get("/api/haproxy-cluster-pools/1/agents")
+    assert res.status_code in REJECT
+    if res.status_code == 200:
+        assert "ip_address" not in res.text and "hostname" not in res.text
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): agent webhooks must return 401 (not a 200 error body)
+# for anonymous callers — the auth raise must propagate, not be swallowed.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/agents/some-agent/config-applied",
+    "/api/agents/some-agent/config-validation-failed",
+    "/api/agents/some-agent/config-sync",
+])
+def test_agent_webhooks_reject_anonymous_with_401(client, path):
+    res = client.post(path, json={})
+    assert res.status_code in REJECT, (
+        f"{path} returned {res.status_code} for an anonymous caller — the auth "
+        f"rejection must be a 401/403, not a swallowed 200 error body"
+    )
+    # Specifically must NOT be a 200 "status: error" body.
+    assert res.status_code != 200
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): GET /api/agents must accept EITHER a JWT OR an agent
+# X-API-Key. Anonymous (neither) is still rejected — agents send a key, so a
+# JWT-only gate would break them (verified end-to-end in the localtest smoke).
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_still_rejects_fully_anonymous(client):
+    """No JWT and no X-API-Key -> 401 (the agent-key accept path needs a valid key)."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+def test_generate_uninstall_script_requires_auth(client):
+    """Agent-management endpoint must not be anonymously reachable (JWT or agent key)."""
+    res = client.get("/api/agents/generate-uninstall-script/linux")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", [
+    "/api/config/validate",
+    "/api/config/optimize",
+    "/api/config/templates/default/generate",
+])
+def test_config_compute_endpoints_require_auth(client, path):
+    """Config compute endpoints (run a HAProxy validator on caller input) were
+    optional-auth; now require a JWT. The dependency rejects before body parsing."""
+    res = client.post(path, json={})
+    assert res.status_code in REJECT
+
+
+def test_script_template_write_enforces_agents_version_permission():
+    """Static guarantee: the write handler checks agents.version (not just authN).
+
+    A behavioral 403-for-viewer test would need a seeded DB + a minted viewer JWT;
+    instead we assert the permission gate is present in source, mirroring the
+    existing audit-style source tests. This is the core RCE fix (GHSA-7rhv).
+    """
+    src_path = os.path.join(os.path.dirname(__file__), "..", "routers", "agent.py")
+    with open(src_path, "r") as f:
+        src = f.read()
+    # Isolate the save_agent_script_template handler body.
+    m = re.search(r"async def save_agent_script_template\(.*?\n(.*?)\n@router\.", src, re.DOTALL)
+    assert m, "save_agent_script_template handler not found"
+    body = m.group(1)
+    assert 'check_user_permission' in body and '"agents", "version"' in body, (
+        "GHSA-7rhv regression: script-template WRITE no longer enforces the "
+        "agents.version permission — any JWT holder could poison the root install script"
+    )

backend/tests/test_site_wizard_round11.py

@@ -276,14 +276,16 @@ def test_categorize_routes_directives_correctly():
 
 def test_emit_buckets_flushed_in_canonical_order():
     """The flush block at end of frontend processing must list buckets
-    in: prelude → stick → tcp_req → acl → http_req → http_resp →
+    in: prelude → filter → stick → tcp_req → acl → http_req → http_resp →
     redirect → use_be → default_be. Pre-fix `http-request` rules
     interleaved with `use_backend` rules in source order, producing
-    HAProxy parser warnings."""
+    HAProxy parser warnings. (Issue #38 added the `filter` bucket, flushed
+    right after `prelude` so SPOE `filter` lines precede `send-spoe-group`.)"""
     src = _gen_src()
     flush_match = re.search(
         r'for\s+_bucket_key\s+in\s+\(\s*'
         r'"prelude"\s*,\s*'
+        r'"filter"\s*,\s*'
         r'"stick"\s*,\s*'
         r'"tcp_req"\s*,\s*'
         r'"acl"\s*,\s*'

backend/tests/test_spoe_filter_import.py

@@ -0,0 +1,203 @@
+"""
+Issue #38 regression tests: HAProxy SPOE `filter` + frontend `log-format` support.
+
+Bug: the bulk-config parser recognised only a fixed set of frontend directives,
+so `filter spoe engine coraza config ...` and `log-format ...` were silently
+dropped on import / manual edit. This regenerated a config missing the SPOE
+engine definition, so HAProxy failed with
+"unable to find SPOE engine 'coraza' used by the send-spoe-group 'coraza-req'".
+
+These tests verify the end-to-end fix without requiring a database:
+1. parser captures `filter` + `log-format` into the new ParsedFrontend fields;
+2. `http-request send-spoe-group` is still preserved (regression guard);
+3. the generator's directive categoriser + bucket flush order emit `filter`
+   BEFORE the `http-request send-spoe-group` rules and keep `log-format`;
+4. reject/rollback restores the new columns;
+5. a non-SPOE frontend is completely unaffected (zero-impact).
+"""
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.haproxy_config_parser import parse_haproxy_config, ParsedFrontend
+from services.haproxy_config import _categorize_haproxy_directive
+from models.frontend import FrontendConfig
+
+
+# The exact frontend/backend config reported in Issue #38 (Coraza-SPOA).
+ISSUE_38_CONFIG = r"""
+frontend web-frontend
+    bind *:8073
+    mode http
+        log-format "%ci:%cp\ [%t]\ %ft\ %b/%s\ %ST\ %B\ %{+Q}r\ %[var(txn.coraza.id)]\ waf-hit:\ %[var(txn.coraza.fail)]"
+        filter spoe engine coraza config /etc/haproxy/coraza.cfg
+      http-request set-var(txn.coraza.app) str(haproxy_waf)
+     http-request send-spoe-group coraza coraza-req
+    http-request deny if { var(txn.coraza.fail) -m int eq 1 }
+    default_backend web-backend
+
+backend web-backend
+    balance roundrobin
+    mode http
+    server server1 192.168.1.10:443 weight 100 ssl verify none
+
+backend coraza-spoa
+    mode tcp
+    option spop-check
+    server coraza_spoa 192.168.12.21:9000
+"""
+
+
+def _get_frontend(parse_result, name):
+    for fe in parse_result.frontends:
+        if fe.name == name:
+            return fe
+    return None
+
+
+class TestParserCapturesSpoe:
+    def test_filter_and_log_format_captured(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe is not None, "web-frontend should be parsed and kept"
+        assert fe.filters is not None
+        assert "filter spoe engine coraza config /etc/haproxy/coraza.cfg" in fe.filters
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format")
+        # the escaped/quoted format string must be preserved verbatim
+        assert "%[var(txn.coraza.fail)]" in fe.log_format
+
+    def test_send_spoe_group_still_preserved(self):
+        # Regression guard: http-request rules (incl. send-spoe-group) must
+        # still be collected into request_headers as before.
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe.request_headers is not None
+        assert "send-spoe-group coraza coraza-req" in fe.request_headers
+
+    def test_multiple_filters_preserved_in_order(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    filter compression
+    filter spoe engine coraza config /etc/haproxy/coraza.cfg
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        lines = fe.filters.split("\n")
+        assert lines == [
+            "filter compression",
+            "filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+        ]
+
+    def test_log_format_sd_variant_captured(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    log-format-sd "[exampleSDID@1234 field=value]"
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format-sd")
+
+
+class TestGeneratorOrderingContract:
+    """The generator routes directives into ordered buckets. Verify SPOE
+    correctness at the (pure) categoriser + documented flush-order level."""
+
+    def test_filter_routes_to_filter_bucket(self):
+        assert _categorize_haproxy_directive("    filter spoe engine coraza config /x.cfg") == "filter"
+
+    def test_send_spoe_group_routes_to_http_req(self):
+        assert _categorize_haproxy_directive("    http-request send-spoe-group coraza coraza-req") == "http_req"
+
+    def test_log_format_routes_to_prelude(self):
+        assert _categorize_haproxy_directive('    log-format "%ci:%cp"') == "prelude"
+        assert _categorize_haproxy_directive('    log-format-sd "[x]"') == "prelude"
+
+    def test_flush_order_places_filter_before_http_req(self):
+        # The bucket flush order is the single source of truth for emission
+        # ordering. Assert `filter` is flushed before `http_req` (and after
+        # `prelude`), guaranteeing `filter ...` renders before
+        # `http-request send-spoe-group ...`.
+        src = _read_source("services/haproxy_config.py")
+        m = re.search(r"for _bucket_key in \((.*?)\):", src, re.DOTALL)
+        assert m, "bucket flush loop not found"
+        order = re.findall(r'"(\w+)"', m.group(1))
+        assert "filter" in order, "new 'filter' bucket missing from flush order"
+        assert order.index("prelude") < order.index("filter") < order.index("http_req")
+
+
+class TestModelAndRollback:
+    def test_model_has_passthrough_fields(self):
+        fc = FrontendConfig(
+            name="f", bind_port=80,
+            filters="filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+            log_format='log-format "%ci"',
+        )
+        assert fc.filters.startswith("filter spoe")
+        assert fc.log_format.startswith("log-format")
+
+    def test_dataclass_defaults_none(self):
+        fe = ParsedFrontend(name="f")
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_rollback_restores_new_columns(self):
+        # Reject/rollback of a frontend UPDATE must restore the new columns,
+        # otherwise the rejected (new) filters/log_format would persist.
+        src = _read_source("utils/entity_snapshot.py")
+        assert "log_format = $" in src
+        assert "filters = $" in src
+        assert "old_values.get('log_format')" in src
+        assert "old_values.get('filters')" in src
+
+
+class TestZeroImpact:
+    def test_non_spoe_frontend_unaffected(self):
+        cfg = """
+frontend plain
+    bind *:80
+    mode http
+    option httplog
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "plain")
+        # No filter / log-format present → new fields stay None (no behaviour change)
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_spop_check_backend_roundtrips_without_warning(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        be = next((b for b in result.backends if b.name == "coraza-spoa"), None)
+        assert be is not None, "coraza-spoa backend should import"
+        assert be.mode == "tcp"
+        assert be.options and "option spop-check" in be.options
+        # spop-check is now a known option → no spurious 'unknown option' warning
+        assert not any(
+            "coraza-spoa" in w and "spop-check" in w and "Unknown" in w
+            for w in result.warnings
+        )
+
+
+def _read_source(relpath):
+    base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    with open(os.path.join(base, relpath), "r", encoding="utf-8") as fh:
+        return fh.read()

backend/tests/test_ssrf_guard.py

@@ -0,0 +1,68 @@
+"""Unit tests for the SSRF guard (GHSA-3vh4-gvxx-wm2p).
+
+The guard protects server-side fetches of ACME `directory_url` values. This
+project uses only public ACME CAs, so every non-public IP must be rejected.
+Tests avoid real network/DNS by using IP literals and scheme checks.
+"""
+import asyncio
+import pytest
+
+from utils.ssrf_guard import is_public_ip, assert_public_url, SSRFValidationError
+
+
+# ---- is_public_ip -----------------------------------------------------------
+
+@pytest.mark.parametrize("ip", [
+    "8.8.8.8", "1.1.1.1", "93.184.216.34",  # public
+])
+def test_public_ips_allowed(ip):
+    assert is_public_ip(ip) is True
+
+
+@pytest.mark.parametrize("ip", [
+    "127.0.0.1",            # loopback
+    "10.0.0.5",             # RFC1918
+    "172.19.0.1",           # RFC1918 (the SSRF PoC docker gateway)
+    "192.168.1.1",          # RFC1918
+    "169.254.169.254",      # link-local / cloud metadata
+    "0.0.0.0",              # unspecified
+    "::1",                  # IPv6 loopback
+    "fe80::1",              # IPv6 link-local
+    "::ffff:127.0.0.1",     # IPv4-mapped IPv6 loopback (R18c bypass)
+    "::ffff:169.254.169.254",  # IPv4-mapped metadata
+    "not-an-ip",            # garbage
+])
+def test_non_public_ips_rejected(ip):
+    assert is_public_ip(ip) is False
+
+
+# ---- assert_public_url ------------------------------------------------------
+
+def _raises(url):
+    with pytest.raises(SSRFValidationError):
+        asyncio.run(assert_public_url(url))
+
+
+def test_rejects_non_https_scheme():
+    # The SSRF PoC used http:// against an internal listener.
+    _raises("http://172.19.0.1:2121/internal-secret")
+    _raises("http://8.8.8.8/")            # even a public IP over http is refused
+    _raises("file:///etc/passwd")
+    _raises("gopher://8.8.8.8/")
+
+
+def test_rejects_private_ip_literals():
+    _raises("https://127.0.0.1/")
+    _raises("https://10.0.0.5/")
+    _raises("https://169.254.169.254/latest/meta-data/")
+    _raises("https://[::1]/")
+
+
+def test_rejects_empty_or_hostless():
+    _raises("")
+    _raises("https://")
+
+
+def test_allows_public_ip_literal_https():
+    # A public IP literal over https must pass (no DNS needed).
+    asyncio.run(assert_public_url("https://8.8.8.8/directory"))

backend/tests/test_version_consistency.py

@@ -0,0 +1,71 @@
+"""v1.8.7: app version is single-source and cannot silently drift.
+
+The UI shows the version via GET /api/version, which returns main.py's `_version_info`. That MUST be
+sourced from the one canonical file backend/version.json (co-located with main.py so `COPY . .`
+bakes it into every image, regardless of pipeline). main.py's in-code fallback must NOT be a real
+version, otherwise it drifts when version.json is bumped but the constant is forgotten — exactly
+what left the UI reporting 1.8.4 after 1.8.5/1.8.6 shipped. These checks fail loudly on regression.
+"""
+import ast
+import json
+import os
+import re
+
+import pytest
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))  # backend/
+_VERSION_JSON = os.path.join(_BACKEND, "version.json")
+_MAIN = os.path.join(_BACKEND, "main.py")
+_SEMVER = re.compile(r"^\d+\.\d+\.\d+$")
+
+
+def test_canonical_version_file_exists_and_valid():
+    assert os.path.exists(_VERSION_JSON), "backend/version.json (single source of truth) is missing"
+    with open(_VERSION_JSON) as f:
+        data = json.load(f)
+    assert _SEMVER.match(data.get("version", "")), \
+        f"backend/version.json version is not semver: {data.get('version')!r}"
+    assert data.get("releaseName"), "backend/version.json must have a releaseName"
+
+
+def _main_fallback_version_info():
+    """The literal dict assigned to _version_info in main.py (the in-code fallback)."""
+    with open(_MAIN) as f:
+        tree = ast.parse(f.read())
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign) and isinstance(node.value, ast.Dict):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == "_version_info":
+                    return ast.literal_eval(node.value)
+    return None
+
+
+def test_main_has_no_hardcoded_real_version():
+    fb = _main_fallback_version_info()
+    assert fb is not None, "could not find the _version_info fallback literal in main.py"
+    # Must be a neutral marker, never a real version that can drift out of sync.
+    assert not _SEMVER.match(str(fb.get("version", ""))), (
+        f"main.py hardcodes a real version {fb.get('version')!r}; it must be a neutral marker "
+        f"(e.g. 'unknown') so the version stays single-source in backend/version.json"
+    )
+
+
+def test_main_loads_the_canonical_file_first():
+    # The first candidate path main.py reads must resolve to the co-located backend/version.json,
+    # so the correct version is available in every image (not dependent on CI staging).
+    with open(_MAIN) as f:
+        src = f.read()
+    assert 'os.path.dirname(__file__), "version.json"' in src, \
+        "main.py must read version.json co-located with the module (backend/version.json)"
+
+
+def test_frontend_package_json_matches_when_present():
+    # Only meaningful in a full-repo checkout; the backend image build context (./backend) has no frontend/.
+    pkg = os.path.join(os.path.dirname(_BACKEND), "frontend", "package.json")
+    if not os.path.exists(pkg):
+        pytest.skip("frontend/package.json not in this context (e.g. backend-only image build)")
+    with open(_VERSION_JSON) as f:
+        canonical = json.load(f)["version"]
+    with open(pkg) as f:
+        fe = json.load(f)["version"]
+    assert fe == canonical, f"frontend/package.json {fe!r} != backend/version.json {canonical!r}"

backend/tests/test_vip_adoption_instance.py

@@ -0,0 +1,641 @@
+"""
+v1.10.8 — the four adoption defects found while testing v1.10.4 on a live HA pair.
+
+B1  The Apply Management "View Change" diff did not recognise the `adopt` action, so the
+    version fell through to the generic HAProxy diff and rendered the cluster's whole
+    haproxy.cfg as removed.
+
+B2  `vip_discoveries.adopted_vip_id` is write-once and nothing clears it, while a VIP is only
+    ever SOFT-deleted — so `ON DELETE SET NULL` never fires. Rejecting an adoption therefore
+    hid the node from the panel permanently: the VIP was gone from the VIP list too, and the
+    agent does not re-report a file whose hash has not changed. Adoptability is now derived
+    from whether the linked VIP is still active.
+
+B3  Adoption took only the node that was clicked. On a two-node pair that meant: adopting the
+    BACKUP produced a VIP whose apply fails ("exactly one member must be MASTER"), adopting the
+    MASTER left the peer unmanaged, and adopting the peer afterwards hit the VRID-collision
+    guard with 409. The pair could never be completed from the panel.
+
+B4  Worst of the four. `render_keepalived_conf` emits the unicast block only when it has peer
+    addresses, so a single-member adoption of a UNICAST instance silently dropped it and
+    keepalived fell back to multicast on that node while its peer stayed unicast — they stop
+    seeing each other and BOTH claim the VIP.
+
+B3 and B4 share one root and one fix: adoption now resolves the whole VRRP instance, keyed on
+(virtual_router_id, virtual address) exactly as keepalived groups nodes.
+
+These are source-level and unit tests: the adoption endpoint needs a live database, so the
+behaviour that can be exercised without one is pinned here, and the SQL/flow invariants are
+pinned by reading the module.
+"""
+import pathlib
+import re
+
+import pytest
+
+BACKEND = pathlib.Path(__file__).resolve().parents[1]
+VIP_ROUTER = (BACKEND / "routers" / "vip.py").read_text()
+CLUSTER_ROUTER = (BACKEND / "routers" / "cluster.py").read_text()
+RENDERER = (BACKEND / "services" / "keepalived_config.py").read_text()
+
+
+def _adopt_body() -> str:
+    start = VIP_ROUTER.index("async def adopt_vip")
+    return VIP_ROUTER[start:]
+
+
+# ----------------------------------------------------------------------------
+# B1 — the diff must recognise `adopt`
+# ----------------------------------------------------------------------------
+
+def test_view_change_diff_recognises_the_adopt_action():
+    match = re.search(r"vip_match = re\.search\(r'vip-\(\\d\+\)-\(([^)]+)\)'", CLUSTER_ROUTER)
+    assert match, "the vip version regex moved; re-point this test"
+    actions = set(match.group(1).split("|"))
+    assert actions == {"create", "update", "delete", "adopt"}, (
+        f"the View Change diff recognises {sorted(actions)}. An action missing here does not "
+        f"degrade gracefully: the version falls through to the generic HAProxy diff and shows "
+        f"the cluster's whole haproxy.cfg as removed."
+    )
+
+
+def test_every_staged_vip_action_is_covered_by_the_diff_regex():
+    """Whatever _stage_vip_version can be called with must be in that alternation."""
+    staged = set(re.findall(r'_stage_vip_version\(conn, vip_id, "(\w+)"', VIP_ROUTER))
+    match = re.search(r"vip_match = re\.search\(r'vip-\(\\d\+\)-\(([^)]+)\)'", CLUSTER_ROUTER)
+    recognised = set(match.group(1).split("|"))
+    assert staged, "no _stage_vip_version call sites found; re-point this test"
+    assert staged <= recognised, (
+        f"staged action(s) {sorted(staged - recognised)} are not recognised by the View Change "
+        f"diff regex {sorted(recognised)}"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B2 — adoptability follows the VIP's liveness, not the bare link
+# ----------------------------------------------------------------------------
+
+def test_discovery_api_exposes_whether_the_adopted_vip_still_stands():
+    assert "adopted_vip_active" in VIP_ROUTER, (
+        "the discovery payload no longer reports whether the adopted VIP is still active; the "
+        "UI would go back to hiding a rejected adoption forever"
+    )
+    assert re.search(r"LEFT JOIN vip_instances av ON av\.id = d\.adopted_vip_id", VIP_ROUTER), (
+        "the discovery query must join the linked VIP to report its is_active"
+    )
+
+
+def test_adopt_refuses_only_while_the_previous_adoption_still_stands():
+    body = _adopt_body()
+    assert re.search(r'if disc\["adopted_vip_id"\] and disc\["adopted_vip_active"\]', body), (
+        "adopt must refuse only when the linked VIP is still ACTIVE — refusing on the bare link "
+        "makes a rejected adoption impossible to retry, because nothing ever clears the column"
+    )
+
+
+def test_nothing_clears_adopted_vip_id_so_the_derivation_is_load_bearing():
+    """If a future change starts clearing the column, this test should be revisited rather than
+    silently left in place — the derived flag is what makes reject recoverable today."""
+    writes = re.findall(r"UPDATE vip_discoveries SET adopted_vip_id = (\S+)", VIP_ROUTER)
+    assert writes, "no adopted_vip_id write found; re-point this test"
+    assert all(w != "NULL" for w in writes), (
+        "adopted_vip_id is now cleared somewhere — re-check that the adopted_vip_active "
+        "derivation and this test still describe reality"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B3 — the whole VRRP instance is adopted, not one node
+# ----------------------------------------------------------------------------
+
+def test_participants_are_resolved_by_vrid_and_address():
+    assert "async def _collect_instance_participants" in VIP_ROUTER
+    start = VIP_ROUTER.index("async def _collect_instance_participants")
+    end = VIP_ROUTER.index("@router.post(\"/adopt\")", start)
+    body = VIP_ROUTER[start:end]
+    assert 'v.get("virtual_router_id") == vrid' in body and 'v.get("virtual_ip") == virtual_ip' in body, (
+        "instance identity must be (VRID, address) — the same key keepalived uses to decide two "
+        "nodes are one VRRP group"
+    )
+    assert 'if r["parse_error"]' in body, "a node whose config failed to parse must not become a member"
+    assert 'r["adopted_vip_id"] and r["adopted_vip_active"]' in body, (
+        "a node already held by a STANDING VIP must not be pulled into a second one"
+    )
+
+
+def test_adopt_inserts_one_member_per_participant():
+    body = _adopt_body()
+    insert = body.index("INSERT INTO vip_members")
+    preceding = body[:insert]
+    assert "for p in participants:" in preceding, (
+        "members must be inserted in a loop over the resolved participants; a single insert is "
+        "the half-adoption bug"
+    )
+    assert 'p["config_hash"]' in body[insert:insert + 800], (
+        "each member must carry ITS OWN takeover hash — the one-shot takeover guard is per node"
+    )
+
+
+def test_adopt_requires_exactly_one_master_across_the_instance():
+    body = _adopt_body()
+    assert 'roles.count("MASTER") != 1' in body, (
+        "adoption must reject an instance that does not have exactly one MASTER, instead of "
+        "letting apply fail later with 'exactly one member must be MASTER'"
+    )
+
+
+def test_adopt_enforces_one_active_vip_per_agent():
+    body = _adopt_body()
+    assert "already a member of VIP" in body, (
+        "adoption must enforce the one-active-VIP-per-agent rule that create/update enforce via "
+        "_validate_members_against_pool; a second membership never converges"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B4 — a unicast instance can never be half-adopted
+# ----------------------------------------------------------------------------
+
+def test_renderer_only_emits_unicast_when_it_has_peers():
+    """The property that makes B4 dangerous. Pinned so the guard below keeps its reason."""
+    assert re.search(r"if use_unicast and peer_ips:", RENDERER), (
+        "the renderer no longer gates the unicast block on having peers; re-derive whether the "
+        "adoption guard is still needed"
+    )
+
+
+def test_adopt_refuses_a_unicast_peer_that_is_not_being_adopted():
+    body = _adopt_body()
+    assert "declared_peers" in body and "member_ips" in body, (
+        "adoption must verify every declared unicast peer is among the nodes being adopted"
+    )
+    assert "fall back to multicast" in body, (
+        "the refusal must explain the consequence — silently dropping a peer puts both nodes in "
+        "MASTER state on the same address"
+    )
+
+
+def test_adopt_refuses_to_strand_any_node_that_references_the_address():
+    """The half-adoption hole that instance resolution alone does not close.
+
+    Participant resolution can only match a node it can READ, that is ENABLED and that is in the
+    SAME pool. Each of those is a door a real member leaves through silently, and the nodes that
+    remain get rewritten while it keeps serving the same address unmanaged. Seen for real: one
+    node of a pair had a missing closing brace, so it parsed to nothing while its partner parsed
+    cleanly. Rather than guard each door, the endpoint asks whether ANY reported config mentions
+    this address and is not among the nodes being adopted.
+    """
+    body = _adopt_body()
+    assert "raw_config_masked LIKE" in body, (
+        "the check must be scoped to configs that reference THIS virtual address, so an unrelated "
+        "file elsewhere in the fleet does not block every adoption"
+    )
+    assert "NOT (a.id = ANY($2::int[]))" in body, (
+        "the guard must catch every node that is NOT a participant, not just the unparseable "
+        "ones — a disabled agent and a peer in another pool are stranded exactly the same way"
+    )
+    assert "COALESCE(av.is_active, FALSE) = FALSE" in body and "d.is_managed = FALSE" in body, (
+        "a node already under management is not stranded and must not block adoption"
+    )
+    for reason in ("could not be parsed", "agent is disabled", "different agent pool"):
+        assert reason in body, f"the refusal must be able to explain '{reason}'"
+
+
+@pytest.mark.parametrize("field,label", [
+    ("prefix_length", "prefix length"),
+    ("use_unicast", "unicast/multicast mode"),
+    ("track_haproxy", "HAProxy tracking"),
+])
+def test_adopt_requires_agreement_on_shared_vip_fields(field, label):
+    """These live on the VIP row and are re-rendered onto EVERY member, so taking them from the
+    node that happened to be clicked imposes its settings on the others. prefix_length is the
+    sharpest: the design refuses to guess a netmask for a live VIP, and copying one node's
+    netmask onto another is that same change by another name."""
+    body = _adopt_body()
+    assert f'("{field}", "{label}")' in body, (
+        f"{field} is written to the VIP row from one node's report; adoption must refuse when the "
+        f"nodes disagree about it"
+    )
+
+
+def test_adopt_requires_one_shared_vrrp_password():
+    body = _adopt_body()
+    assert "decrypt_vrrp_secret(enc)" in body, (
+        "Fernet is non-deterministic, so the per-node tokens cannot be compared as ciphertext — "
+        "they must be decrypted and compared as plaintext"
+    )
+    assert "do not share one VRRP password" in body, (
+        "adoption stores ONE secret and renders it onto every member, so a mismatch must be "
+        "refused rather than silently normalised"
+    )
+    assert "cannot be decrypted" in body, (
+        "a token we cannot decrypt must be an error, not silently treated as equal to another"
+    )
+
+
+def test_adopt_requires_reported_ips_before_trusting_the_peer_check():
+    body = _adopt_body()
+    assert "have not reported an IP address yet" in body, (
+        "the unicast peer check compares against member IPs, so a member without a reported IP "
+        "must block the check rather than silently pass it"
+    )
+
+
+# ----------------------------------------------------------------------------
+# The takeover authorisation is genuinely one-shot
+# ----------------------------------------------------------------------------
+
+def test_takeover_authorisation_is_retired_once_the_node_acks_our_config():
+    """`takeover_expected_hash` is the permission to overwrite a keepalived.conf that does NOT
+    carry our ownership marker. It was written at adoption and never cleared, so the release
+    notes' "authorises exactly ONE takeover" was only true as "for exactly that file content,
+    indefinitely" — restoring the pre-adoption file would have been silently overwritten again
+    with no fresh human approval."""
+    agent_router = (BACKEND / "routers" / "agent.py").read_text()
+    assert "takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL" in agent_router, (
+        "the status ack must retire the takeover authorisation once the node confirms our config"
+    )
+    assert "ELSE takeover_expected_hash END" in agent_router, (
+        "a non-matching ack must LEAVE the authorisation in place — dropping it on a partial or "
+        "failed deploy would leave the VIP unable to converge"
+    )
+
+
+def test_validation_judges_the_output_not_the_exit_code():
+    """keepalived's config-test exit code cannot separate fatal from benign.
+
+    Measured on keepalived 2.2.8 rather than assumed:
+
+        clean config ..................... 0
+        auth_pass longer than 8 chars .... 5   "Truncating auth_pass to 8 characters"
+        missing '}' ...................... 5   "There are 1 missing '}'s"
+        unknown keyword .................. 5   "Unknown keyword '...'"
+        script without script_security ... 6   "SECURITY VIOLATION ..."
+
+    So 5 covers both a harmless truncation and a broken file. Treating any non-zero exit as
+    invalid rejected VALID configs: a VRRP password over 8 characters is enough, and keepalived
+    truncates it to 8 anyway, exactly as it does for the file the operator already runs. Found
+    on the first live adoption, where the node's OWN running config also exited non-zero.
+
+    The gate must therefore judge the output, and it must fail CLOSED: anything not on the
+    benign list still counts as fatal.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count("grep -v 'Truncating auth_pass to 8 characters'") == 2, (
+        "both daemon copies must drop the known-benign truncation warning before judging"
+    )
+    assert script.count('if [[ -n "$kp_fatal" ]]; then') == 2, (
+        "the decision must be made on what REMAINS after the benign lines are dropped"
+    )
+    # Fail-closed: the benign list is an allowlist, never a denylist of fatal messages. The
+    # measured table above is quoted in a comment inside the script, so assert on what is used
+    # as a grep PATTERN rather than on the text appearing anywhere.
+    patterns = re.findall(r"grep -v '([^']*)'", script)
+    assert set(patterns) == {"Truncating auth_pass to 8 characters", "^[[:space:]]*$"}, (
+        f"the validation filter greps for {sorted(set(patterns))}. It must drop only messages "
+        f"known to be benign; matching on fatal messages instead would let an unrecognised "
+        f"error through."
+    )
+
+
+def test_validation_fails_closed_when_keepalived_says_nothing():
+    """A non-zero exit with no output must stay fatal.
+
+    Filtering the output introduces a way to reach the accept path with an EMPTY filter result,
+    and some keepalived builds log to syslog rather than stderr — on such a host every config
+    would then be accepted regardless of what is wrong with it. Verified against a stub that
+    exits non-zero silently, and against one that emits only whitespace.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count('if [[ -z "${kp_out//[[:space:]]/}" ]]; then') == 2, (
+        "both daemon copies must treat a non-zero exit with no readable output as fatal"
+    )
+    assert script.count('kp_fatal="keepalived -t exited non-zero without output"') == 2
+
+
+def test_validation_failure_reports_what_keepalived_said():
+    """The fail-safe protected the node correctly on a live adoption but logged only
+    "config validation failed", with keepalived's own output sent to /dev/null. The operator
+    had no way to act on it without reproducing the check by hand on the node."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert 'keepalived -t -f "$tmp_conf" >/dev/null 2>&1' not in script, (
+        "keepalived's output must not be discarded — a fail-safe that cannot say why it fired "
+        "is only half a safety feature"
+    )
+    assert script.count('kp_out=$(keepalived -t -f "$tmp_conf" 2>&1)') == 2, (
+        "both daemon copies must capture the validation output"
+    )
+    # The text is interpolated into JSON by both `log` and _kp_report, so it must be sanitised.
+    assert script.count(r"""tr -d '"\\'""") == 2, (
+        "captured output must have quotes and backslashes stripped before it reaches the JSON "
+        "log line and the status report"
+    )
+    assert script.count('keepalived -t failed: ${kp_err}') == 2, (
+        "the reason must also travel to the server so the UI can show it"
+    )
+
+
+def test_converged_node_keeps_acknowledging():
+    """The idempotent path must still report, or a lost ack is never recovered.
+
+    The status report is the server's ONLY evidence that a member converged, and it used to be
+    sent solely on the write path. Once the rendered config was on disk the agent took the
+    idempotency early return every cycle and never spoke again, so a single lost report - a
+    backend restart, a 5xx, a network blip - left the VIP reading SYNCING forever with an empty
+    "Last ack" while the node was demonstrably running the right config. Seen in the field after
+    acks were dropped for an unrelated reason: the node was correct, the page was not, and
+    nothing would ever reconcile them.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count('_kp_report "enabled" "$vip_id" "$new_hash" "already converged"') == 2, (
+        "both daemon copies must re-assert the deploy state on the idempotent path; without it "
+        "the server can never recover a lost acknowledgement"
+    )
+    # The report has to come BEFORE the early return in both copies.
+    for m in re.finditer(r'if \[\[ -n "\$cur_hash" && "\$cur_hash" == "\$would_hash" \]\]; then(.*?)fi',
+                         script, re.S):
+        body = m.group(1)
+        assert body.index("_kp_report") < body.index("return 0"), (
+            "the acknowledgement must be sent before returning, or the early return skips it"
+        )
+
+
+def test_status_ack_statements_bind_each_placeholder_once():
+    """Every `$n` in the keepalived-status UPDATEs must be used exactly once, and the count must
+    match the arguments passed.
+
+    Reusing one placeholder for both the assignment (`last_deploy_hash=$n`, a VARCHAR column)
+    and the comparison inside the takeover-retirement CASE made PostgreSQL deduce two types for
+    it, and asyncpg rejected the whole statement with AmbiguousParameterError. The failure was
+    not partial: no ack was written at all, so every VIP sat at SYNCING forever and teardown acks
+    were lost too. Shipped in v1.10.12 and caught in the field.
+
+    The suite has no database, so this pins the shape that made it possible rather than the SQL
+    behaviour: one placeholder, one binding site.
+    """
+    src = (BACKEND / "routers" / "agent.py").read_text()
+    start = src.index("async def agent_keepalived_status")
+    seg = src[start:src.index('return {"status": "ok"}', start)]
+
+    retire = "".join(re.findall(r'"([^"]*)"',
+                                re.search(r"_retire_takeover = \((.*?)\)\n", seg, re.S).group(1)))
+    calls = re.findall(r'await conn\.execute\(f"""(.*?)""",\s*(.*?)\)\n', seg, re.S)
+    assert len(calls) == 2, f"expected the two ack UPDATEs, found {len(calls)}"
+
+    for sql, args in calls:
+        placeholder = re.search(r'_retire_takeover\.format\(p="(\$\d+)"\)', sql).group(1)
+        rendered = re.sub(r"\{_retire_takeover\.format\(p=\"\$\d+\"\)\}",
+                          retire.replace("{p}", placeholder), sql)
+        used = re.findall(r"\$(\d+)", rendered)
+        dupes = {n for n in used if used.count(n) > 1}
+        assert not dupes, (
+            f"placeholder(s) {sorted('$'+d for d in dupes)} are bound more than once. PostgreSQL "
+            f"deduces a type per USE, so a placeholder that is both assigned to a column and "
+            f"compared against one is ambiguous and the whole UPDATE is rejected."
+        )
+        n_args = len([a for a in args.split(",") if a.strip()])
+        assert max(int(n) for n in used) == n_args, (
+            f"the statement uses ${max(int(n) for n in used)} but {n_args} arguments are passed"
+        )
+
+
+def test_takeover_still_requires_the_pinned_hash_to_match_on_disk():
+    """The guard that stops an edit between adoption and Apply from being overwritten."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    guard = '[[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]'
+    assert script.count(guard) == 2, (
+        f"the takeover guard must be present in BOTH daemon copies (found {script.count(guard)}); "
+        f"a self-upgraded agent runs the in-script copy, a freshly installed one the heredoc"
+    )
+
+
+# ----------------------------------------------------------------------------
+# Backward compatibility
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize("guard", [
+    "version_name NOT LIKE 'vip-%'",     # bulk apply/reject still skip VIP versions
+])
+def test_vip_versions_stay_excluded_from_the_haproxy_apply_flow(guard):
+    assert guard in CLUSTER_ROUTER, (
+        "vip-* versions must stay out of the HAProxy apply/reject sweep; they are owned by the "
+        "VIP endpoints and are never served as haproxy.cfg"
+    )
+
+
+def test_vip_version_transition_matches_any_action():
+    """_transition_vip_versions must key on the VIP id alone, or a new action's PENDING row
+    would be stranded in Apply Management after apply/reject."""
+    assert 'f"vip-{vip_id}-%"' in VIP_ROUTER, (
+        "the PENDING -> APPLIED/REJECTED transition must match every action for the VIP"
+    )
+
+
+# ----------------------------------------------------------------------------
+# v1.11.1 — a discovery report that was never accepted must be retried
+# ----------------------------------------------------------------------------
+
+def test_discovery_is_cached_only_when_the_server_accepted_it():
+    """`curl` without -f exits 0 on 500/403/404, so the previous `if curl ...` recorded a
+    REJECTED discovery as delivered. The cache then suppressed every later attempt, and since
+    the file never changes on its own the node stayed out of the adoption panel permanently:
+    the only cure was deleting the cache on the node by hand."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert "if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST" not in script, (
+        "the discovery POST must not be judged by curl's exit code; it is 0 for 5xx as well"
+    )
+    assert script.count('if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then') == 2, (
+        "both daemon copies must cache only on a 2xx"
+    )
+
+
+def test_discovery_cache_defers_to_the_server():
+    """Recovery without touching the node. The server reports whether it actually holds a
+    discovery for this agent; only an explicit `false` overrides the cache, so a backend older
+    than v1.11.1 (which omits the field) keeps the previous behaviour instead of being flooded
+    with re-posts."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    agent_router = (BACKEND / "routers" / "agent.py").read_text()
+
+    assert agent_router.count('AS discovery_known') == 1, (
+        "the keepalived-config query must report whether a discovery row exists"
+    )
+    assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 5, (
+        "every response path that knows the agent must carry the flag — the adoptable nodes are "
+        "precisely the not_configured ones"
+    )
+    assert script.count('jq -r \'if has("discovery_known")') == 2
+    assert script.count('[[ "$disc_known" != "false" ]] && return 0') == 2, (
+        "only an explicit false may override the cache, or an older backend — which omits the "
+        "field — would cause a re-post on every cycle"
+    )
+    assert script.count('conf chk disc_known=""') == 2, (
+        "disc_known must be function-local; in the in-script daemon the enclosing scope is the "
+        "poll loop, so a stale value would outlive the response it came from"
+    )
+
+
+def test_keepalived_config_path_is_resolved_deterministically():
+    """A pool may hold more than one cluster, and the join multiplies the agent row. Without an
+    ordering the fetch took an arbitrary cluster, so the keepalived.conf PATH handed to the agent
+    was non-deterministic whenever two clusters in a pool disagreed on it: the agent would look at
+    the wrong file, find nothing, and the node would never appear for adoption."""
+    src = (BACKEND / "routers" / "agent.py").read_text()
+    start = src.index("async def get_agent_keepalived_config")
+    q_start = src.index('agent = await conn.fetchrow("""', start)
+    query = src[q_start:src.index('""", agent_name)', q_start)]
+    assert "LEFT JOIN haproxy_clusters" in query, "re-point this test; the join moved"
+    assert "ORDER BY" in query and "LIMIT 1" in query, (
+        "the cluster row must be picked deterministically, or the config path the agent is told "
+        "to inspect can change between polls"
+    )
+    assert "hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'" in query, (
+        "the ordering must prefer a CUSTOMISED path over the shipped default. The column defaults "
+        "to that path rather than NULL, so ordering by id alone could pick a default-valued row "
+        "over one the operator deliberately set, turning 'undefined' into 'reliably wrong'."
+    )
+
+
+def test_daemon_copies_agree_on_the_whole_keepalived_path():
+    """Everything the adoption flow depends on must behave identically on BOTH install routes:
+    a freshly installed agent runs the heredoc body, a self-upgraded one runs the in-script
+    daemon. Comments may differ; logic may not."""
+    import difflib
+    lines = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text().splitlines()
+    term = next(i for i, l in enumerate(lines) if l.strip() == "AGENT_SCRIPT")
+
+    def strip_comment(s):
+        out, q, esc = [], None, False
+        for ch in s:
+            if esc:
+                out.append(ch); esc = False; continue
+            if ch == "\\":
+                out.append(ch); esc = True; continue
+            if q:
+                out.append(ch)
+                if ch == q:
+                    q = None
+                continue
+            if ch in ('"', "'"):
+                q = ch; out.append(ch); continue
+            if ch == "#":
+                break
+            out.append(ch)
+        return "".join(out).rstrip()
+
+    def funcs(block):
+        found = {}
+        for idx, l in enumerate(block):
+            m = re.match(r"^(\s*)([a-zA-Z_][a-zA-Z0-9_]*)\(\)\s*\{\s*(#.*)?$", l)
+            if not m:
+                continue
+            close = m.group(1) + "}"
+            end = next((j for j in range(idx + 1, len(block)) if block[j].rstrip() == close), None)
+            if end is None:
+                continue
+            found[m.group(2)] = [re.sub(r"\s+", " ", strip_comment(x).strip())
+                                 for x in block[idx + 1:end] if strip_comment(x).strip()]
+        return found
+
+    here, insc = funcs(lines[923:term]), funcs(lines[term + 1:])
+    for name in ("_kp_discover", "_kp_report", "_kp_teardown",
+                 "fetch_and_deploy_keepalived_config", "get_keepalive_state"):
+        assert name in here and name in insc, f"{name} is missing from one daemon copy"
+        if here[name] != insc[name]:
+            d = "\n".join(x for x in difflib.unified_diff(here[name], insc[name], lineterm="")
+                          if x[:1] in "+-" and x[:3] not in ("+++", "---"))
+            raise AssertionError(
+                f"{name}() differs between the daemon copies, so a self-upgraded agent would "
+                f"behave differently from a freshly installed one:\n{d}"
+            )
+
+
+def test_discovery_flag_distinguishes_false_from_absent():
+    """jq's `//` returns the alternative for **false** as well as null.
+
+    `.discovery_known // empty` therefore yields an empty string both when the backend omits the
+    field (older release) and when it explicitly says `false` (no discovery on record) — the one
+    case the recovery exists for. Written that way the fix is inert: the cache is never overridden
+    and a stuck node stays hidden. Caught in review, before it shipped, by parsing a real response
+    rather than passing the value in by hand.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert "'.discovery_known // empty'" not in script, (
+        "jq's // treats false like null, so this cannot tell 'no record' from 'old backend'"
+    )
+    expected = ('disc_known=$(echo "$resp" | jq -r \'if has("discovery_known") '
+                'then (.discovery_known|tostring) else "" end\' 2>/dev/null)')
+    assert script.count(expected) == 2, (
+        "both daemon copies must distinguish an explicit false from an absent field"
+    )
+
+
+def test_discovery_backs_off_on_a_permanent_rejection():
+    """A 4xx means the payload itself is unacceptable, so re-posting the same bytes cannot help.
+
+    Retrying forever is not free here: 4xx and 5xx agent calls are never sampled out of the
+    request log (see request_log_sink), so an unattended loop writes a row carrying the whole
+    keepalived.conf every poll cycle, on every affected node — the exact "polling noise evicts
+    the forensic record" failure the log's own defaults exist to prevent. The cache therefore
+    records the rejection and stays quiet until the file changes; a 5xx or a transport failure
+    is still retried, which is what the recovery depends on.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    brake = ('elif [[ "$disc_code" == "400" || "$disc_code" == "413" '
+             '|| "$disc_code" == "422" ]]; then')
+    assert script.count(brake) == 2, (
+        "the brake must be limited to the codes that mean 'these bytes are unacceptable'"
+    )
+    # 401 and 404 are 4xx but TRANSIENT here: a token rotation, or an agent row briefly absent
+    # while it re-registers. Braking on them would silence discovery for every affected node
+    # until its keepalived.conf changed, which for a hand-maintained file may be never — the
+    # exact failure this release removes.
+    for transient in ('"401"', '"404"'):
+        assert transient not in brake, (
+            f"{transient} must stay in the retry class; it does not mean the payload is bad"
+        )
+    assert script.count("""printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null""") == 2
+    assert script.count('[[ "$cached_state" == "rejected" ]] && return 0') == 2, (
+        "a recorded rejection must suppress the post even when the server reports no record, "
+        "or the flag override turns into an unbounded retry loop"
+    )
+    # The rejection must be keyed to the CONTENT, so a fixed config is retried.
+    assert script.count('cached_hash="${cached_line%% *}"') == 2, (
+        "the rejection is stored against the hash; changing the file must clear the brake"
+    )
+
+
+def test_config_import_reaches_a_freshly_installed_agent():
+    """`check_config_requests` uploads the node's live haproxy.cfg when the operator asks for it.
+
+    It was defined in the installer body and in the in-script daemon, but NOT in the heredoc a
+    fresh install writes to /usr/local/bin/haproxy-agent. Its call site is guarded by
+    `type check_config_requests`, so on a freshly installed agent the whole feature was a silent
+    no-op: the operator requested a config from the node and nothing ever arrived, with no error.
+    Agents that had self-upgraded at least once did have it, which is why it went unnoticed.
+    """
+    lines = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text().splitlines()
+    term = next(i for i, l in enumerate(lines) if l.strip() == "AGENT_SCRIPT")
+    pattern = re.compile(r"^\s*check_config_requests\(\)\s*\{")
+
+    in_heredoc = sum(1 for l in lines[923:term] if pattern.match(l))
+    in_daemon = sum(1 for l in lines[term + 1:] if pattern.match(l))
+    assert in_heredoc == 1, (
+        "the heredoc a fresh install writes must define check_config_requests, or Config Import "
+        "silently does nothing on any node that has never self-upgraded"
+    )
+    assert in_daemon == 1, "the self-upgrade daemon must keep its definition"
+
+    # And the two must be the same function, not two drifting implementations.
+    def body(block):
+        idx = next(i for i, l in enumerate(block) if pattern.match(l))
+        out = []
+        for l in block[idx:]:
+            out.append(l.strip())
+            if l.strip() == "}" and len(out) > 5:
+                break
+        return out
+
+    assert body(lines[923:term]) == body(lines[term + 1:]), (
+        "the two copies of check_config_requests have drifted"
+    )

backend/tests/test_vip_discoveries_scope.py

@@ -0,0 +1,87 @@
+"""
+v1.10.6 — GET /api/vip/discoveries resolves to its own handler and honours cluster_id.
+
+Two defects, one endpoint, found in that order on a live fleet:
+
+1. The route was declared after `GET /{vip_id}`, so FastAPI matched it there and answered
+   422 ("discoveries" is not an int) before the handler ran. The static declaration-order
+   guard lives in test_router_path_shadowing.py; this file pins the observable behaviour,
+   because a 422 is what the browser actually saw.
+
+2. Once reachable, it returned every discovery in the fleet regardless of the cluster
+   selected in the header, so a multi-cluster install saw one undifferentiated list. The
+   endpoint now takes the same optional `cluster_id` the VIP list takes.
+
+The auth tests here deliberately assert `!= 422`: the repo's generic endpoint-auth tests
+accept 401/403/422 together, which is precisely why defect 1 slipped through them.
+"""
+import pathlib
+import re
+
+import pytest
+
+VIP_ROUTER = pathlib.Path(__file__).resolve().parents[1] / "routers" / "vip.py"
+
+
+# ----------------------------------------------------------------------------
+# 1. The route reaches its own handler (defect 1)
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/vip/discoveries",
+    "/api/vip/discoveries?cluster_id=7",
+])
+def test_discoveries_route_is_not_captured_by_the_vip_id_route(client, path):
+    res = client.get(path)
+    assert res.status_code != 422, (
+        f"GET {path} returned 422 — the request was routed into the get-one-VIP handler, "
+        f"which parses the path segment as an int. Declaration order regressed. "
+        f"Body: {res.text[:200]}"
+    )
+    assert res.status_code in (401, 403), (
+        f"GET {path} without a token should be refused by the vip.read gate, got "
+        f"{res.status_code}. Body: {res.text[:200]}"
+    )
+
+
+def test_get_one_vip_still_parses_a_numeric_id(client):
+    """Moving /discoveries above /{vip_id} must not shadow the numeric route itself."""
+    res = client.get("/api/vip/12")
+    assert res.status_code in (401, 403), res.text[:200]
+
+
+# ----------------------------------------------------------------------------
+# 2. cluster_id is accepted and actually scopes the query (defect 2)
+# ----------------------------------------------------------------------------
+
+def test_handler_accepts_cluster_id():
+    from routers.vip import list_vip_discoveries
+    import inspect
+
+    params = inspect.signature(list_vip_discoveries).parameters
+    assert "cluster_id" in params, (
+        "list_vip_discoveries no longer takes cluster_id; the HA/VIP page would show every "
+        "cluster's nodes at once again"
+    )
+    assert params["cluster_id"].default is None, (
+        "cluster_id must stay optional — omitting it returns the whole fleet, which is what "
+        "a caller that predates the parameter expects"
+    )
+
+
+def test_discovery_query_scopes_by_the_cluster_pool():
+    """The filter must resolve cluster -> pool the same way the VIP list does, and must be a
+    no-op when the parameter is absent."""
+    source = VIP_ROUTER.read_text()
+    start = source.index("async def list_vip_discoveries")
+    end = source.index("def _find_candidate", start)
+    body = source[start:end]
+
+    assert "FROM vip_discoveries" in body, "the discovery query moved; re-point this test"
+    assert re.search(r"a\.pool_id\s*=\s*\(\s*SELECT\s+pool_id\s+FROM\s+haproxy_clusters", body), (
+        "the cluster filter must map cluster -> pool via haproxy_clusters, matching list_vips"
+    )
+    assert "IS NULL" in body, (
+        "the filter must short-circuit when cluster_id is absent, so an unscoped call still "
+        "returns the whole fleet"
+    )

backend/utils/acme_backend_url.py

@@ -0,0 +1,266 @@
+"""Validation and resolution for the ACME HTTP-01 challenge backend URL.
+
+This URL tells HAProxy where to proxy ``/.well-known/acme-challenge/*``. It is
+rendered into ``backend _acme_challenge_backend`` as ``server _acme_mgmt host:port``
+and — this is the part that makes it unlike every other URL in the product —
+**resolved on the HAProxy node, not on the management host**. A value that works
+when pasted into the management server's own browser can be completely dead from
+the data plane.
+
+Two entry points, deliberately asymmetric:
+
+``validate_acme_backend_url``
+    Called at the WRITE BOUNDARY (cluster PUT, settings PUT). Rejects values that
+    cannot express a reachable target. Strict here is safe: it only ever affects a
+    value an operator is typing right now, and the error text can teach.
+
+``resolve_acme_backend_target``
+    Called at RENDER TIME. Never raises, never rejects. Strictness here would be a
+    catastrophe: the shipped defaults (``config.py`` ``http://localhost:8000``,
+    ``docker-compose.yml`` ``http://localhost:8080``) mean essentially every
+    existing install resolves to loopback today, and refusing to render would make
+    every ``acme_enabled`` cluster unappliable — including for urgent changes that
+    have nothing to do with ACME. It reports problems instead of enforcing them.
+
+Two conscious departures from ``utils/ssrf_guard.py``, whose policy is the exact
+opposite of what is needed here:
+
+* **RFC1918 is allowed, and is usually the correct answer.** The guard exists to
+  stop the server being tricked into dialling internal space. Here the operator is
+  deliberately naming their own management host, which on a split deployment is
+  private by definition.
+* **No DNS resolution.** Resolving from the management host would re-introduce the
+  very wrong-vantage-point mistake this work exists to remove: what this box can
+  resolve says nothing about what the HAProxy node can reach.
+"""
+import ipaddress
+import re
+from typing import List, NamedTuple, Optional
+from urllib.parse import urlparse
+
+# `haproxy_clusters.acme_backend_url` / `system_settings.value` are VARCHAR(500).
+# Without this check asyncpg raises 22001 and the operator gets an opaque 500.
+MAX_URL_LENGTH = 500
+
+ALLOWED_SCHEMES = ("http", "https")
+
+# Port assumed when the URL omits one. NOT the scheme's default: the bundled
+# docker-compose publishes nginx on 8080 (`nginx/nginx.conf` listens 8080,
+# `docker-compose.yml` maps 8080:8080), so an operator who wrote a bare
+# `http://10.0.0.5` has a WORKING path today that resolves to :8080. Changing this
+# to 80 would break those installs silently — the first symptom would be the
+# unattended renewal loop failing months later. The value is kept and the omission
+# is surfaced as a warning instead.
+DEFAULT_HTTP_PORT = 8080
+DEFAULT_HTTPS_PORT = 443
+
+# RFC 1123 host label set. Deliberately not a full IDN implementation: an operator
+# naming their management host in a config file pushed to HAProxy nodes should use
+# ASCII, and HAProxy itself would not accept anything else on a `server` line.
+_HOSTNAME_RE = re.compile(
+    r"^(?=.{1,253}$)[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?"
+    r"(\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*\.?$"
+)
+
+_CONTROL_CHARS = frozenset("\t\n\r\v\f\x00")
+
+
+class AcmeBackendUrlError(ValueError):
+    """A value that cannot express a usable challenge backend target.
+
+    ``code`` is stable and machine-readable so the UI can map it to help text;
+    ``args[0]`` is operator-facing prose.
+    """
+
+    def __init__(self, code: str, message: str):
+        super().__init__(message)
+        self.code = code
+
+
+class AcmeBackendTarget(NamedTuple):
+    """What the renderer should emit, plus everything worth telling the operator."""
+
+    host: str
+    port: int
+    ssl_flag: str
+    #: Non-fatal observations. Rendered anyway; surfaced in logs and the panel.
+    warnings: List[str]
+    #: Set when the value could not be parsed at all and the caller must not emit
+    #: a `server` line. None on success.
+    error_code: Optional[str]
+    error_message: Optional[str]
+
+
+def _classify_host(host: str) -> Optional[str]:
+    """Return a rejection code for hosts that cannot be a management address."""
+    try:
+        ip = ipaddress.ip_address(host)
+    except ValueError:
+        lowered = host.rstrip(".").lower()
+        # `localhost` is loopback by name and is the single most likely wrong value
+        # here — it is what both shipped defaults contain. Catching only the numeric
+        # form would let the exact failure this module exists to prevent straight
+        # through. RFC 6761 also reserves the whole `.localhost` tree.
+        if lowered == "localhost" or lowered.endswith(".localhost"):
+            return "loopback"
+        return None if _HOSTNAME_RE.match(host) else "invalid_host"
+
+    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    if ip.is_loopback:
+        return "loopback"
+    if ip.is_unspecified:
+        return "unspecified"
+    # Includes 169.254.169.254, the cloud metadata endpoint.
+    if ip.is_link_local:
+        return "link_local"
+    if ip.is_multicast:
+        return "multicast"
+    # NOTE: private (RFC1918) addresses fall through on purpose — see module docstring.
+    return None
+
+
+_REJECTION_PROSE = {
+    "too_long": f"URL must be at most {MAX_URL_LENGTH} characters.",
+    "whitespace": (
+        "URL must not contain spaces or line breaks. A trailing space survives parsing "
+        "and would be written into haproxy.cfg as part of the address."
+    ),
+    "no_scheme": (
+        "URL must start with http:// or https://. Without a scheme the value cannot be "
+        "parsed as an address and silently falls back to localhost, which on a HAProxy "
+        "node means the node itself."
+    ),
+    "bad_scheme": "URL scheme must be http or https.",
+    "userinfo": "URL must not contain credentials.",
+    "has_path": (
+        "Enter only the scheme, host and port — no path, query or fragment. The "
+        "challenge path is appended by HAProxy."
+    ),
+    "bad_port": "Port must be a number between 1 and 65535.",
+    "no_host": "URL must contain a host.",
+    "invalid_host": "Host is not a valid IP address or hostname.",
+    "loopback": (
+        "Loopback addresses cannot work here. HAProxy resolves this address on the "
+        "HAProxy node, so 127.0.0.1 means the node itself, not the management server. "
+        "Use the management server's routable address."
+    ),
+    "unspecified": (
+        "0.0.0.0 is a listen address, not a destination. Use the management server's "
+        "routable address."
+    ),
+    "link_local": "Link-local addresses cannot be used as a management address.",
+    "multicast": "Multicast addresses cannot be used as a management address.",
+}
+
+
+def validate_acme_backend_url(value: Optional[str]) -> Optional[str]:
+    """Validate an operator-supplied URL at the write boundary.
+
+    Returns the normalised value (stripped), or ``None`` for empty input, which
+    legitimately means "inherit from the next level of the resolution chain".
+    Raises :class:`AcmeBackendUrlError` otherwise.
+    """
+    if value is None:
+        return None
+    if not isinstance(value, str):
+        raise AcmeBackendUrlError("invalid_host", _REJECTION_PROSE["invalid_host"])
+
+    stripped = value.strip()
+    if not stripped:
+        return None
+
+    if len(stripped) > MAX_URL_LENGTH:
+        raise AcmeBackendUrlError("too_long", _REJECTION_PROSE["too_long"])
+    if any(c in _CONTROL_CHARS for c in stripped) or " " in stripped:
+        raise AcmeBackendUrlError("whitespace", _REJECTION_PROSE["whitespace"])
+
+    parsed = urlparse(stripped)
+
+    if not parsed.scheme:
+        raise AcmeBackendUrlError("no_scheme", _REJECTION_PROSE["no_scheme"])
+    if parsed.scheme.lower() not in ALLOWED_SCHEMES:
+        # `10.0.0.5:8080` parses as scheme='10.0.0.5' with no netloc, and
+        # `localhost:8080` as scheme='localhost'. Both are the same operator mistake,
+        # so point at the missing scheme rather than the nonsense one.
+        if not parsed.netloc:
+            raise AcmeBackendUrlError("no_scheme", _REJECTION_PROSE["no_scheme"])
+        raise AcmeBackendUrlError("bad_scheme", _REJECTION_PROSE["bad_scheme"])
+
+    if parsed.username is not None or parsed.password is not None:
+        raise AcmeBackendUrlError("userinfo", _REJECTION_PROSE["userinfo"])
+    if parsed.path not in ("", "/") or parsed.query or parsed.fragment:
+        raise AcmeBackendUrlError("has_path", _REJECTION_PROSE["has_path"])
+
+    try:
+        port = parsed.port
+    except ValueError:
+        # urlparse defers port parsing to attribute access; an out-of-range or
+        # non-numeric port raises here. Unguarded, this exception reaches the config
+        # generator's blanket `except` and collapses the cluster's whole config.
+        raise AcmeBackendUrlError("bad_port", _REJECTION_PROSE["bad_port"]) from None
+    if port is not None and not (1 <= port <= 65535):
+        raise AcmeBackendUrlError("bad_port", _REJECTION_PROSE["bad_port"])
+
+    host = parsed.hostname
+    if not host:
+        raise AcmeBackendUrlError("no_host", _REJECTION_PROSE["no_host"])
+
+    code = _classify_host(host)
+    if code is not None:
+        raise AcmeBackendUrlError(code, _REJECTION_PROSE[code])
+
+    return stripped
+
+
+def resolve_acme_backend_target(url: Optional[str]) -> AcmeBackendTarget:
+    """Resolve a stored URL into what the renderer emits. Never raises.
+
+    Values already in the database predate validation (and the shipped defaults are
+    themselves loopback), so anything unparseable or discouraged is reported through
+    ``warnings`` / ``error_code`` rather than refused.
+    """
+    warnings: List[str] = []
+    raw = (url or "").strip()
+
+    if not raw:
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "empty", "No challenge backend URL configured."
+        )
+
+    if any(c in _CONTROL_CHARS for c in raw) or " " in raw:
+        # Must never reach haproxy.cfg: a newline here writes attacker- or
+        # accident-chosen directives into a file pushed to every node.
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "whitespace", _REJECTION_PROSE["whitespace"]
+        )
+
+    parsed = urlparse(raw)
+    scheme = (parsed.scheme or "").lower()
+
+    try:
+        port = parsed.port
+    except ValueError:
+        return AcmeBackendTarget("", 0, "", warnings, "bad_port", _REJECTION_PROSE["bad_port"])
+
+    host = parsed.hostname
+    if not host or scheme not in ALLOWED_SCHEMES:
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "no_scheme", _REJECTION_PROSE["no_scheme"]
+        )
+
+    if port is None:
+        port = DEFAULT_HTTPS_PORT if scheme == "https" else DEFAULT_HTTP_PORT
+        warnings.append(
+            f"No port given, assuming {port}. State the port explicitly — the assumed "
+            f"value is the bundled reverse proxy's port, not the scheme's default."
+        )
+
+    code = _classify_host(host)
+    if code == "invalid_host":
+        return AcmeBackendTarget("", 0, "", warnings, code, _REJECTION_PROSE[code])
+    if code is not None:
+        warnings.append(_REJECTION_PROSE[code])
+
+    ssl_flag = " ssl verify none" if scheme == "https" else ""
+    return AcmeBackendTarget(host, port, ssl_flag, warnings, None, None)

backend/utils/agent_scripts/linux_install.sh

@@ -470,7 +470,12 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "haproxy-agent.service"; do
+# issue #31: match ONLY the installed agent (binary path + service), never the bare string
+# "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command line
+# or a sudo/PAM ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself ("Killed", install aborts). The systemd service is also stopped below; the
+# "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "haproxy-agent.service"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -1055,7 +1060,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1357,7 +1367,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -1676,7 +1691,7 @@ check_ssl_updates() {
 # (kept in sync with the live token in both daemon loops).
 fetch_and_deploy_keepalived_config() {
     local marker="# Managed by HAProxy OpenManager"
-    local resp status http_code we_own="false" conf chk
+    local resp status http_code we_own="false" conf chk disc_known=""
 
     # Timeouts so a hung management server can never stall the daemon loop.
     resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
@@ -1695,6 +1710,95 @@ fetch_and_deploy_keepalived_config() {
     [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
     chk="$(dirname "$conf")/check_haproxy.sh"
     if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+    # Whether the SERVER already holds a discovery for this node. Absent on backends older than
+    # v1.11.1, in which case the discovery cache keeps its previous meaning.
+    disc_known=$(echo "$resp" | jq -r 'if has("discovery_known") then (.discovery_known|tostring) else "" end' 2>/dev/null)
+
+    # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
+    # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
+    # to the node. The heartbeat cannot carry this — it has the VIP address and a best-effort
+    # MASTER/BACKUP, while rendering a node's config needs eleven fields.
+    #
+    # Rate limited by content: the hash of the last report is cached next to the config, so the
+    # file (which may contain the VRRP password) is posted only when it actually changes, not on
+    # every cycle. Once we own the file there is nothing to adopt, so the record is cleared once.
+    _kp_discover() {
+        local cache="$(dirname "$conf")/.hom_discovery_hash" cur_hash="" body content_json
+        if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
+            # Nothing adoptable here. Clear a previous report exactly once.
+            [[ -f "$cache" ]] || return 0
+            # Clear ONLY when the server accepted it. curl's exit code is 0 for 5xx too, so
+            # dropping the cache on a rejected clear lost the fact that this node is ours: the
+            # stale discovery row would keep offering a MANAGED node for adoption, and with the
+            # cache gone nothing would ever send the clear again.
+            local clr_code
+            clr_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"config_path\":\"$conf\",\"exists\":false}" 2>/dev/null)
+            [[ "$clr_code" =~ ^2[0-9][0-9]$ ]] || return 0
+            rm -f "$cache"
+            return 0
+        fi
+        cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        [[ -z "$cur_hash" ]] && return 0
+        # The cache may only suppress a report while the SERVER agrees it already holds one.
+        # v1.11.1: it suppressed unconditionally, so a report the server REJECTED was recorded
+        # as delivered and the node stayed out of the adoption panel for good — the file never
+        # changes, so nothing ever triggered another attempt and the only cure was deleting this
+        # file on the node by hand. `discovery_known` is absent on older backends, and only an
+        # explicit "false" overrides the cache, so an old server keeps the previous behaviour
+        # rather than being flooded with re-posts.
+        # Cache line is "<hash>" after a success, or "<hash> rejected" after a 4xx.
+        local cached_line cached_hash cached_state=""
+        cached_line=$(cat "$cache" 2>/dev/null)
+        cached_hash="${cached_line%% *}"
+        [[ "$cached_line" == *" "* ]] && cached_state="${cached_line#* }"
+        if [[ -n "$cached_hash" && "$cached_hash" == "$cur_hash" ]]; then
+            # A 4xx means the server refuses THESE BYTES. Re-posting them cannot succeed, and
+            # every attempt is stored as a failed agent call (4xx/5xx are never sampled out), so
+            # an unattended loop would write a row carrying the whole config every cycle. Stay
+            # quiet until the file changes — the reason was logged when it was rejected.
+            [[ "$cached_state" == "rejected" ]] && return 0
+            # Otherwise the cache only holds while the SERVER agrees it has the report.
+            [[ "$disc_known" != "false" ]] && return 0
+        fi
+        # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
+        # server hashes is byte-identical to what is on disk — the takeover authorisation is
+        # pinned to that hash.
+        content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
+        body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
+            '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
+        # Cache ONLY on a 2xx. curl without -f exits 0 on 500/403/404 too, so the previous
+        # `if curl ...` recorded a REJECTED report as delivered — and since the cache suppresses
+        # every later attempt until the file itself changes, one server-side error hid the node
+        # from the adoption panel permanently. Same failure shape as the deploy acknowledgement
+        # fixed in v1.10.14, on the discovery path.
+        local disc_code
+        disc_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+            -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            --data-binary "$body" 2>/dev/null)
+        if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then
+            printf '%s' "$cur_hash" > "$cache" 2>/dev/null
+            log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+        elif [[ "$disc_code" == "400" || "$disc_code" == "413" || "$disc_code" == "422" ]]; then
+            # ONLY the codes that mean "these bytes are unacceptable" stop the retry: too large
+            # to analyse, malformed, rejected by validation. Re-posting identical content cannot
+            # change any of those answers.
+            #
+            # 401 and 404 are deliberately NOT here even though they are 4xx. Both are transient
+            # in this system — a token rotation, or an agent row briefly absent while it
+            # re-registers — and treating them as permanent would silence discovery for every
+            # affected node until its keepalived.conf changed, which for a hand-maintained file
+            # may be never. That is the exact failure this release set out to remove.
+            printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null
+            log "WARN" "KEEPALIVED: discovery report refused (HTTP $disc_code); not retrying until the config changes"
+        else
+            log "WARN" "KEEPALIVED: discovery report failed (HTTP ${disc_code:-none}); will retry next cycle"
+        fi
+    }
+    _kp_discover
 
     _kp_report() {  # $1=state  $2=vip_id(or empty)  $3=hash  $4=message
         local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
@@ -1759,10 +1863,29 @@ fetch_and_deploy_keepalived_config() {
     [[ -z "$new_conf" ]] && return 0
 
     # Ownership guard: never overwrite a keepalived.conf we don't own.
+    #
+    # v1.10.4 adoption is the ONE exception, and it does not weaken the guard: the server
+    # authorises a single takeover of a specific file by pinning the md5 the operator adopted
+    # from. We overwrite only when that hash still matches what is on disk, so a config edited
+    # between adoption and Apply is still refused — the operator's later edit wins over a stale
+    # adoption rather than being silently destroyed.
     if [[ -f "$conf" && "$we_own" != "true" ]]; then
-        log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
-        _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
-        return 0
+        local allow_takeover expected_hash disk_hash
+        allow_takeover=$(echo "$resp" | jq -r '.keepalived.allow_takeover // false' 2>/dev/null)
+        expected_hash=$(echo "$resp" | jq -r '.keepalived.takeover_expected_hash // empty' 2>/dev/null)
+        disk_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        if [[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]; then
+            log "INFO" "KEEPALIVED: adopting $conf (one-shot takeover authorised; on-disk hash matches)"
+        elif [[ "$allow_takeover" == "true" ]]; then
+            log "WARN" "KEEPALIVED: adoption authorised but $conf changed since it was adopted — refusing"
+            _kp_report "externally_managed" "$vip_id" "$disk_hash" \
+                "config changed after adoption; re-adopt to pick up the current file"
+            return 0
+        else
+            log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+            _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+            return 0
+        fi
     fi
 
     # Hybrid install: install keepalived only if missing.
@@ -1792,6 +1915,13 @@ fetch_and_deploy_keepalived_config() {
         cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
         would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
         if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            # STILL ACK. This report is the server's only evidence that the node converged, and
+            # it used to be sent on the write path alone — so a single lost ack (a backend
+            # restart, a 5xx, a network blip) left the VIP reading SYNCING forever: the node was
+            # already correct on disk, took this early return every cycle, and never spoke again.
+            # Re-asserting the state makes the loop self-healing, costs one request per ~2.5
+            # minutes, and touches nothing on the node.
+            _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
             return 0
         fi
     fi
@@ -1807,11 +1937,43 @@ fetch_and_deploy_keepalived_config() {
     local tmp_conf="${conf}.hom.tmp"
     printf '%s' "$new_conf" > "$tmp_conf"
     chmod 0644 "$tmp_conf"
-    if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
-        rm -f "$tmp_conf"
-        log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
-        _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
-        return 0
+    # Capture what keepalived actually said. Discarding it made the fail-safe useless in
+    # practice: the node was correctly protected, but neither the log nor the UI could say WHY,
+    # so the only way forward was to reproduce the check by hand on the node. Sanitised hard
+    # (quotes, backslashes and newlines removed, tail kept) because both `log` and _kp_report
+    # embed the text in JSON built by string interpolation.
+    local kp_out kp_err kp_fatal
+    if ! kp_out=$(keepalived -t -f "$tmp_conf" 2>&1); then
+        # keepalived's config-test EXIT CODE does not separate a fatal config error from a
+        # harmless warning. Measured on 2.2.8, not assumed:
+        #     clean config .................. 0
+        #     auth_pass longer than 8 chars . 5   "Truncating auth_pass to 8 characters"
+        #     missing '}' ................... 5   "There are 1 missing '}'s"
+        #     unknown keyword ............... 5   "Unknown keyword '...'"
+        #     script without script_security  6   "SECURITY VIOLATION ..."
+        # So 5 covers both a benign truncation and a broken file, and treating any non-zero
+        # exit as invalid rejected VALID configs: a VRRP password over 8 characters is enough,
+        # and keepalived truncates it to 8 regardless, exactly as it does for the file the
+        # operator already runs. Judge on the OUTPUT instead, dropping only messages known to
+        # be benign; anything unrecognised is still fatal, so this fails CLOSED.
+        if [[ -z "${kp_out//[[:space:]]/}" ]]; then
+            # Non-zero with NOTHING to read. We cannot confirm the reason is benign, and some
+            # builds log to syslog rather than stderr, so proceeding here would silently accept
+            # every config on such a host. Treat as fatal — the gate must fail closed.
+            kp_fatal="keepalived -t exited non-zero without output"
+        else
+            kp_fatal=$(printf '%s\n' "$kp_out" \
+                | grep -v 'Truncating auth_pass to 8 characters' \
+                | grep -v '^[[:space:]]*$' || true)
+        fi
+        if [[ -n "$kp_fatal" ]]; then
+            rm -f "$tmp_conf"
+            kp_err=$(printf '%s' "$kp_fatal" | tr '\n\r\t' '   ' | tr -d '"\\' | tail -c 300)
+            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t): ${kp_err} — keeping current config, not (re)starting"
+            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed: ${kp_err}"
+            return 0
+        fi
+        log "WARN" "KEEPALIVED: keepalived -t exited non-zero with only known-benign warnings; proceeding"
     fi
     mv -f "$tmp_conf" "$conf"
     chown root:root "$conf" 2>/dev/null
@@ -2081,6 +2243,95 @@ UPGRADE_EOF
 }
 
 # Check and apply configuration updates (Linux)
+# Config Import: upload this node's live haproxy.cfg when the operator asks for it.
+#
+# This function existed ONLY in the installer body and in the in-script daemon, never in the
+# heredoc that a FRESH install writes to /usr/local/bin/haproxy-agent. The call site below is
+# guarded by `type check_config_requests`, so on a freshly installed agent the whole feature
+# was a silent no-op: the operator requested a config from the node and nothing ever arrived,
+# with no error anywhere. Agents that had self-upgraded at least once did have it, which is why
+# it went unnoticed. Copied verbatim from the in-script daemon so both install routes behave
+# identically (v1.11.1).
+check_config_requests() {
+    local curl_bin="${CURL_BIN:-$(find_binary curl)}"
+    
+    # Get pending config requests from backend
+    log "DEBUG" "CONFIG: Checking for pending config requests from: ${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/pending-requests"
+    
+    local response=$("$curl_bin" -k -s -X GET "${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/pending-requests" \
+        -H "Content-Type: application/json" \
+        -H "X-API-Key: ${AGENT_TOKEN}")
+    
+    log "DEBUG" "CONFIG: Response: ${response:0:200}..."
+    
+    # Check if there are pending requests
+    local pending_count=$(echo "$response" | jq -r '.pending_requests | length' 2>/dev/null)
+    
+    log "DEBUG" "CONFIG: Pending count: $pending_count"
+    
+    if [[ "$pending_count" -gt 0 ]]; then
+        log "INFO" "CONFIG: Found $pending_count pending config request(s)"
+        
+        # Process each request
+        echo "$response" | jq -c '.pending_requests[]' 2>/dev/null | while read -r request; do
+            local request_id=$(echo "$request" | jq -r '.request_id')
+            local request_type=$(echo "$request" | jq -r '.request_type')
+            
+            log "INFO" "CONFIG: Processing config request #$request_id (type: $request_type)"
+            
+            # Read haproxy.cfg content
+            local config_path="${HAPROXY_CONFIG_PATH}"
+            if [[ -f "$config_path" ]]; then
+                local file_size=$(wc -c < "$config_path")
+                log "INFO" "CONFIG: Reading config from: $config_path (size: $file_size bytes)"
+                
+                local config_content=$(cat "$config_path")
+                
+                # Escape config content for JSON
+                log "DEBUG" "CONFIG: Escaping config content for JSON..."
+                local escaped_content=$(echo "$config_content" | jq -Rs .)
+                
+                if [[ -z "$escaped_content" ]]; then
+                    log "ERROR" "CONFIG: JSON escaping failed for config content!"
+                    continue
+                fi
+                
+                local escaped_size=${#escaped_content}
+                log "DEBUG" "CONFIG: JSON escaped content size: $escaped_size bytes"
+                
+                # Submit config response
+                local response_payload=$(cat <<CONFIG_RESPONSE_EOF
+{
+"request_id": $request_id,
+"config_content": $escaped_content,
+"config_path": "$config_path"
+}
+CONFIG_RESPONSE_EOF
+)
+                
+                local payload_size=${#response_payload}
+                log "INFO" "CONFIG: Sending config response (payload size: $payload_size bytes)..."
+                
+                local http_response=$("$curl_bin" -k -s -w "\n%{http_code}" -X POST "${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/config-response" \
+                    -H "Content-Type: application/json" \
+                    -H "X-API-Key: ${AGENT_TOKEN}" \
+                    -d "$response_payload")
+                
+                local http_code=$(echo "$http_response" | tail -n1)
+                local response_body=$(echo "$http_response" | head -n-1)
+                
+                if [[ "$http_code" == "200" || "$http_code" == "201" ]]; then
+                    log "INFO" "CONFIG: Config response sent for request #$request_id (size: $file_size bytes, HTTP $http_code)"
+                else
+                    log "ERROR" "CONFIG: Config response failed for request #$request_id (HTTP $http_code): $response_body"
+                fi
+            else
+                log "ERROR" "CONFIG: Config file not found: $config_path"
+            fi
+        done
+    fi
+}
+
 check_config_updates() {
     # Only log config check in debug mode to reduce log spam
     [[ "${DEBUG_MODE:-0}" == "1" ]] && log "INFO" "Checking for configuration updates..."
@@ -3227,7 +3478,7 @@ CONFIG_RESPONSE_EOF
     # the live token at the top of each loop iteration below.
     fetch_and_deploy_keepalived_config() {
         local marker="# Managed by HAProxy OpenManager"
-        local resp status http_code we_own="false" conf chk
+        local resp status http_code we_own="false" conf chk disc_known=""
 
         # Timeouts so a hung management server can never stall the daemon loop.
         resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
@@ -3246,6 +3497,84 @@ CONFIG_RESPONSE_EOF
         [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
         chk="$(dirname "$conf")/check_haproxy.sh"
         if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+        # See the heredoc copy.
+        disc_known=$(echo "$resp" | jq -r 'if has("discovery_known") then (.discovery_known|tostring) else "" end' 2>/dev/null)
+
+        # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
+        # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
+        # to the node. The heartbeat cannot carry this — it has the VIP address and a best-effort
+        # MASTER/BACKUP, while rendering a node's config needs eleven fields.
+        #
+        # Rate limited by content: the hash of the last report is cached next to the config, so the
+        # file (which may contain the VRRP password) is posted only when it actually changes, not on
+        # every cycle. Once we own the file there is nothing to adopt, so the record is cleared once.
+        _kp_discover() {
+            local cache="$(dirname "$conf")/.hom_discovery_hash" cur_hash="" body content_json
+            if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
+                # Nothing adoptable here. Clear a previous report exactly once.
+                [[ -f "$cache" ]] || return 0
+                # See the heredoc copy: clear only on a 2xx, or a managed node keeps being
+                # offered for adoption and nothing ever retries the clear.
+                local clr_code
+                clr_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                    -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                    -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                    -d "{\"config_path\":\"$conf\",\"exists\":false}" 2>/dev/null)
+                [[ "$clr_code" =~ ^2[0-9][0-9]$ ]] || return 0
+                rm -f "$cache"
+                return 0
+            fi
+            cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            [[ -z "$cur_hash" ]] && return 0
+            # See the heredoc copy: the cache may only suppress while the server agrees it holds
+            # a discovery for this node, or a rejected report hides the node permanently.
+            # Cache line is "<hash>" after a success, or "<hash> rejected" after a 4xx.
+            local cached_line cached_hash cached_state=""
+            cached_line=$(cat "$cache" 2>/dev/null)
+            cached_hash="${cached_line%% *}"
+            [[ "$cached_line" == *" "* ]] && cached_state="${cached_line#* }"
+            if [[ -n "$cached_hash" && "$cached_hash" == "$cur_hash" ]]; then
+                # A 4xx means the server refuses THESE BYTES. Re-posting them cannot succeed, and
+                # every attempt is stored as a failed agent call (4xx/5xx are never sampled out), so
+                # an unattended loop would write a row carrying the whole config every cycle. Stay
+                # quiet until the file changes — the reason was logged when it was rejected.
+                [[ "$cached_state" == "rejected" ]] && return 0
+                # Otherwise the cache only holds while the SERVER agrees it has the report.
+                [[ "$disc_known" != "false" ]] && return 0
+            fi
+            # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
+            # server hashes is byte-identical to what is on disk — the takeover authorisation is
+            # pinned to that hash.
+            content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
+            body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
+                '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
+            # See the heredoc copy: cache ONLY on a 2xx, or a rejected report is recorded as
+            # delivered and the node never reappears in the adoption panel.
+            local disc_code
+            disc_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                --data-binary "$body" 2>/dev/null)
+            if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then
+                printf '%s' "$cur_hash" > "$cache" 2>/dev/null
+                log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+            elif [[ "$disc_code" == "400" || "$disc_code" == "413" || "$disc_code" == "422" ]]; then
+                # ONLY the codes that mean "these bytes are unacceptable" stop the retry: too large
+                # to analyse, malformed, rejected by validation. Re-posting identical content cannot
+                # change any of those answers.
+                #
+                # 401 and 404 are deliberately NOT here even though they are 4xx. Both are transient
+                # in this system — a token rotation, or an agent row briefly absent while it
+                # re-registers — and treating them as permanent would silence discovery for every
+                # affected node until its keepalived.conf changed, which for a hand-maintained file
+                # may be never. That is the exact failure this release set out to remove.
+                printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null
+                log "WARN" "KEEPALIVED: discovery report refused (HTTP $disc_code); not retrying until the config changes"
+            else
+                log "WARN" "KEEPALIVED: discovery report failed (HTTP ${disc_code:-none}); will retry next cycle"
+            fi
+        }
+        _kp_discover
 
         _kp_report() {
             local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
@@ -3306,11 +3635,26 @@ CONFIG_RESPONSE_EOF
         [[ -z "$new_conf" ]] && return 0
 
         if [[ -f "$conf" && "$we_own" != "true" ]]; then
-            log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
-            _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
-            return 0
+            local allow_takeover expected_hash disk_hash
+            allow_takeover=$(echo "$resp" | jq -r '.keepalived.allow_takeover // false' 2>/dev/null)
+            expected_hash=$(echo "$resp" | jq -r '.keepalived.takeover_expected_hash // empty' 2>/dev/null)
+            disk_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            if [[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]; then
+                log "INFO" "KEEPALIVED: adopting $conf (one-shot takeover authorised; on-disk hash matches)"
+            elif [[ "$allow_takeover" == "true" ]]; then
+                log "WARN" "KEEPALIVED: adoption authorised but $conf changed since it was adopted — refusing"
+                _kp_report "externally_managed" "$vip_id" "$disk_hash" \
+                    "config changed after adoption; re-adopt to pick up the current file"
+                return 0
+            else
+                log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+                _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+                return 0
+            fi
         fi
 
+        # Hybrid install: install keepalived only if missing.
+
         if ! command -v keepalived >/dev/null 2>&1; then
             if [[ "$install_if" == "true" ]]; then
                 log "INFO" "KEEPALIVED: installing package..."
@@ -3336,6 +3680,9 @@ CONFIG_RESPONSE_EOF
             cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
             would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
             if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                # See the heredoc copy: the ack must be re-asserted here or a single lost report
+                # leaves the VIP reading SYNCING forever.
+                _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
                 return 0
             fi
         fi
@@ -3351,11 +3698,26 @@ CONFIG_RESPONSE_EOF
         local tmp_conf="${conf}.hom.tmp"
         printf '%s' "$new_conf" > "$tmp_conf"
         chmod 0644 "$tmp_conf"
-        if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
-            rm -f "$tmp_conf"
-            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
-            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
-            return 0
+        # See the heredoc copy for why the output is captured rather than discarded.
+        # See the heredoc copy for the measured exit-code table and why the OUTPUT, not the
+        # exit code, decides. Fails closed on anything not known to be benign.
+        local kp_out kp_err kp_fatal
+        if ! kp_out=$(keepalived -t -f "$tmp_conf" 2>&1); then
+            if [[ -z "${kp_out//[[:space:]]/}" ]]; then
+                kp_fatal="keepalived -t exited non-zero without output"
+            else
+                kp_fatal=$(printf '%s\n' "$kp_out" \
+                    | grep -v 'Truncating auth_pass to 8 characters' \
+                    | grep -v '^[[:space:]]*$' || true)
+            fi
+            if [[ -n "$kp_fatal" ]]; then
+                rm -f "$tmp_conf"
+                kp_err=$(printf '%s' "$kp_fatal" | tr '\n\r\t' '   ' | tr -d '"\\' | tail -c 300)
+                log "ERROR" "KEEPALIVED: config validation failed (keepalived -t): ${kp_err} — keeping current config, not (re)starting"
+                _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed: ${kp_err}"
+                return 0
+            fi
+            log "WARN" "KEEPALIVED: keepalived -t exited non-zero with only known-benign warnings; proceeding"
         fi
         mv -f "$tmp_conf" "$conf"
         chown root:root "$conf" 2>/dev/null

backend/utils/agent_scripts/macos_install.sh

@@ -318,7 +318,11 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "com.haproxy.agent"; do
+# issue #31: match ONLY the installed agent (binary path + LaunchDaemon label), never the bare
+# string "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command
+# line or a sudo ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself. The "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "com.haproxy.agent"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -920,7 +924,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1191,7 +1200,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then

backend/utils/csr_key_crypto.py

@@ -0,0 +1,120 @@
+"""Issue #53 — at-rest encryption for the pending CSR private key (v1.10.1).
+
+Mirrors the established Fernet + HKDF(SECRET_KEY) pattern already used for the VRRP secret
+(services/keepalived_config.py), TOTP secrets (services/mfa_service.py) and DNS provider
+credentials (utils/dns_credentials.py): prefer an explicit CSR_ENCRYPTION_KEY env var (enables
+key rotation), else derive a stable key from SECRET_KEY via HKDF with its own versioned info
+string, so a rotation of one secret class never affects another.
+
+WHY this key and not every key in the system: the CSR private key is the one key that sits IDLE.
+It is generated at CSR creation, waits for an external CA to sign the request (days to weeks),
+and is destroyed the moment the signed certificate is imported — it is never transmitted to an
+agent and never leaves the server. `ssl_certificates.private_key_content` and the ACME order keys
+are different: agents must receive them in plaintext on every poll, so encrypting them at rest
+buys nothing without an end-to-end redesign.
+
+STORAGE: the Fernet token replaces the PEM in the SAME `ssl_csrs.private_key_pem` TEXT column.
+No new column, no new table, and deliberately NO `SCHEMA_VERSION` bump — a bump would re-run the
+migration sequence and re-seed the four built-in roles to their defaults (see UPGRADE_GUIDE.md),
+which is a needless side effect for a storage-format change.
+
+BACKWARD COMPATIBILITY: rows written before this release hold a raw PEM. `decrypt_csr_private_key`
+detects those by their `-----BEGIN` header and returns them unchanged. The discriminator is exact,
+not a heuristic: a Fernet token is base64url text and can never contain "-----". Legacy rows drain
+naturally, since a CSR's key copy is NULLed on import.
+
+KEY ROTATION: if SECRET_KEY rotates while CSR_ENCRYPTION_KEY is unset, previously stored keys
+become undecryptable and `decrypt_csr_private_key` returns None. Callers MUST surface a clear
+"delete this CSR and create a new one" error — the CSR is unusable at that point, because the
+signed certificate can no longer be paired with its key.
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+from typing import Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# A PEM private key always carries this header; a Fernet token is base64url and never can.
+_PEM_MARKER = "-----BEGIN"
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit CSR_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF with a
+    versioned info string (so stored keys survive restarts)."""
+    explicit = os.getenv("CSR_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("CSR_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    logger.warning(
+        "CSR_ENCRYPTION_KEY not set; deriving the CSR private-key encryption key from SECRET_KEY. "
+        "Set CSR_ENCRYPTION_KEY to a Fernet key to enable key rotation."
+    )
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"csr-private-key-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def is_encrypted(stored: Optional[str]) -> bool:
+    """True when the stored value is a Fernet token rather than a legacy raw PEM.
+
+    Single source of the format discriminator: `decrypt_csr_private_key` branches on this, so
+    the "what does a stored value look like" rule is stated exactly once.
+    """
+    return bool(stored) and _PEM_MARKER not in stored
+
+
+def encrypt_csr_private_key(pem: str) -> str:
+    """Fernet-encrypt a PEM private key to a storable token string."""
+    return _get_fernet().encrypt(pem.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_csr_private_key(stored: Optional[str]) -> Optional[str]:
+    """Return the PEM private key for a stored value.
+
+    Accepts BOTH shapes so an upgrade needs no data migration:
+      - a raw PEM written before v1.10.1 -> returned unchanged
+      - a Fernet token -> decrypted
+
+    Returns None when the value is empty or cannot be decrypted (e.g. SECRET_KEY rotated without
+    CSR_ENCRYPTION_KEY). Callers MUST treat None as "this CSR's key is unrecoverable" and tell the
+    operator to delete it and create a new one; never fall through to a pairing attempt.
+    """
+    if not stored:
+        return None
+    if not is_encrypted(stored):
+        return stored  # legacy plaintext row, pre-v1.10.1
+    try:
+        return _get_fernet().decrypt(stored.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt a stored CSR private key (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting a stored CSR private key: %s", exc)
+        return None

backend/utils/entity_snapshot.py

@@ -329,9 +329,10 @@ async def _rollback_update(
                     tcp_request_rules = $26, timeout_client = $27, timeout_http_request = $28,
                     rate_limit = $29, compression = $30, log_separate = $31,
                     monitor_uri = $32, maxconn = $33,
-                    cluster_id = $34, is_active = $35, last_config_status = $36, 
+                    cluster_id = $34, is_active = $35, last_config_status = $36,
+                    log_format = $37, filters = $38,
                     updated_at = CURRENT_TIMESTAMP
-                WHERE id = $37
+                WHERE id = $39
             """,
                 old_values.get('name'),
                 old_values.get('bind_address'),
@@ -369,6 +370,8 @@ async def _rollback_update(
                 old_values.get('cluster_id'),
                 old_values.get('is_active'),
                 old_values.get('last_config_status'),
+                old_values.get('log_format'),  # Issue #38
+                old_values.get('filters'),     # Issue #38
                 entity_id
             )
             

backend/utils/haproxy_config_parser.py

@@ -105,6 +105,11 @@ class ParsedFrontend:
     response_headers: Optional[str] = None
     options: Optional[str] = None  # HAProxy frontend options (option httplog, option forwardfor, etc.)
     tcp_request_rules: Optional[str] = None  # TCP request directives (for TCP mode)
+    # Issue #38: SPOE (and other) filter directives + frontend log-format.
+    # Stored as full directive lines; `filters` is newline-joined to preserve
+    # ordering when multiple `filter ...` lines exist.
+    log_format: Optional[str] = None  # `log-format` / `log-format-sd` line(s)
+    filters: Optional[str] = None  # `filter ...` line(s), e.g. `filter spoe engine coraza config ...`
 
 
 @dataclass
@@ -256,8 +261,23 @@ class HAProxyConfigParser:
             acl_rules_list = []
             use_backend_rules_list = []
             tcp_request_rules_list = []
+            filters_list = []
+            log_format_list = []
 
             for line in lines:
+                # Issue #38: capture `filter ...` (SPOE/Coraza etc.) and
+                # `log-format`/`log-format-sd` directives. Pre-fix these matched
+                # no branch below and were silently dropped, so an imported SPOE
+                # config lost `filter spoe engine coraza ...` (→ HAProxy fatal
+                # "unable to find SPOE engine") and the frontend log-format.
+                # `continue` isolates them from the header/option handling below.
+                if line.startswith('filter '):
+                    filters_list.append(line.strip())
+                    continue
+                if re.match(r'^log-format(-sd)?\s', line, re.IGNORECASE):
+                    log_format_list.append(line.strip())
+                    continue
+
                 # Parse bind directive
                 # IMPORTANT: Handle multiple bind lines correctly
                 # Example: bind *:1002 (HTTP) and bind *:443 ssl (HTTPS)
@@ -540,6 +560,13 @@ class HAProxyConfigParser:
             if tcp_request_rules_list:
                 frontend.tcp_request_rules = '\n'.join(tcp_request_rules_list)
 
+            # Issue #38: assign captured SPOE filters + log-format
+            if filters_list:
+                frontend.filters = '\n'.join(filters_list)
+
+            if log_format_list:
+                frontend.log_format = '\n'.join(log_format_list)
+
             self.frontends.append(frontend)
             logger.info(f"Parsed frontend: {name} -> {frontend.default_backend}")
 
@@ -666,9 +693,14 @@ class HAProxyConfigParser:
                             'transparent', 'abortonclose', 'allbackups', 'checkcache', 'clitcpka',
                             'srvtcpka', 'http-no-delay', 'socket-stats', 'tcp-smart-accept',
                             'tcp-smart-connect', 'independant-streams', 'log-separate-errors',
-                            'log-health-checks', 'accept-invalid-http-request', 'accept-invalid-http-response'
+                            'log-health-checks', 'accept-invalid-http-request', 'accept-invalid-http-response',
+                            # Issue #38: SPOP health check for SPOE agent backends
+                            # (e.g. coraza-spoa). Already collected below regardless, but
+                            # listing it suppresses the spurious "unknown option" warning
+                            # for the exact SPOE use-case.
+                            'spop-check'
                         ]
-                        
+
                         if option_name not in valid_options:
                             # Unknown/invalid option - add warning but still collect it
                             self.warnings.append(

backend/utils/http_instrumentation.py

@@ -0,0 +1,328 @@
+"""v1.11.0 — outbound half of the unified request/response log.
+
+This is deliberately NOT a session or connector factory. Three incompatible
+connector policies coexist in this codebase:
+
+  * `utils.ssrf_guard.safe_connector()` — IPv4-pinned, TLS verification on;
+    returns a NEW connector per call because `ClientSession` closes the one it
+    owns, so a shared long-lived connector would raise "Connector is closed".
+  * `services/acme_diagnostics.py` — IPv4-pinned with `ssl=False` for the
+    plain-HTTP port-80 probe.
+  * the DNS providers and the CA-chain import — the default dual-stack
+    connector.
+
+On top of that, `backend/tests/test_acme_diagnostics.py` monkeypatches
+`aiohttp.ClientSession` globally with fakes that implement only
+`__aenter__/__aexit__/head(...)`. Centralising session construction would break
+all of it. So this module wraps the CALL, never the session.
+
+Two hard rules, both load-bearing:
+
+1.  `outbound_span` NEVER raises. Both DNS provider funnels end in
+    `except Exception: raise DnsProviderError("Unexpected ... failure")`, and in
+    GoDaddy's publish path that reverts `dns_record_published` and stalls the
+    ACME order — an instrumentation bug must not masquerade as a provider
+    outage.
+2.  `outbound_span` NEVER swallows. An exception raised inside the block is
+    recorded (status_class 0) and re-raised unchanged.
+"""
+import asyncio
+import logging
+import time
+import uuid
+from contextlib import asynccontextmanager
+from typing import Any, Dict, Optional
+
+from utils.request_log_redaction import safe_error_text, scrub_query_string, scrub_url
+from utils.request_log_settings import get_config
+from utils.request_log_sink import RequestLogRow, request_id_context, request_log_sink
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Stable identifiers for the `request_logs.target` column — this is the
+# "kime gitti" (who did we call) axis of the log.
+TARGET_ACME = "acme"
+TARGET_ACME_DIAG = "acme_diag"
+TARGET_LETSENCRYPT_CA = "letsencrypt_ca"
+TARGET_DNS_CLOUDFLARE = "dns_cloudflare"
+TARGET_DNS_GODADDY = "dns_godaddy"
+TARGET_AGENT = "agent"
+TARGET_HAPROXY_STATS = "haproxy_stats"
+TARGET_SETTINGS_PROBE = "settings_probe"
+
+
+def begin_background_trace(label: str) -> str:
+    """Open a fresh correlation id for ONE iteration of a background loop.
+
+    Without this, background outbound rows fell back to `bg:<asyncio task
+    name>`. Nothing in main.py passes `name=` to `create_task`, so a loop is
+    `Task-5` for its entire life and EVERY call it ever makes carries the same
+    `request_id` — measured: fifteen ACME calls across five renewal ticks came
+    out as one id. `GET /api/request-logs/{id}` then answers with up to 100 rows
+    under `related`, presented as "the calls this request made", which in a
+    forensics tool is worse than having no trace: an operator reading a failed
+    renewal is shown a hundred unrelated calls spanning days. Task numbers are
+    also reused across restarts, so `bg:Task-5` can mean a different loop after
+    a redeploy.
+
+    Called at the top of each iteration; the loop task is dedicated, so the next
+    iteration simply overwrites it and there is nothing to reset.
+    """
+    trace_id = f"bg:{label}:{uuid.uuid4().hex[:12]}"[:64]
+    request_id_context.set(trace_id)
+    return trace_id
+
+
+def _correlation_id() -> str:
+    """Inherit the inbound request's id when there is one, so an API call and
+    the CA/DNS calls it triggered share a trace. Background work gets the id
+    opened by begin_background_trace() for the current iteration."""
+    existing = request_id_context.get()
+    if existing:
+        return existing
+    # No inbound request and no iteration trace: background code that has not
+    # been wrapped. Mint a unique id rather than falling back to the task name,
+    # which would silently re-collapse every such call into one row group.
+    try:
+        task = asyncio.current_task()
+        name = task.get_name() if task else "unknown"
+    except Exception:
+        name = "unknown"
+    return f"bg:{name}:{uuid.uuid4().hex[:12]}"[:64]
+
+
+class OutboundSpan:
+    """Handle passed to the `async with` body so the call site can attach the
+    response it just read."""
+
+    __slots__ = (
+        "target", "method", "url", "capture_request_body", "capture_response_body",
+        "safe_error_only",
+        "_status", "_response_headers", "_response_body", "_response_bytes",
+        "_response_content_type", "_request_body", "_request_headers", "_error",
+    )
+
+    def __init__(
+        self,
+        *,
+        target: str,
+        method: str,
+        url: str,
+        capture_request_body: bool,
+        capture_response_body: bool,
+        safe_error_only: bool,
+        request_body: Any = None,
+        request_headers: Optional[Dict[str, str]] = None,
+    ):
+        self.target = target
+        self.method = (method or "GET").upper()
+        self.url = url
+        # Two independent switches on purpose: the ACME JWS request body is a
+        # replayable credential and must never be stored, but the CA's RESPONSE
+        # (problem JSON, order state) is exactly what an operator needs to see.
+        self.capture_request_body = capture_request_body
+        self.capture_response_body = capture_response_body
+        self.safe_error_only = safe_error_only
+        self._request_body = request_body
+        self._request_headers = request_headers
+        self._status: Optional[int] = None
+        self._response_headers: Optional[Dict[str, str]] = None
+        self._response_body: Any = None
+        self._response_bytes: int = 0
+        self._response_content_type: Optional[str] = None
+        self._error: Optional[str] = None
+
+    def set_response(
+        self,
+        status: Optional[int],
+        headers: Optional[Dict[str, str]] = None,
+        body: Any = None,
+    ) -> None:
+        """Record what came back. Safe to call with a partially-read response;
+        never raises, so a call site can hand us whatever it happens to have."""
+        try:
+            self._status = int(status) if status is not None else None
+        except (TypeError, ValueError):
+            self._status = None
+        try:
+            if headers:
+                self._response_headers = {str(k).lower(): str(v) for k, v in dict(headers).items()}
+                self._response_content_type = self._response_headers.get("content-type")
+        except Exception:
+            self._response_headers = None
+
+        if body is None or not self.capture_response_body:
+            return
+        try:
+            if isinstance(body, (bytes, bytearray)):
+                self._response_bytes = len(body)
+                cap = get_config().max_body_bytes
+                self._response_body = bytes(body[:cap]) if cap else None
+            elif isinstance(body, str):
+                encoded = body.encode("utf-8", "replace")
+                self._response_bytes = len(encoded)
+                cap = get_config().max_body_bytes
+                self._response_body = encoded[:cap] if cap else None
+            else:
+                # Already-decoded JSON (the common case: `await resp.json()`).
+                self._response_body = body
+        except Exception:
+            self._response_body = None
+
+    def set_error(self, exc: BaseException, *, type_only: Optional[bool] = None) -> None:
+        try:
+            only = self.safe_error_only if type_only is None else type_only
+            self._error = safe_error_text(exc, type_only=only)
+        except Exception:
+            self._error = "UnknownError"
+
+    def to_row(self, duration_ms: int) -> RequestLogRow:
+        scrubbed = scrub_url(self.url)
+        path = None
+        query_params = None
+        try:
+            import urllib.parse
+
+            parts = urllib.parse.urlsplit(self.url)
+            path = parts.path or "/"
+            _, query_params = scrub_query_string(parts.query)
+        except Exception:
+            pass
+
+        row = RequestLogRow(
+            request_id=_correlation_id(),
+            direction="outbound",
+            target=self.target,
+            method=self.method,
+            url=scrubbed,
+            path=path,
+            query_params=query_params,
+            status_code=self._status,
+            duration_ms=duration_ms,
+            request_headers=self._request_headers,
+            response_headers=self._response_headers,
+            error=self._error,
+        )
+
+        if self._request_body is not None:
+            if not self.capture_request_body:
+                # The call site handed us a synthetic SUMMARY instead of the real
+                # payload (the ACME JWS case) — store the summary as-is.
+                row.request_body_value = _redacted_value(self._request_body)
+            elif isinstance(self._request_body, (bytes, bytearray)):
+                row.request_body_bytes = len(self._request_body)
+                cap = get_config().max_body_bytes
+                row.request_body_raw = bytes(self._request_body[:cap]) if cap else None
+            else:
+                row.request_body_value = _redacted_value(self._request_body)
+
+        if isinstance(self._response_body, (bytes, bytearray)):
+            row.response_body_raw = bytes(self._response_body)
+            row.response_body_bytes = self._response_bytes or len(self._response_body)
+            row.response_content_type = self._response_content_type
+        elif self._response_body is not None:
+            row.response_body_value = _redacted_value(self._response_body)
+
+        return row
+
+
+def _redacted_value(value: Any) -> Any:
+    from utils.request_log_redaction import redact
+
+    return redact(value)
+
+
+@asynccontextmanager
+async def outbound_span(
+    *,
+    target: str,
+    method: str,
+    url: str,
+    request_body: Any = None,
+    request_headers: Optional[Dict[str, str]] = None,
+    capture_body: bool = True,
+    capture_response_body: bool = True,
+    safe_error_only: bool = False,
+):
+    """Time an outbound HTTP call and record one `direction='outbound'` row.
+
+    `capture_body=False` applies to the REQUEST body only, for payloads that
+    are themselves credentials — the ACME JWS body is a replayable, signed
+    capability for the lifetime of its nonce, so the call site passes a
+    description of it instead. The CA's response is still captured, because
+    that is the half an operator actually needs when an order fails.
+
+    `safe_error_only=True` reduces a recorded exception to its type name, for
+    the DNS providers whose own error handling already refuses to surface
+    `str(exc)` (it can carry the request URL and, through it, zone identifiers).
+    """
+    span: Optional[OutboundSpan] = None
+    started = time.perf_counter()
+    try:
+        cfg = get_config()
+        if cfg.enabled and cfg.capture_outbound:
+            span = OutboundSpan(
+                target=target,
+                method=method,
+                url=url,
+                capture_request_body=capture_body and cfg.capture_bodies,
+                capture_response_body=capture_response_body and cfg.capture_bodies,
+                safe_error_only=safe_error_only,
+                request_body=request_body,
+                request_headers=request_headers,
+            )
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"outbound_span: could not start span for {target}: {exc}")
+        span = None
+
+    if span is None:
+        # Logging is off (or failed to initialise) — yield a throwaway span so
+        # the call site's `span.set_response(...)` still works.
+        span = OutboundSpan(
+            target=target, method=method, url=url,
+            capture_request_body=False, capture_response_body=False,
+            safe_error_only=safe_error_only,
+        )
+        try:
+            yield span
+        finally:
+            pass
+        return
+
+    try:
+        yield span
+    except BaseException as exc:
+        try:
+            span.set_error(exc)
+        except Exception:
+            pass
+        raise
+    finally:
+        try:
+            duration_ms = int((time.perf_counter() - started) * 1000)
+            request_log_sink.offer(span.to_row(duration_ms))
+        except Exception as exc:  # pragma: no cover - defensive
+            logger.debug(f"outbound_span: failed to record row for {target}: {exc}")
+
+
+async def instrumented_request(session, method: str, url: str, *, target: str,
+                               safe_error_only: bool = True, capture_body: bool = True,
+                               **kwargs):
+    """Convenience wrapper for the call sites that already funnel through
+    `session.request(...)` (the two DNS providers).
+
+    Returns `(status, headers, text)` and leaves error handling entirely to the
+    caller — this helper only adds the log row.
+    """
+    async with outbound_span(
+        target=target,
+        method=method,
+        url=url,
+        request_body=kwargs.get("json"),
+        capture_body=capture_body,
+        safe_error_only=safe_error_only,
+    ) as span:
+        async with session.request(method, url, **kwargs) as resp:
+            text = await resp.text()
+            span.set_response(resp.status, dict(resp.headers), text)
+            return resp.status, dict(resp.headers), text

backend/utils/request_log_prune.py

@@ -0,0 +1,201 @@
+"""v1.11.0 — retention prune for `request_logs`.
+
+Three independent limits, applied in order:
+
+  1. successful rows (`status_class` 1..3) older than `success_retention_days`
+  2. errored rows (`status_class` 0, 4, 5 — 0 meaning "no HTTP response at
+     all") older than `error_retention_days`
+  3. a hard row cap: anything below the `max_rows`-th newest id
+
+Splitting success from error is the point of the design: a busy install can
+keep a week of ordinary traffic while still holding three months of failures
+for forensics, without paying for both.
+
+Deliberately NOT folded into `utils/activity_log.prune_acme_events_and_drafts_if_due`:
+that function is driven by tests with fixed `execute.side_effect` lists and an
+exact return dict, and it is gated behind a `letsencrypt_orders`-exists check
+that would silently disable this prune on an ACME-free install.
+
+Three safety properties, all of which matter at scale:
+
+  * **Batched deletes.** The pool sets `command_timeout=60`; an unbounded
+    DELETE over a multi-million-row table raises `asyncpg.TimeoutError` and
+    then nothing is ever pruned.
+  * **Advisory lock.** `pg_try_advisory_lock` (try, never block) so N replicas
+    × M uvicorn workers do not all scan at once.
+  * **Watermark stamped only after a complete pass.** A pass that times out
+    mid-way is retried at the next tick instead of being recorded as done.
+"""
+import json
+import logging
+from datetime import datetime
+from typing import Dict, Optional
+
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_settings import get_config
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Fresh namespace. Already taken in this codebase: 18181818 (draft cap),
+# 18181819 (wizard create), 18181820 (apply), 0x41434D45 (per-ACME-order),
+# 1836016242 (migration lock).
+PRUNE_LOCK_KEY = 18181821
+
+WATERMARK_KEY = "requestlog.last_pruned_at"
+
+BATCH_SIZE = 5000
+MAX_BATCHES = 40  # ceiling of 200k rows removed per pass
+
+# Retention days ALWAYS travel as a bind parameter. They are operator-supplied,
+# so interpolating them into the SQL string would be an injection point.
+_SQL_TTL_SUCCESS = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs
+    WHERE status_class BETWEEN 1 AND 3
+      AND created_at < NOW() - ($1 || ' days')::INTERVAL
+    LIMIT $2
+)
+"""
+
+_SQL_TTL_ERROR = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs
+    WHERE (status_class = 0 OR status_class >= 4)
+      AND created_at < NOW() - ($1 || ' days')::INTERVAL
+    LIMIT $2
+)
+"""
+
+_SQL_CAP_CUTOFF = "SELECT id FROM request_logs ORDER BY id DESC OFFSET $1 LIMIT 1"
+
+_SQL_CAP_DELETE = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs WHERE id <= $1 LIMIT $2
+)
+"""
+
+
+def _deleted_count(result) -> int:
+    """asyncpg returns the command tag ('DELETE 42') from execute()."""
+    if isinstance(result, str) and result.startswith("DELETE "):
+        try:
+            return int(result.split()[-1])
+        except (ValueError, IndexError):
+            return 0
+    return 0
+
+
+async def _batched_delete(conn, sql: str, first_param) -> int:
+    """Run `sql` repeatedly until a short batch comes back or the ceiling hits."""
+    total = 0
+    for _ in range(MAX_BATCHES):
+        result = await conn.execute(sql, first_param, BATCH_SIZE)
+        count = _deleted_count(result)
+        total += count
+        if count < BATCH_SIZE:
+            break
+    else:
+        logger.info(
+            f"request_logs prune hit the {MAX_BATCHES}-batch ceiling "
+            f"({total} rows this pass); the remainder is removed on the next run"
+        )
+    return total
+
+
+async def _is_due(conn, key: str, min_interval_seconds: int) -> bool:
+    """Watermark gate. Unlike the hardcoded 24h in utils/activity_log.py the
+    interval here is operator-configurable."""
+    row = await conn.fetchrow("SELECT value FROM system_settings WHERE key = $1", key)
+    if not row or row["value"] is None:
+        return True
+    raw = row["value"]
+    if isinstance(raw, str):
+        try:
+            raw = json.loads(raw)
+        except (json.JSONDecodeError, ValueError):
+            return True
+    if not isinstance(raw, str):
+        return True
+    try:
+        last = datetime.fromisoformat(raw.replace("Z", "+00:00"))
+    except ValueError:
+        return True
+    age = (datetime.utcnow() - last.replace(tzinfo=None)).total_seconds()
+    return age >= min_interval_seconds
+
+
+async def _stamp(conn, key: str) -> None:
+    await conn.execute(
+        """
+        INSERT INTO system_settings (key, value, category, description)
+        VALUES ($1, $2::jsonb, 'requestlog', 'Internal: last request_logs prune timestamp')
+        ON CONFLICT (key) DO UPDATE
+        SET value = EXCLUDED.value, updated_at = CURRENT_TIMESTAMP
+        """,
+        key,
+        json.dumps(datetime.utcnow().isoformat() + "Z"),
+    )
+
+
+async def _prune_row_cap(conn, max_rows: int) -> int:
+    """Delete everything below the `max_rows`-th newest id."""
+    cutoff: Optional[int] = await conn.fetchval(_SQL_CAP_CUTOFF, max_rows)
+    if cutoff is None:
+        return 0  # fewer rows than the cap — nothing to do
+    return await _batched_delete(conn, _SQL_CAP_DELETE, cutoff)
+
+
+async def prune_request_logs_if_due(force: bool = False) -> Dict[str, int]:
+    """Run one retention pass if the watermark says it is due.
+
+    Never raises: a prune failure must not take down the loop that calls it.
+    `force=True` skips the watermark gate (used by the manual purge endpoint).
+    """
+    counts = {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+    cfg = get_config()
+
+    conn = None
+    locked = False
+    try:
+        conn = await get_database_connection()
+
+        # One replica only. try-lock: never block a pod waiting on another's pass.
+        locked = await conn.fetchval("SELECT pg_try_advisory_lock($1)", PRUNE_LOCK_KEY)
+        if not locked:
+            return counts
+
+        if not force and not await _is_due(conn, WATERMARK_KEY, cfg.prune_interval_minutes * 60):
+            return counts
+
+        counts["success"] = await _batched_delete(conn, _SQL_TTL_SUCCESS, str(cfg.success_retention_days))
+        counts["error"] = await _batched_delete(conn, _SQL_TTL_ERROR, str(cfg.error_retention_days))
+        counts["overflow"] = await _prune_row_cap(conn, cfg.max_rows)
+        counts["ran"] = 1
+
+        # Only after all three steps completed — a partial pass must be retried,
+        # not recorded as done.
+        await _stamp(conn, WATERMARK_KEY)
+
+        if counts["success"] or counts["error"] or counts["overflow"]:
+            logger.info(
+                f"request_logs prune: {counts['success']} successful, {counts['error']} errored, "
+                f"{counts['overflow']} over-cap row(s) removed"
+            )
+        return counts
+    except Exception as exc:
+        logger.warning(f"prune_request_logs_if_due: {exc}")
+        return counts
+    finally:
+        if conn is not None:
+            if locked:
+                try:
+                    await conn.execute("SELECT pg_advisory_unlock($1)", PRUNE_LOCK_KEY)
+                except Exception:
+                    pass
+            try:
+                await close_database_connection(conn)
+            except Exception:
+                pass

backend/utils/request_log_redaction.py

@@ -0,0 +1,517 @@
+"""v1.11.0 — redaction for the unified request/response log.
+
+Everything that lands in `request_logs.request_body` / `response_body` /
+`request_headers` / `response_headers` / `query_params` passes through here
+first. The rules, in order of how much they are trusted:
+
+1.  **Headers are an ALLOWLIST.** Anything not explicitly listed is dropped.
+    A small set of high-signal headers (`Authorization`, `Cookie`, …) is kept
+    as a presence marker with the value replaced, so an operator debugging a
+    401 can still see *that* a credential was sent.
+2.  **Body keys are matched by a normalized name** (lowercased, punctuation
+    stripped), against an exact set for short generic names that would
+    over-match as substrings (`key`, `payload`) and a contains set for the
+    compound ones (`cert_private_key`, `eab_hmac_key`, …).
+3.  **Values are shape-checked too.** A PEM private key or a JWT-shaped string
+    is redacted no matter what key it arrived under — this is the net that
+    catches a route echoing a secret under a renamed field.
+
+None of these functions raise: a redaction failure must never turn into a
+failed request or a failed provider call, so callers get a safe placeholder
+instead of an exception.
+"""
+import json
+import logging
+import re
+import urllib.parse
+from typing import Any, Dict, Optional, Tuple
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+REDACTED = "***REDACTED***"
+
+# Short, generic names. Matched EXACTLY after normalization, because as
+# substrings they would swallow innocent fields (`key_suffix`, `monkey`,
+# `payload_size`, `keyboard`, `nonce_count`).
+REDACT_EXACT = {
+    "password", "passwd", "pwd", "secret", "token", "key", "auth",
+    "authorization", "cookie", "signature", "protected", "payload",
+    "nonce", "credentials", "credential", "otp", "pin", "jwk", "csr",
+    # `auth_pass` is keepalived's VRRP password and it is the plaintext field
+    # name on `POST/PUT /api/vip` (routers/vip.py binds `payload.auth_pass`
+    # straight into encrypt_vrrp_secret). It normalizes to "authpass", which
+    # matches NOTHING above: "password" is not a substring of "authpass", and
+    # the bare "auth" entry is an EXACT match, not a prefix. Without this line
+    # the VRRP secret is written to request_logs in cleartext on every VIP
+    # create and edit.
+    "authpass",
+}
+
+# Compound names. Matched as SUBSTRINGS of the normalized key.
+#
+# `token` is in here on purpose, not just its compounds. In this domain EVERY
+# field whose name contains "token" is a credential — api_token (the Cloudflare
+# provider credential), agent_token, access_token, session_token — and the cost
+# of over-redacting a hypothetical innocent one is a blanked field, while the
+# cost of under-redacting is a live credential sitting in an audit table.
+REDACT_CONTAINS = {
+    "password", "passwordhash", "secret", "apisecret", "clientsecret",
+    "token", "accesstoken", "refreshtoken", "mfatoken", "resettoken",
+    "sessiontoken", "apitoken", "agenttoken", "csrftoken",
+    "apikey", "xapikey", "privatekey", "publicprivate", "jwkprivatekey",
+    "certprivatekey", "csrprivatekey", "keypem", "privkey",
+    "hmac", "eabhmackey", "eabkid",
+    "credentialsencrypted", "encryptedcredentials", "dnscredentials",
+    "authorization", "cookie", "setcookie", "keyauthorization",
+    "backupcode", "backupcodes", "totp", "totpcode", "totpsecret",
+    "replaynonce", "sessionid", "statspassword", "encryptionkey",
+    "bearer", "signature",
+}
+
+_NORMALIZE_RE = re.compile(r"[^a-z0-9]")
+
+# Value-shaped guards — these fire regardless of the key name.
+_PEM_RE = re.compile(r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----")
+_JWT_RE = re.compile(r"^[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}$")
+
+# ---------------------------------------------------------------------------
+# Secrets embedded INSIDE a value, not carried as their own field.
+# ---------------------------------------------------------------------------
+# Key-name and whole-value matching both miss the biggest source of secrets in
+# this system: a rendered config file handed around as one long string under an
+# innocent key.
+#
+#   GET  /api/agents/{n}/keepalived-config   -> keepalived.config_content
+#   POST /api/agents/{n}/keepalived-discovery -> config_content
+#
+# Both carry a full keepalived.conf whose `auth_pass <secret>` line is the VRRP
+# password in cleartext, and the delivery endpoint is polled every ~2.5 minutes
+# per member node. routers/vip.py states the rule this restores: "the secret
+# never leaves the server in cleartext ... only the at-rest Fernet token and the
+# agent-delivery endpoint ever see the real value" — and routers/agent.py's
+# discovery handler already masks the very same text with the very same pattern
+# before storing it in `vip_discoveries.raw_config_masked`. Capturing the
+# unmasked original into request_logs would sit that plaintext right next to the
+# masked copy the codebase went to three separate lengths to produce.
+#
+# ONE compiled alternation, applied in a single pass over each string value:
+# `re.sub` with no match costs one scan, whereas a pre-filter plus N patterns
+# costs a scan each. Mask the WHOLE remainder of the line (vip.py's reasoning),
+# so a secret containing whitespace cannot partially leak.
+#
+# The `uri` branch is the same class of miss in a different shape. Redacting a
+# key called `secret` does nothing when the SAME secret is also handed back
+# inside a URI under a key called `otpauth_uri`:
+#
+#   POST /api/mfa/enroll -> {"secret": "***REDACTED***",
+#                            "otpauth_uri": "otpauth://totp/X?secret=JBSWY3DP..."}
+#
+# routers/mfa.py's own activity log says "NEVER log the secret itself" and
+# records only `secret_len`. scrub_query_string already knows `secret` is a
+# credential; it was simply never pointed at query strings that arrive inside a
+# body value rather than on the request line. Any URI in any captured string is
+# now run through scrub_url, which also strips userinfo (`https://u:p@host`)
+# and drops the fragment.
+#
+# The `statsauth` / `userlist` branches cover the same shape for HAProxy. This
+# application never RENDERS a credential into a haproxy.cfg, so nothing we
+# generate is at risk - but the agent uploads the node's REAL on-disk file:
+#
+#   linux_install.sh: config_content=$(cat "$config_path")
+#                     -> POST /api/configuration/agents/{n}/config-response
+#
+# plus `POST /api/config/validate` and the bulk import, which take whatever the
+# operator pastes. A production haproxy.cfg routinely carries `stats auth
+# admin:<password>` and a `userlist` block, so the content is credential-bearing
+# even though our generator's output is not. Patterns are anchored to HAProxy
+# keyword syntax rather than the bare word "password", so ordinary prose in an
+# error message ("invalid password format") is left alone.
+#
+# `_EOL` instead of a plain `[^\r\n]` / `\S`: a captured body is only decoded
+# from JSON when it PARSES, and a config upload is routinely larger than the
+# capture cap, so the common case for exactly these payloads is the truncated
+# `{"_raw": ...}` fallback - where the line breaks are still the two-character
+# escape `\n`, not real newlines. Matching to "end of line" without knowing that
+# makes `auth_pass ...` swallow the entire rest of the string: no leak, but the
+# whole remainder of the config is masked and the row becomes useless. So every
+# value here stops at a real newline OR at a literal backslash-n.
+#
+# PERFORMANCE. These patterns run on the writer task, on every string value of
+# every captured body, so their cost is paid per row forever. Measured on an
+# 8 KB config body, 300 iterations:
+#
+#   one combined alternation, IGNORECASE, \b-anchored     308.2 us
+#   the same four patterns run separately (sum)           219.7 us
+#   text.lower() once + four substring pre-checks           5.3 us
+#
+# Two things make the naive version expensive, and neither is the matching:
+# `\b` and IGNORECASE both defeat the regex engine's literal-prefix scan, so
+# every alphanumeric position in 8 KB becomes a candidate start. A body with
+# none of these keywords - which is almost every body - paid the full 308 us to
+# find nothing.
+#
+# So: split the alternation, and gate each pattern behind a substring test on
+# one lowercased copy. A `str.lower()` and an `in` are C-level scans; the regex
+# only runs when its keyword is actually present, and then it runs on text that
+# genuinely contains it. Cost becomes O(total string bytes) rather than
+# O(bytes x patterns), and the common case is ~5 us instead of ~308 us.
+#
+# The pre-checks are on the LOWERCASED copy, so the patterns must stay
+# IGNORECASE: the marker may well have been `AUTH_PASS` in the original.
+_EOL = r"(?:(?!\\n)[^\r\n])"
+_NON_SPACE = r"(?:(?!\\n)\S)"
+
+_RE_AUTH_PASS = re.compile(r"(auth_pass[ \t]+)\S" + _EOL + r"*", re.IGNORECASE)
+# `stats auth <user>:<passwd>` - keep the user, mask the password half, so an
+# operator can still tell WHICH account a 401 was about.
+_RE_STATS_AUTH = re.compile(
+    r"(stats[ \t]+auth[ \t]+[^\s:]*:)" + _NON_SPACE + r"+", re.IGNORECASE
+)
+# `user <name> password <hash>` / `user <name> insecure-password <plain>`
+_RE_USERLIST = re.compile(
+    r"(user[ \t]+" + _NON_SPACE + r"+[ \t]+(?:insecure-)?password[ \t]+)" + _NON_SPACE + r"+",
+    re.IGNORECASE,
+)
+# No IGNORECASE and no `\b`: the character class already covers both cases, and
+# `://` gives the engine a literal to scan for.
+_RE_URI = re.compile(r"[a-zA-Z][a-zA-Z0-9+.\-]*://[^\s\"'<>\\]+")
+
+_EMBEDDED_MASK = "********"
+
+# Only strings long enough to hold `auth_pass ` plus a value are worth scanning.
+_EMBEDDED_MIN_LENGTH = 11
+
+def _keep_prefix(match: "re.Match") -> str:
+    """Keep group 1 (the directive, and for `stats auth` the account name too),
+    replace the value that follows it."""
+    return f"{match.group(1)}{_EMBEDDED_MASK}"
+
+
+def _mask_uri(match: "re.Match") -> str:
+    uri = match.group(0)
+    # A URI with neither a query nor userinfo cannot carry a credential in the
+    # place we scrub, and rebuilding it would only risk changing a value for no
+    # benefit.
+    if "?" not in uri and "@" not in uri:
+        return uri
+    try:
+        scrubbed = scrub_url(uri)
+    except Exception:  # pragma: no cover - scrub_url already swallows
+        return uri
+    # scrub_url reports its own failure as a placeholder string. Inside a larger
+    # text value that would corrupt the surrounding sentence, so keep the
+    # original: it parsed badly enough that urlsplit found no query to scrub.
+    if not scrubbed or scrubbed == "***URL_PARSE_ERROR***":
+        return uri
+    return scrubbed
+
+
+def scrub_embedded_secrets(text: str) -> str:
+    """Mask credentials that live inside a larger text value (config blobs).
+
+    Each pattern is gated behind a substring test on one lowercased copy: see
+    the block comment above for the measurements. Never raises - a scrub
+    failure must not turn into a failed log write, and returning the input
+    unchanged would be the wrong failure direction, so the whole value is
+    replaced instead.
+    """
+    if not text or len(text) < _EMBEDDED_MIN_LENGTH:
+        return text
+    try:
+        lowered = text.lower()
+        if "auth_pass" in lowered:
+            text = _RE_AUTH_PASS.sub(_keep_prefix, text)
+        # Not "stats auth": the directive may be separated by tabs or by more
+        # than one space, which the pattern allows and a substring test does not.
+        if "stats" in lowered and "auth" in lowered:
+            text = _RE_STATS_AUTH.sub(_keep_prefix, text)
+        if "password" in lowered:
+            text = _RE_USERLIST.sub(_keep_prefix, text)
+        if "://" in text:
+            text = _RE_URI.sub(_mask_uri, text)
+        return text
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_embedded_secrets() failed, blanking value: {exc}")
+        return REDACTED
+
+
+_MAX_DEPTH = 6
+_MAX_NODES = 2000
+_MAX_STRING = 4096
+_MAX_LIST_ITEMS = 200
+
+# Header handling. Allowlist wins; presence-only names are emitted with the
+# value replaced so the operator knows the header was there.
+HEADER_ALLOWLIST = {
+    "content-type", "content-length", "content-encoding", "accept",
+    "accept-encoding", "accept-language", "user-agent", "referer", "origin",
+    "host", "connection", "cache-control", "pragma", "date", "server",
+    "x-correlation-id", "x-request-id", "x-response-time",
+    "x-forwarded-for", "x-forwarded-proto", "x-forwarded-host", "x-real-ip",
+    "location", "retry-after", "ratelimit-reset", "ratelimit-remaining",
+    "link", "etag", "vary",
+}
+
+HEADER_PRESENCE_ONLY = {
+    "authorization", "cookie", "set-cookie", "x-api-key", "api-key",
+    "proxy-authorization", "replay-nonce", "www-authenticate",
+    "x-auth-token", "x-agent-token", "x-agent-api-key",
+}
+
+_MAX_HEADERS = 40
+
+
+class _NodeBudget:
+    """Shared mutable counter so a single body can't blow the CPU budget by
+    being wide as well as deep."""
+
+    __slots__ = ("remaining",)
+
+    def __init__(self, remaining: int = _MAX_NODES):
+        self.remaining = remaining
+
+    def spend(self) -> int:
+        self.remaining -= 1
+        return self.remaining
+
+
+def _normalize_key(key: Any) -> str:
+    try:
+        return _NORMALIZE_RE.sub("", str(key).lower())
+    except Exception:
+        return ""
+
+
+def is_secret_key(key: Any) -> bool:
+    """True when a dict key / query param name names a secret."""
+    norm = _normalize_key(key)
+    if not norm:
+        return False
+    if norm in REDACT_EXACT:
+        return True
+    return any(needle in norm for needle in REDACT_CONTAINS)
+
+
+def _is_secret_value(value: str) -> bool:
+    """Shape-based guard for secrets that arrive under an innocent key."""
+    if len(value) < 32:
+        # Neither a PEM block nor a JWT fits in less than this; skip the
+        # regex work on the overwhelmingly common short-string case.
+        return False
+    if _PEM_RE.search(value):
+        return True
+    return bool(_JWT_RE.match(value.strip()))
+
+
+def redact(value: Any, *, depth: int = 0, budget: Optional[_NodeBudget] = None) -> Any:
+    """Recursively redact a decoded body.
+
+    Depth- and node-capped so a hostile or merely pathological payload cannot
+    burn CPU on the writer task. Never raises.
+    """
+    if budget is None:
+        budget = _NodeBudget()
+
+    try:
+        if depth > _MAX_DEPTH:
+            return "***DEPTH_LIMIT***"
+
+        if isinstance(value, dict):
+            out: Dict[str, Any] = {}
+            for k, v in value.items():
+                if budget.spend() <= 0:
+                    out["_node_limit"] = True
+                    break
+                if is_secret_key(k):
+                    out[str(k)] = REDACTED
+                else:
+                    out[str(k)] = redact(v, depth=depth + 1, budget=budget)
+            return out
+
+        if isinstance(value, (list, tuple)):
+            out_list = []
+            for item in list(value)[:_MAX_LIST_ITEMS]:
+                if budget.spend() <= 0:
+                    out_list.append("_node_limit")
+                    break
+                out_list.append(redact(item, depth=depth + 1, budget=budget))
+            if len(value) > _MAX_LIST_ITEMS:
+                out_list.append(f"…[{len(value) - _MAX_LIST_ITEMS} more items]")
+            return out_list
+
+        if isinstance(value, str):
+            if _is_secret_value(value):
+                return REDACTED
+            # Scrub BEFORE truncating. Truncation is not a security control:
+            # `auth_pass` sits inside the first 400 bytes of a rendered
+            # keepalived.conf, well under _MAX_STRING, so relying on the cut to
+            # drop it would be relying on luck about where the secret happens
+            # to fall in the file.
+            value = scrub_embedded_secrets(value)
+            if len(value) > _MAX_STRING:
+                return value[:_MAX_STRING] + f"…[truncated {len(value) - _MAX_STRING} chars]"
+            return value
+
+        return value
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"redact() failed, substituting placeholder: {exc}")
+        return "***REDACTION_ERROR***"
+
+
+def redact_headers(headers: Optional[Dict[str, str]]) -> Optional[Dict[str, str]]:
+    """Allowlist-filter a header mapping.
+
+    Allowlisted headers keep their value, `HEADER_PRESENCE_ONLY` headers keep
+    only the fact they were present, everything else is dropped silently.
+    """
+    if not headers:
+        return None
+    try:
+        out: Dict[str, str] = {}
+        for raw_name, raw_value in headers.items():
+            name = str(raw_name).lower()
+            if name in HEADER_PRESENCE_ONLY:
+                out[name] = REDACTED
+            elif name in HEADER_ALLOWLIST:
+                value = str(raw_value)
+                out[name] = value[:1024]
+            if len(out) >= _MAX_HEADERS:
+                break
+        return out or None
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"redact_headers() failed: {exc}")
+        return None
+
+
+def scrub_query_string(query: Optional[str]) -> Tuple[str, Optional[Dict[str, str]]]:
+    """Return (scrubbed_query_string, scrubbed_dict) for a raw query string."""
+    if not query:
+        return "", None
+    try:
+        pairs = urllib.parse.parse_qsl(query, keep_blank_values=True)
+        scrubbed = [(k, REDACTED if is_secret_key(k) else v) for k, v in pairs]
+        return urllib.parse.urlencode(scrubbed), dict(scrubbed)
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_query_string() failed: {exc}")
+        return "", None
+
+
+def scrub_url(url: str) -> str:
+    """Strip userinfo and scrub the query string of an absolute URL.
+
+    `https://user:pass@api.example.com/v1?api_key=x`
+      → `https://api.example.com/v1?api_key=***REDACTED***`
+    """
+    if not url:
+        return ""
+    try:
+        parts = urllib.parse.urlsplit(url)
+        netloc = parts.hostname or ""
+        if parts.port:
+            netloc = f"{netloc}:{parts.port}"
+        query, _ = scrub_query_string(parts.query)
+        # Fragments are dropped: they never reach a server and can carry tokens.
+        return urllib.parse.urlunsplit((parts.scheme, netloc, parts.path, query, ""))
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_url() failed: {exc}")
+        return "***URL_PARSE_ERROR***"
+
+
+# Content types whose bodies are worth buffering. Anything else (octet-stream,
+# images, text/event-stream) is size-counted but never copied, which is what
+# keeps streaming and file responses safe.
+CAPTURABLE_CONTENT_TYPES = (
+    "application/json",
+    "application/problem+json",
+    "application/jose+json",
+    "application/x-www-form-urlencoded",
+    "text/plain",
+    "text/html",
+    "text/xml",
+    "application/xml",
+)
+
+
+def is_capturable_content_type(content_type: Optional[str]) -> bool:
+    if not content_type:
+        # No Content-Type on a body-bearing message is rare; assume JSON-ish
+        # rather than dropping the one field the operator wanted to see.
+        return True
+    ct = content_type.split(";")[0].strip().lower()
+    return any(ct.startswith(prefix) for prefix in CAPTURABLE_CONTENT_TYPES)
+
+
+def decode_body(
+    raw: Optional[bytes],
+    content_type: Optional[str],
+    total_bytes: int = 0,
+) -> Tuple[Optional[Any], bool]:
+    """Decode + redact a captured body fragment.
+
+    `raw` is what the middleware managed to buffer (already capped);
+    `total_bytes` is how large the body actually was on the wire. Returns
+    `(jsonb_value, truncated)`. Non-JSON payloads are wrapped as
+    `{"_raw": "..."}` so the column stays a uniform JSONB object that the
+    detail view and any future `->>` query can rely on.
+    """
+    if not raw:
+        return None, False
+
+    truncated = total_bytes > len(raw)
+    ct = (content_type or "").split(";")[0].strip().lower()
+
+    try:
+        text = raw.decode("utf-8", "replace")
+    except Exception:  # pragma: no cover - decode with 'replace' can't raise
+        return {"_raw": "***DECODE_ERROR***"}, truncated
+
+    value: Any
+    if ct in ("application/json", "application/problem+json", "application/jose+json") or (
+        not ct and text[:1] in ("{", "[")
+    ):
+        try:
+            value = redact(json.loads(text))
+        except Exception:
+            # A truncated JSON body will not parse — keep the raw prefix so the
+            # operator still sees what was sent.
+            value = {"_raw": redact(text)}
+    elif ct == "application/x-www-form-urlencoded":
+        try:
+            value = redact(dict(urllib.parse.parse_qsl(text, keep_blank_values=True)))
+        except Exception:
+            value = {"_raw": redact(text)}
+    else:
+        value = {"_raw": redact(text)}
+
+    if truncated:
+        if isinstance(value, dict):
+            value["_truncated"] = True
+            value["_original_bytes"] = total_bytes
+        else:
+            value = {
+                "_value": value,
+                "_truncated": True,
+                "_original_bytes": total_bytes,
+            }
+
+    return value, truncated
+
+
+def safe_error_text(exc: BaseException, *, type_only: bool = False, limit: int = 2000) -> str:
+    """Render an exception for the `error` column.
+
+    `type_only=True` is used for the DNS providers, whose own error paths
+    deliberately never surface `str(exc)` — it can carry the request URL and,
+    through it, tenant/zone identifiers (see services/dns_providers/*.py).
+    """
+    try:
+        name = type(exc).__name__
+        if type_only:
+            return name
+        text = f"{name}: {exc}"
+        redacted = redact(text)
+        if not isinstance(redacted, str):
+            return name
+        return redacted[:limit]
+    except Exception:  # pragma: no cover - defensive
+        return "UnknownError"

backend/utils/request_log_settings.py

@@ -0,0 +1,272 @@
+"""v1.11.0 — operator-tunable settings for the request/response log.
+
+The middleware runs on EVERY request, so the hot path must not touch the
+database. `get_config()` returns a module-global immutable snapshot with no
+`await`; `refresh_config()` reloads it from `system_settings` and is called
+
+  * once at startup, right after migrations,
+  * every `_TTL_SECONDS` from the sink's writer loop (off the request path),
+  * synchronously at the end of `PUT /api/request-logs/settings`, so an
+    operator's change takes effect immediately instead of up to 30s later.
+
+asyncpg has no JSONB codec registered on this pool (see
+database/connection.py), so every value comes back as a raw JSON *string* and
+needs the `isinstance(v, str)` + `json.loads` guard used elsewhere in this
+codebase (services/acme_service.py, utils/activity_log.py).
+"""
+import json
+import logging
+import time
+from dataclasses import dataclass, replace
+from typing import Any, Dict, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+SETTINGS_CATEGORY = "requestlog"
+
+# Kept in sync with the seed in database/migrations.ensure_request_log_settings().
+# backend/tests/test_request_log_settings.py asserts the two agree, so a change
+# here without a change there fails the suite rather than drifting silently.
+DEFAULT_EXCLUDE_PATHS = (
+    "/api/request-logs",
+    "/api/health",
+    "/api/docs",
+    "/api/redoc",
+    "/api/openapi.json",
+    "/.well-known/acme-challenge",
+    "/api/agents/heartbeat",
+    "/static",
+    "/favicon.ico",
+)
+
+
+@dataclass(frozen=True)
+class RequestLogConfig:
+    enabled: bool = True
+    capture_inbound: bool = True
+    capture_outbound: bool = True
+    capture_bodies: bool = True
+    capture_get: bool = True
+    # SUCCESSFUL agent polls only. Off by default because the row rate of this
+    # table is otherwise a linear function of fleet size, not of operator
+    # activity: each agent runs a 30s cycle that issues three logged calls
+    # (config, pending-requests, upgrade-status; the heartbeat is already
+    # excluded) plus two more every fifth cycle. Measured, that is ~9 800 rows
+    # per day PER AGENT, so a 200-node fleet writes ~2M rows/day and reaches the
+    # 500 000 max_rows cap in about six hours - at which point the shipped
+    # "7 days of successes, 30 days of failures" is not 7 and 30, it is 0.25.
+    # FAILED agent calls are always kept regardless of this flag: they are the
+    # half an operator actually needs, and they are rare.
+    capture_agent_success: bool = False
+    max_body_bytes: int = 8192
+    sample_rate: float = 1.0
+    exclude_paths: Tuple[str, ...] = DEFAULT_EXCLUDE_PATHS
+    success_retention_days: int = 7
+    error_retention_days: int = 30
+    max_rows: int = 500000
+    prune_interval_minutes: int = 60
+
+    def as_dict(self) -> Dict[str, Any]:
+        return {
+            "enabled": self.enabled,
+            "capture_inbound": self.capture_inbound,
+            "capture_outbound": self.capture_outbound,
+            "capture_bodies": self.capture_bodies,
+            "capture_get": self.capture_get,
+            "capture_agent_success": self.capture_agent_success,
+            "max_body_bytes": self.max_body_bytes,
+            "sample_rate": self.sample_rate,
+            "exclude_paths": list(self.exclude_paths),
+            "success_retention_days": self.success_retention_days,
+            "error_retention_days": self.error_retention_days,
+            "max_rows": self.max_rows,
+            "prune_interval_minutes": self.prune_interval_minutes,
+        }
+
+
+DEFAULT_CONFIG = RequestLogConfig()
+
+_CACHE: RequestLogConfig = DEFAULT_CONFIG
+_CACHE_AT: float = 0.0
+_TTL_SECONDS: float = 30.0
+
+# Bounds, mirrored by the Pydantic model in routers/request_logs.py. Kept here
+# too because refresh_config() reads whatever is in the table, which may have
+# been written by an older build or by hand.
+_BOUNDS = {
+    "max_body_bytes": (0, 262144),
+    "success_retention_days": (1, 365),
+    "error_retention_days": (1, 365),
+    "max_rows": (1000, 50_000_000),
+    "prune_interval_minutes": (5, 1440),
+}
+
+MAX_EXCLUDE_PATHS = 64
+MAX_EXCLUDE_PATH_LENGTH = 200
+
+
+def get_config() -> RequestLogConfig:
+    """Hot-path read: no await, no DB, no lock. Returns the last snapshot."""
+    return _CACHE
+
+
+def set_config(config: RequestLogConfig) -> None:
+    """Replace the snapshot directly. Used by the settings PUT handler (which
+    already has the validated values) and by tests."""
+    global _CACHE, _CACHE_AT
+    _CACHE = config
+    _CACHE_AT = time.monotonic()
+
+
+def _clamp_int(raw: Any, field: str, fallback: int) -> int:
+    try:
+        value = int(raw)
+    except (TypeError, ValueError):
+        return fallback
+    low, high = _BOUNDS[field]
+    return max(low, min(high, value))
+
+
+def _clamp_float(raw: Any, fallback: float, low: float, high: float) -> float:
+    try:
+        value = float(raw)
+    except (TypeError, ValueError):
+        return fallback
+    return max(low, min(high, value))
+
+
+def _as_bool(raw: Any, fallback: bool) -> bool:
+    if isinstance(raw, bool):
+        return raw
+    if isinstance(raw, (int, float)):
+        return bool(raw)
+    if isinstance(raw, str):
+        lowered = raw.strip().lower()
+        if lowered in ("true", "1", "yes", "on"):
+            return True
+        if lowered in ("false", "0", "no", "off"):
+            return False
+    return fallback
+
+
+def normalize_exclude_paths(raw: Any, fallback: Tuple[str, ...]) -> Tuple[str, ...]:
+    """Coerce whatever is stored into a bounded tuple of path prefixes."""
+    if not isinstance(raw, (list, tuple)):
+        return fallback
+    out = []
+    for item in raw:
+        if not isinstance(item, str):
+            continue
+        candidate = item.strip()
+        if not candidate.startswith("/") or len(candidate) > MAX_EXCLUDE_PATH_LENGTH:
+            continue
+        out.append(candidate)
+        if len(out) >= MAX_EXCLUDE_PATHS:
+            break
+    return tuple(out) if out else fallback
+
+
+def config_from_mapping(values: Dict[str, Any], base: Optional[RequestLogConfig] = None) -> RequestLogConfig:
+    """Build a config from a plain suffix→value mapping, clamping every field.
+
+    Unknown keys are ignored and missing keys keep the value from `base`
+    (default: the shipped defaults), so a partially-seeded table still yields a
+    complete, usable config.
+    """
+    base = base or DEFAULT_CONFIG
+    return replace(
+        base,
+        enabled=_as_bool(values.get("enabled", base.enabled), base.enabled),
+        capture_inbound=_as_bool(values.get("capture_inbound", base.capture_inbound), base.capture_inbound),
+        capture_outbound=_as_bool(values.get("capture_outbound", base.capture_outbound), base.capture_outbound),
+        capture_bodies=_as_bool(values.get("capture_bodies", base.capture_bodies), base.capture_bodies),
+        capture_get=_as_bool(values.get("capture_get", base.capture_get), base.capture_get),
+        capture_agent_success=_as_bool(
+            values.get("capture_agent_success", base.capture_agent_success),
+            base.capture_agent_success,
+        ),
+        max_body_bytes=_clamp_int(values.get("max_body_bytes", base.max_body_bytes), "max_body_bytes", base.max_body_bytes),
+        sample_rate=_clamp_float(values.get("sample_rate", base.sample_rate), base.sample_rate, 0.0, 1.0),
+        exclude_paths=normalize_exclude_paths(values.get("exclude_paths"), base.exclude_paths),
+        success_retention_days=_clamp_int(
+            values.get("success_retention_days", base.success_retention_days),
+            "success_retention_days", base.success_retention_days,
+        ),
+        error_retention_days=_clamp_int(
+            values.get("error_retention_days", base.error_retention_days),
+            "error_retention_days", base.error_retention_days,
+        ),
+        max_rows=_clamp_int(values.get("max_rows", base.max_rows), "max_rows", base.max_rows),
+        prune_interval_minutes=_clamp_int(
+            values.get("prune_interval_minutes", base.prune_interval_minutes),
+            "prune_interval_minutes", base.prune_interval_minutes,
+        ),
+    )
+
+
+def _decode_setting_value(raw: Any) -> Any:
+    """JSONB comes back as a raw string on this pool — parse it, but keep the
+    original text if it is not valid JSON (an operator may have hand-written
+    `7` or `seven`)."""
+    if isinstance(raw, str):
+        try:
+            return json.loads(raw)
+        except (json.JSONDecodeError, ValueError):
+            return raw
+    return raw
+
+
+async def load_settings_rows(conn) -> Dict[str, Any]:
+    """Read the `requestlog.*` rows into a suffix→value mapping."""
+    rows = await conn.fetch(
+        "SELECT key, value FROM system_settings WHERE category = $1",
+        SETTINGS_CATEGORY,
+    )
+    values: Dict[str, Any] = {}
+    for row in rows:
+        key = row["key"]
+        suffix = key.split(".", 1)[1] if "." in key else key
+        values[suffix] = _decode_setting_value(row["value"])
+    return values
+
+
+async def refresh_config(force: bool = True) -> RequestLogConfig:
+    """Reload the snapshot from the database.
+
+    Never raises and never leaves a half-built config behind: on any failure
+    the previous snapshot is kept, so a transient DB blip cannot silently turn
+    logging off (or on).
+    """
+    global _CACHE_AT
+    if not force and (time.monotonic() - _CACHE_AT) < _TTL_SECONDS:
+        return _CACHE
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        values = await load_settings_rows(conn)
+        if values:
+            set_config(config_from_mapping(values))
+        else:
+            # Table not seeded yet (fresh install mid-migration) — keep the
+            # in-code defaults but stamp the timestamp so we don't re-query
+            # every tick.
+            _CACHE_AT = time.monotonic()
+        return _CACHE
+    except Exception as exc:
+        logger.debug(f"refresh_config: keeping previous snapshot ({exc})")
+        _CACHE_AT = time.monotonic()
+        return _CACHE
+    finally:
+        if conn is not None:
+            try:
+                await close_database_connection(conn)
+            except Exception:
+                pass
+
+
+async def maybe_refresh_config() -> RequestLogConfig:
+    """TTL-gated refresh, called from the sink's writer loop."""
+    return await refresh_config(force=False)

backend/utils/request_log_sink.py

@@ -0,0 +1,388 @@
+"""v1.11.0 — batching writer for the unified request/response log.
+
+One row per API call is the highest write volume in this system, and the
+asyncpg pool (min=10/max=50, see database/connection.py) is shared with every
+request handler and four background loops. Acquiring a connection per logged
+request would exhaust it under any real load, so instead:
+
+    hot path  ──offer(row)──▶  bounded asyncio.Queue  ──▶  single writer task
+                                (drops when full)          (executemany batches)
+
+The hot path never awaits I/O and never raises. When the queue is full rows are
+counted as dropped and reported through `GET /api/request-logs/stats`, so a
+saturated logger is visible rather than silent.
+
+Redaction deliberately happens HERE, on the writer task, not in the middleware:
+the recursive walk is the most expensive part of building a row and it has no
+business running inside the request coroutine.
+"""
+import asyncio
+import json
+import logging
+import random
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
+
+from config import (
+    REQUEST_LOG_BATCH_SIZE,
+    REQUEST_LOG_FLUSH_MS,
+    REQUEST_LOG_QUEUE_MAX,
+    REQUEST_LOG_QUEUE_MAX_BYTES,
+)
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_redaction import decode_body, redact_headers
+from utils.request_log_settings import get_config, maybe_refresh_config
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Set by the inbound middleware; read by outbound_span so an outbound call
+# inherits the id of the inbound request that caused it. That is what turns
+# "operator clicked Issue Certificate" and "we POSTed to Let's Encrypt" into
+# one readable trace.
+request_id_context: ContextVar[Optional[str]] = ContextVar(
+    "request_log_request_id", default=None
+)
+
+# `target` on an INBOUND row means the same thing it means on an outbound one:
+# who was on the other end. Set by the middleware when the caller authenticated
+# with an agent API key rather than a user JWT, which is how agent traffic is
+# told apart from operator traffic without a database lookup on the hot path.
+# Deliberately the same literal as http_instrumentation.TARGET_AGENT, so
+# `target = 'agent'` selects the whole conversation with the fleet in both
+# directions; `direction` separates them when that matters.
+TARGET_INBOUND_AGENT = "agent"
+
+_INSERT_SQL = """
+INSERT INTO request_logs (
+    request_id, direction, target, method, url, path, query_params,
+    status_code, status_class, duration_ms, user_id, username, client_ip,
+    user_agent, request_headers, request_body, request_body_bytes,
+    response_headers, response_body, response_body_bytes, error, truncated,
+    created_at
+) VALUES (
+    $1, $2, $3, $4, $5, $6, $7::jsonb,
+    $8, $9, $10, $11, $12, $13::inet,
+    $14, $15::jsonb, $16::jsonb, $17,
+    $18::jsonb, $19::jsonb, $20, $21, $22,
+    $23
+)
+"""
+
+
+def _jsonb(value: Any) -> Optional[str]:
+    """asyncpg has no JSONB codec on this pool, so JSONB params travel as text
+    and are cast in SQL — the house idiom (utils/activity_log.py)."""
+    if value is None:
+        return None
+    try:
+        return json.dumps(value, default=str)
+    except Exception:
+        return json.dumps({"_serialize_error": True})
+
+
+@dataclass
+class RequestLogRow:
+    """One captured exchange, still holding RAW body bytes.
+
+    Decoding and redaction run in `to_params()` on the writer task.
+    """
+
+    request_id: str
+    direction: str
+    method: str
+    url: str
+    target: Optional[str] = None
+    path: Optional[str] = None
+    query_string: Optional[str] = None
+    query_params: Optional[Dict[str, Any]] = None
+    status_code: Optional[int] = None
+    duration_ms: int = 0
+    user_id: Optional[int] = None
+    username: Optional[str] = None
+    client_ip: Optional[str] = None
+    user_agent: Optional[str] = None
+    request_headers: Optional[Dict[str, str]] = None
+    response_headers: Optional[Dict[str, str]] = None
+    request_body_raw: Optional[bytes] = None
+    request_body_bytes: int = 0
+    request_content_type: Optional[str] = None
+    response_body_raw: Optional[bytes] = None
+    response_body_bytes: int = 0
+    response_content_type: Optional[str] = None
+    # Pre-decoded body override, used by outbound spans that hold a dict/str
+    # rather than wire bytes (e.g. the synthetic JWS summary).
+    request_body_value: Optional[Any] = None
+    response_body_value: Optional[Any] = None
+    error: Optional[str] = None
+    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+
+    def queue_weight(self) -> int:
+        """Approximate bytes this row holds while it waits in the queue.
+
+        The buffered bodies are the only part that varies by orders of
+        magnitude (0 to 2 x max_body_bytes); everything else is a handful of
+        short strings and two small header dicts, measured at ~1.4 KB per row.
+        Used to enforce a byte budget alongside the row count, so memory does
+        not become a function of an operator-editable setting.
+        """
+        weight = 1400
+        if self.request_body_raw:
+            weight += len(self.request_body_raw)
+        if self.response_body_raw:
+            weight += len(self.response_body_raw)
+        return weight
+
+    @property
+    def status_class(self) -> int:
+        """`status_code // 100`, or 0 when there was no HTTP response at all
+        (transport error / unhandled exception). 0 is what the error-retention
+        prune treats as an error alongside >= 4."""
+        if not self.status_code:
+            return 0
+        return int(self.status_code) // 100
+
+    def to_params(self) -> List[Any]:
+        req_body, req_truncated = (self.request_body_value, False)
+        if req_body is None:
+            req_body, req_truncated = decode_body(
+                self.request_body_raw, self.request_content_type, self.request_body_bytes
+            )
+
+        res_body, res_truncated = (self.response_body_value, False)
+        if res_body is None:
+            res_body, res_truncated = decode_body(
+                self.response_body_raw, self.response_content_type, self.response_body_bytes
+            )
+
+        return [
+            self.request_id[:64],
+            self.direction,
+            self.target[:32] if self.target else None,
+            (self.method or "")[:10],
+            self.url or "",
+            self.path[:512] if self.path else None,
+            _jsonb(self.query_params),
+            self.status_code,
+            self.status_class,
+            max(0, int(self.duration_ms)),
+            self.user_id,
+            self.username[:50] if self.username else None,
+            self.client_ip,
+            self.user_agent[:1024] if self.user_agent else None,
+            _jsonb(redact_headers(self.request_headers)),
+            _jsonb(req_body),
+            max(0, int(self.request_body_bytes)),
+            _jsonb(redact_headers(self.response_headers)),
+            _jsonb(res_body),
+            max(0, int(self.response_body_bytes)),
+            self.error[:4000] if self.error else None,
+            bool(req_truncated or res_truncated),
+            self.created_at,
+        ]
+
+
+class RequestLogSink:
+    """Bounded queue + single batching writer task (one per uvicorn worker)."""
+
+    def __init__(self, maxsize: int, batch_size: int, flush_ms: int,
+                 max_bytes: int = REQUEST_LOG_QUEUE_MAX_BYTES):
+        self._maxsize = maxsize
+        self._max_bytes = max_bytes
+        self._queued_bytes = 0
+        self._batch_size = batch_size
+        self._flush_seconds = flush_ms / 1000.0
+        self._queue: Optional[asyncio.Queue] = None
+        self._dropped = 0
+        self._written = 0
+        self._failed = 0
+        self._running = False
+
+    # -- lifecycle ---------------------------------------------------------
+
+    def _ensure_queue(self) -> asyncio.Queue:
+        # Created lazily so importing this module never needs a running loop
+        # (matters for the test suite, which imports main.py without one).
+        if self._queue is None:
+            self._queue = asyncio.Queue(maxsize=self._maxsize)
+        return self._queue
+
+    @property
+    def stats(self) -> Dict[str, int]:
+        return {
+            "queued": self._queue.qsize() if self._queue is not None else 0,
+            "queue_capacity": self._maxsize,
+            "queued_bytes": self._queued_bytes,
+            "queue_capacity_bytes": self._max_bytes,
+            "written": self._written,
+            "dropped": self._dropped,
+            "failed_batches": self._failed,
+            "running": 1 if self._running else 0,
+        }
+
+    # -- producer side (hot path) -----------------------------------------
+
+    def offer(self, row: RequestLogRow) -> None:
+        """Enqueue a row. NEVER blocks, NEVER raises.
+
+        Sampling is applied here rather than in the middleware so both
+        directions go through one policy: successful *inbound* traffic can be
+        sampled down, errors never are.
+        """
+        try:
+            cfg = get_config()
+            if not cfg.enabled:
+                return
+            if row.direction == "inbound" and not cfg.capture_inbound:
+                return
+            if row.direction == "outbound" and not cfg.capture_outbound:
+                return
+            # Fleet-scale gate, and the reason this table's size is a function of
+            # operator activity rather than of node count. An agent's 30s cycle
+            # issues three logged calls, plus two more every fifth cycle: ~9 800
+            # rows/day PER AGENT, all of them 200s saying "nothing changed". At a
+            # few hundred nodes that is millions of rows a day, and the row cap is
+            # then reached in hours, which silently shortens the configured
+            # retention for EVERYTHING else in the table - including the failures
+            # the log exists for. Successes are dropped, failures never are.
+            if (
+                row.direction == "inbound"
+                and row.target == TARGET_INBOUND_AGENT
+                and not cfg.capture_agent_success
+                and row.status_class in (1, 2, 3)
+            ):
+                return
+            if (
+                row.direction == "inbound"
+                and cfg.sample_rate < 1.0
+                and row.status_class in (1, 2, 3)
+                and random.random() > cfg.sample_rate
+            ):
+                return
+            if not cfg.capture_bodies:
+                row.request_body_raw = None
+                row.response_body_raw = None
+                row.request_body_value = None
+                row.response_body_value = None
+
+            # Byte budget, checked BEFORE the row count. The count alone does
+            # not bound memory: how much a row weighs is an operator setting,
+            # and `max_body_bytes` at its documented 256 KB ceiling puts the
+            # default 2 000-row queue at ~1 GiB, which is the whole pod limit.
+            # Dropping here is the same visible, counted drop as a full queue.
+            weight = row.queue_weight()
+            if self._queued_bytes + weight > self._max_bytes:
+                raise asyncio.QueueFull
+
+            self._ensure_queue().put_nowait(row)
+            self._queued_bytes += weight
+        except asyncio.QueueFull:
+            self._dropped += 1
+            if self._dropped % 500 == 1:
+                logger.warning(
+                    f"request_log: queue full, {self._dropped} row(s) dropped so far "
+                    f"({self._queue.qsize() if self._queue is not None else 0}/{self._maxsize} rows, "
+                    f"{self._queued_bytes // 1024} KiB/{self._max_bytes // 1024} KiB). "
+                    f"Lower requestlog.max_body_bytes or requestlog.sample_rate. "
+                    f"Raising REQUEST_LOG_QUEUE_MAX also raises the memory this "
+                    f"worker can hold, so raise REQUEST_LOG_QUEUE_MAX_BYTES with it."
+                )
+        except Exception as exc:
+            # Instrumentation must never break the thing it instruments.
+            logger.debug(f"request_log: offer() failed: {exc}")
+
+    # -- consumer side (writer task) --------------------------------------
+
+    async def _collect(self) -> List[RequestLogRow]:
+        """Wait for at least one row, then drain up to batch_size or flush_ms."""
+        queue = self._ensure_queue()
+        first = await queue.get()
+        self._queued_bytes = max(0, self._queued_bytes - first.queue_weight())
+        batch = [first]
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + self._flush_seconds
+        while len(batch) < self._batch_size:
+            remaining = deadline - loop.time()
+            if remaining <= 0:
+                break
+            try:
+                nxt = await asyncio.wait_for(queue.get(), timeout=remaining)
+                self._queued_bytes = max(0, self._queued_bytes - nxt.queue_weight())
+                batch.append(nxt)
+            except asyncio.TimeoutError:
+                break
+        return batch
+
+    async def _write(self, batch: List[RequestLogRow]) -> None:
+        if not batch:
+            return
+        params = []
+        for row in batch:
+            try:
+                params.append(row.to_params())
+            except Exception as exc:
+                logger.debug(f"request_log: row serialization failed, skipped: {exc}")
+        if not params:
+            return
+
+        conn = None
+        try:
+            conn = await get_database_connection()
+            await conn.executemany(_INSERT_SQL, params)
+            self._written += len(params)
+        except Exception as exc:
+            self._failed += 1
+            # A missing table (pre-migration) or a transient pool error must not
+            # take the writer loop down — drop the batch and carry on.
+            logger.warning(f"request_log: batch write failed ({len(params)} rows): {exc}")
+        finally:
+            if conn is not None:
+                try:
+                    await close_database_connection(conn)
+                except Exception:
+                    pass
+
+    async def run(self) -> None:
+        """Writer loop. Started once per worker from startup_event()."""
+        self._running = True
+        logger.info(
+            f"request_log sink started (queue={self._maxsize}, batch={self._batch_size}, "
+            f"flush={int(self._flush_seconds * 1000)}ms)"
+        )
+        try:
+            while True:
+                try:
+                    batch = await self._collect()
+                    await self._write(batch)
+                    await maybe_refresh_config()
+                except asyncio.CancelledError:
+                    raise
+                except Exception as exc:  # pragma: no cover - defensive
+                    logger.error(f"request_log sink loop error: {exc}")
+                    await asyncio.sleep(1)
+        finally:
+            self._running = False
+
+    async def flush(self, timeout: float = 3.0) -> int:
+        """Drain and persist whatever is queued. Called on shutdown."""
+        queue = self._queue
+        if queue is None or queue.empty():
+            return 0
+        written = 0
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + timeout
+        while not queue.empty() and loop.time() < deadline:
+            batch: List[RequestLogRow] = []
+            while not queue.empty() and len(batch) < self._batch_size:
+                row = queue.get_nowait()
+                self._queued_bytes = max(0, self._queued_bytes - row.queue_weight())
+                batch.append(row)
+            await self._write(batch)
+            written += len(batch)
+        return written
+
+
+request_log_sink = RequestLogSink(
+    REQUEST_LOG_QUEUE_MAX, REQUEST_LOG_BATCH_SIZE, REQUEST_LOG_FLUSH_MS
+)

backend/utils/ssrf_guard.py

@@ -0,0 +1,116 @@
+"""
+SSRF guard for outbound HTTP fetches to user/DB-controlled URLs.
+
+GHSA-3vh4-gvxx-wm2p: the ACME `directory_url` was fetched server-side with no
+validation, turning the backend into a request-forwarding primitive against
+loopback / RFC1918 / link-local / cloud-metadata IP space (and reflecting the
+upstream JSON keys back to the caller).
+
+The classification logic mirrors the hardened ACME diagnostics probe
+(services/acme_diagnostics.py, R18b/R18c audits): unwrap IPv4-mapped IPv6, reject
+loopback/link-local/private/multicast/reserved/unspecified, resolve DNS off the
+event loop, and pin the aiohttp connector to IPv4 so the family the guard
+classifies equals the family the connector dials (no dual-stack AAAA bypass).
+
+Deployment note: this project uses ONLY public ACME CAs (e.g. Let's Encrypt), so
+every non-public IP is rejected — there is no internal/private-IP CA to allow.
+Residual: DNS rebinding between validate-time and fetch-time is not fully closed
+(fetching by hostname keeps TLS cert validation working); the IPv4 pin +
+https-only + admin-gating + internal-only exposure keep this residual low.
+"""
+import asyncio
+import ipaddress
+import socket
+from typing import List
+from urllib.parse import urlparse
+
+import aiohttp
+
+# Only https is legitimate for a public ACME directory URL.
+_ALLOWED_SCHEMES = {"https"}
+
+
+class SSRFValidationError(ValueError):
+    """Raised when a URL fails SSRF validation (bad scheme or non-public host)."""
+
+
+def is_public_ip(ip_str: str) -> bool:
+    """Return True only for globally-routable IPv4/IPv6 addresses.
+
+    Unwraps IPv4-mapped IPv6 (``::ffff:127.0.0.1``) before classification so an
+    attacker-controlled AAAA record cannot smuggle loopback/metadata through the
+    IPv6 checks.
+    """
+    try:
+        ip = ipaddress.ip_address(ip_str)
+    except (ValueError, TypeError):
+        return False
+    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    if ip.is_loopback or ip.is_link_local or ip.is_private:
+        return False
+    if ip.is_multicast or ip.is_reserved or ip.is_unspecified:
+        return False
+    return True
+
+
+async def _resolve_ips(host: str, *, timeout: float = 5.0) -> List[str]:
+    """Resolve `host` to IPv4 addresses without blocking the event loop."""
+    loop = asyncio.get_running_loop()
+    _, _, ips = await asyncio.wait_for(
+        loop.run_in_executor(None, socket.gethostbyname_ex, host),
+        timeout=timeout,
+    )
+    return ips or []
+
+
+async def assert_public_url(url: str, *, timeout: float = 5.0) -> None:
+    """Validate that `url` is safe to fetch server-side.
+
+    Requirements: https scheme, and a host that either is a public IP literal or
+    resolves entirely to public IPv4 addresses. Raises ``SSRFValidationError``
+    otherwise. Intended to be called immediately before the outbound request,
+    which MUST use ``safe_connector()`` and ``allow_redirects=False``.
+    """
+    if not url or not isinstance(url, str):
+        raise SSRFValidationError("A URL is required")
+    parsed = urlparse(url.strip())
+    if parsed.scheme.lower() not in _ALLOWED_SCHEMES:
+        raise SSRFValidationError(f"URL scheme must be https (got '{parsed.scheme or 'none'}')")
+    host = parsed.hostname
+    if not host:
+        raise SSRFValidationError("URL has no host")
+
+    # Literal IP host: classify directly, no DNS needed.
+    try:
+        ipaddress.ip_address(host)
+        if not is_public_ip(host):
+            raise SSRFValidationError(f"URL host {host} is not a public IP address")
+        return
+    except ValueError:
+        pass  # hostname, not an IP literal -> resolve below
+
+    try:
+        ips = await _resolve_ips(host, timeout=timeout)
+    except asyncio.TimeoutError:
+        raise SSRFValidationError(f"DNS resolution timed out for {host}")
+    except Exception as e:  # socket.gaierror etc.
+        raise SSRFValidationError(f"DNS resolution failed for {host}: {e}")
+
+    if not ips:
+        raise SSRFValidationError(f"{host} did not resolve to any address")
+    if not all(is_public_ip(ip) for ip in ips):
+        raise SSRFValidationError(
+            f"{host} resolves to a non-public IP {ips} — refusing to fetch (SSRF guard)"
+        )
+
+
+def safe_connector() -> aiohttp.TCPConnector:
+    """IPv4-pinned aiohttp connector.
+
+    Forces the connect family to match what :func:`assert_public_url` classified
+    (closes the dual-stack AAAA bypass). TLS verification stays ON (default), so
+    the request must target the validated hostname. Always combine with
+    ``allow_redirects=False`` at the request call site.
+    """
+    return aiohttp.TCPConnector(family=socket.AF_INET)

backend/version.json

@@ -0,0 +1,5 @@
+{
+  "version": "1.11.1",
+  "releaseName": "Adoption cannot hide a node; Config Import on fresh installs",
+  "releaseDate": "2026-08-15"
+}

docker-compose.localtest.yml

@@ -10,7 +10,6 @@ services:
       dockerfile: Dockerfile
     volumes:
       - haproxy_configs:/etc/haproxy
-      - ./version.json:/app/version.json:ro
 
   frontend:
     image: haproxy-openmanager-frontend:localtest

docker-compose.yml

@@ -22,7 +22,7 @@ services:
 
   # Redis Cache
   redis:
-    image: redis:7-alpine
+    image: redis:8.8.0-alpine
     container_name: haproxy-openmanager-redis
     command: redis-server --maxmemory 2gb --maxmemory-policy volatile-lru --save ""
     ports:
@@ -49,10 +49,22 @@ services:
       - SECRET_KEY=your-secret-key-change-this-in-production
       - DEBUG=False
       - LOG_LEVEL=INFO
-      - PUBLIC_URL=http://localhost:8080
-      - MANAGEMENT_BASE_URL=http://localhost:8080
+      # Interpolated, not hardcoded: these were literals, so a value set in the
+      # host environment or .env was silently ignored and every install kept the
+      # localhost default. That default is also what the ACME challenge backend
+      # falls back to, and HAProxy resolves it ON THE HAPROXY NODE — so on any
+      # deployment where HAProxy is not this machine, it points at the wrong box.
+      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:8080}
+      - MANAGEMENT_BASE_URL=${MANAGEMENT_BASE_URL:-http://localhost:8080}
+      # Empty when unset on the host: the image CMD then falls back to
+      # WEB_CONCURRENCY (uvicorn's native env) and finally to 1.
+      - UVICORN_WORKERS=${UVICORN_WORKERS:-}
     volumes:
       - haproxy_configs:/etc/haproxy
+    # NOT published to the host: the API is reachable only through the nginx
+    # service (host :8080). Host port 8000 is therefore NOT this API — on a box
+    # running Portainer it is Portainer's edge tunnel, which answers 404 and looks
+    # deceptively like a working challenge endpoint.
     expose:
       - "8000"
     depends_on:

frontend/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.7.8",
+  "version": "1.9.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "haproxy-openmanager-frontend",
-      "version": "1.7.8",
+      "version": "1.9.0",
       "license": "AGPL-3.0-or-later",
       "dependencies": {
         "@ant-design/icons": "^5.0.0",
@@ -20,7 +20,7 @@
         "react": "^18.2.0",
         "react-ace": "^10.1.0",
         "react-dom": "^18.2.0",
-        "react-router-dom": "^6.8.0",
+        "react-router-dom": "^6.30.4",
         "react-window": "^1.8.10",
         "react18-json-view": "^0.2.9",
         "recharts": "^2.5.0"
@@ -179,18 +179,18 @@
       }
     },
     "node_modules/@babel/core": {
-      "version": "7.29.0",
-      "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.0.tgz",
-      "integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
+      "version": "7.29.6",
+      "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.6.tgz",
+      "integrity": "sha512-QdxmAo/ikZqqRGA8s43ww8lcql6naWRvEz0FFrl6MIlc7Gi6TroXnSdWa5U/kq6fzcpqpHesicQxFZIieZbyIA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "@babel/code-frame": "^7.29.0",
-        "@babel/generator": "^7.29.0",
+        "@babel/generator": "^7.29.6",
         "@babel/helper-compilation-targets": "^7.28.6",
         "@babel/helper-module-transforms": "^7.28.6",
-        "@babel/helpers": "^7.28.6",
-        "@babel/parser": "^7.29.0",
+        "@babel/helpers": "^7.29.2",
+        "@babel/parser": "^7.29.3",
         "@babel/template": "^7.28.6",
         "@babel/traverse": "^7.29.0",
         "@babel/types": "^7.29.0",
@@ -239,14 +239,14 @@
       }
     },
     "node_modules/@babel/generator": {
-      "version": "7.29.1",
-      "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.1.tgz",
-      "integrity": "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw==",
+      "version": "7.29.7",
+      "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.7.tgz",
+      "integrity": "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/parser": "^7.29.0",
-        "@babel/types": "^7.29.0",
+        "@babel/parser": "^7.29.7",
+        "@babel/types": "^7.29.7",
         "@jridgewell/gen-mapping": "^0.3.12",
         "@jridgewell/trace-mapping": "^0.3.28",
         "jsesc": "^3.0.2"
@@ -472,9 +472,9 @@
       }
     },
     "node_modules/@babel/helper-string-parser": {
-      "version": "7.27.1",
-      "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
-      "integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==",
+      "version": "7.29.7",
+      "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
+      "integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -482,9 +482,9 @@
       }
     },
     "node_modules/@babel/helper-validator-identifier": {
-      "version": "7.28.5",
-      "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz",
-      "integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
+      "version": "7.29.7",
+      "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz",
+      "integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -531,13 +531,13 @@
       }
     },
     "node_modules/@babel/parser": {
-      "version": "7.29.2",
-      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.2.tgz",
-      "integrity": "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA==",
+      "version": "7.29.7",
+      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
+      "integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/types": "^7.29.0"
+        "@babel/types": "^7.29.7"
       },
       "bin": {
         "parser": "bin/babel-parser.js"
@@ -2274,14 +2274,14 @@
       }
     },
     "node_modules/@babel/types": {
-      "version": "7.29.0",
-      "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.0.tgz",
-      "integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==",
+      "version": "7.29.7",
+      "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
+      "integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/helper-string-parser": "^7.27.1",
-        "@babel/helper-validator-identifier": "^7.28.5"
+        "@babel/helper-string-parser": "^7.29.7",
+        "@babel/helper-validator-identifier": "^7.29.7"
       },
       "engines": {
         "node": ">=6.9.0"
@@ -2669,10 +2669,20 @@
       "license": "Python-2.0"
     },
     "node_modules/@eslint/eslintrc/node_modules/js-yaml": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz",
-      "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==",
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz",
+      "integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==",
       "dev": true,
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/puzrin"
+        },
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/nodeca"
+        }
+      ],
       "license": "MIT",
       "dependencies": {
         "argparse": "^2.0.1"
@@ -2756,6 +2766,20 @@
         "node": ">=6"
       }
     },
+    "node_modules/@istanbuljs/load-nyc-config/node_modules/js-yaml": {
+      "version": "3.15.0",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.15.0.tgz",
+      "integrity": "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog==",
+      "dev": true,
+      "license": "MIT",
+      "dependencies": {
+        "argparse": "^1.0.7",
+        "esprima": "^4.0.0"
+      },
+      "bin": {
+        "js-yaml": "bin/js-yaml.js"
+      }
+    },
     "node_modules/@istanbuljs/schema": {
       "version": "0.1.3",
       "resolved": "https://registry.npmjs.org/@istanbuljs/schema/-/schema-0.1.3.tgz",
@@ -4260,9 +4284,9 @@
       }
     },
     "node_modules/@remix-run/router": {
-      "version": "1.23.2",
-      "resolved": "https://registry.npmjs.org/@remix-run/router/-/router-1.23.2.tgz",
-      "integrity": "sha512-Ic6m2U/rMjTkhERIa/0ZtXJP17QUi2CbWE7cqx4J58M8aA3QTfW+2UlQ4psvTX9IO1RfNVhK3pcpdjej7L+t2w==",
+      "version": "1.23.3",
+      "resolved": "https://registry.npmjs.org/@remix-run/router/-/router-1.23.3.tgz",
+      "integrity": "sha512-4An71tdz9X8+3sI4Qqqd2LWd9vS39J7sqd9EU4Scw7TJE/qB10Flv/UuqbPVgfQV9XoK8Np6jNquZitnZq5i+Q==",
       "license": "MIT",
       "engines": {
         "node": ">=14.0.0"
@@ -4654,6 +4678,27 @@
         "url": "https://github.com/sponsors/gregberge"
       }
     },
+    "node_modules/@testing-library/dom": {
+      "version": "10.4.1",
+      "resolved": "https://registry.npmjs.org/@testing-library/dom/-/dom-10.4.1.tgz",
+      "integrity": "sha512-o4PXJQidqJl82ckFaXUeoAW+XysPLauYI43Abki5hABd853iMhitooc6znOnczgbTYmEP6U6/y1ZyKAIsvMKGg==",
+      "dev": true,
+      "license": "MIT",
+      "peer": true,
+      "dependencies": {
+        "@babel/code-frame": "^7.10.4",
+        "@babel/runtime": "^7.12.5",
+        "@types/aria-query": "^5.0.1",
+        "aria-query": "5.3.0",
+        "dom-accessibility-api": "^0.5.9",
+        "lz-string": "^1.5.0",
+        "picocolors": "1.1.1",
+        "pretty-format": "^27.0.2"
+      },
+      "engines": {
+        "node": ">=18"
+      }
+    },
     "node_modules/@testing-library/jest-dom": {
       "version": "5.17.0",
       "resolved": "https://registry.npmjs.org/@testing-library/jest-dom/-/jest-dom-5.17.0.tgz",
@@ -6534,6 +6579,22 @@
         "proxy-from-env": "^2.1.0"
       }
     },
+    "node_modules/axios/node_modules/form-data": {
+      "version": "4.0.6",
+      "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz",
+      "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==",
+      "license": "MIT",
+      "dependencies": {
+        "asynckit": "^0.4.0",
+        "combined-stream": "^1.0.8",
+        "es-set-tostringtag": "^2.1.0",
+        "hasown": "^2.0.4",
+        "mime-types": "^2.1.35"
+      },
+      "engines": {
+        "node": ">= 6"
+      }
+    },
     "node_modules/axobject-query": {
       "version": "4.1.0",
       "resolved": "https://registry.npmjs.org/axobject-query/-/axobject-query-4.1.0.tgz",
@@ -9779,10 +9840,20 @@
       }
     },
     "node_modules/eslint/node_modules/js-yaml": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz",
-      "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==",
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz",
+      "integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==",
       "dev": true,
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/puzrin"
+        },
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/nodeca"
+        }
+      ],
       "license": "MIT",
       "dependencies": {
         "argparse": "^2.0.1"
@@ -10560,22 +10631,6 @@
         "node": ">=6"
       }
     },
-    "node_modules/form-data": {
-      "version": "4.0.5",
-      "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
-      "integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
-      "license": "MIT",
-      "dependencies": {
-        "asynckit": "^0.4.0",
-        "combined-stream": "^1.0.8",
-        "es-set-tostringtag": "^2.1.0",
-        "hasown": "^2.0.2",
-        "mime-types": "^2.1.12"
-      },
-      "engines": {
-        "node": ">= 6"
-      }
-    },
     "node_modules/forwarded": {
       "version": "0.2.0",
       "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
@@ -11082,9 +11137,9 @@
       }
     },
     "node_modules/hasown": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
-      "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
+      "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
       "license": "MIT",
       "dependencies": {
         "function-bind": "^1.1.2"
@@ -11344,9 +11399,9 @@
       }
     },
     "node_modules/http-proxy-middleware": {
-      "version": "2.0.9",
-      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.9.tgz",
-      "integrity": "sha512-c1IyJYLYppU574+YI7R4QyX2ystMtVXZwIdzazUIPIJsHuWNd+mho2j+bKoHftndicGj9yh+xjd+l0yj7VeT1Q==",
+      "version": "2.0.10",
+      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.10.tgz",
+      "integrity": "sha512-RKzRWNPxUZqbuk3BC5mGVJbBnWgr+diEnjJexIOytFbBzDy88Fbh/YvBr3DsNrl1jYAfjWfpATEv0NO35FDuPQ==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -15165,20 +15220,6 @@
       "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==",
       "license": "MIT"
     },
-    "node_modules/js-yaml": {
-      "version": "3.14.2",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.2.tgz",
-      "integrity": "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg==",
-      "dev": true,
-      "license": "MIT",
-      "dependencies": {
-        "argparse": "^1.0.7",
-        "esprima": "^4.0.0"
-      },
-      "bin": {
-        "js-yaml": "bin/js-yaml.js"
-      }
-    },
     "node_modules/jsdom": {
       "version": "16.7.0",
       "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-16.7.0.tgz",
@@ -15227,16 +15268,16 @@
       }
     },
     "node_modules/jsdom/node_modules/form-data": {
-      "version": "3.0.4",
-      "resolved": "https://registry.npmjs.org/form-data/-/form-data-3.0.4.tgz",
-      "integrity": "sha512-f0cRzm6dkyVYV3nPoooP8XlccPQukegwhAnpoLcXy+X+A8KfpGOoXwDr9FLZd3wzgLaBGQBE3lY93Zm/i1JvIQ==",
+      "version": "3.0.5",
+      "resolved": "https://registry.npmjs.org/form-data/-/form-data-3.0.5.tgz",
+      "integrity": "sha512-j23EibVLnp4zNXGW7LjryXYa2X6U/M96yoOX+ybZxwkYajdxRNEqYY3zhh7y0i6kfISKS2jr+EJq1YTUDEv5+w==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "asynckit": "^0.4.0",
         "combined-stream": "^1.0.8",
         "es-set-tostringtag": "^2.1.0",
-        "hasown": "^2.0.2",
+        "hasown": "^2.0.4",
         "mime-types": "^2.1.35"
       },
       "engines": {
@@ -15431,14 +15472,14 @@
       }
     },
     "node_modules/launch-editor": {
-      "version": "2.13.2",
-      "resolved": "https://registry.npmjs.org/launch-editor/-/launch-editor-2.13.2.tgz",
-      "integrity": "sha512-4VVDnbOpLXy/s8rdRCSXb+zfMeFR0WlJWpET1iA9CQdlZDfwyLjUuGQzXU4VeOoey6AicSAluWan7Etga6Kcmg==",
+      "version": "2.14.1",
+      "resolved": "https://registry.npmjs.org/launch-editor/-/launch-editor-2.14.1.tgz",
+      "integrity": "sha512-QWBrQsMpH7gPr965dsKD/3cKWiNoTjpATQf++Xq63N6sKRGMwlVXz41O1IZTMfZQgBctD/K5Zt06+/I6pP6+HA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "picocolors": "^1.1.1",
-        "shell-quote": "^1.8.3"
+        "shell-quote": "^1.8.4"
       }
     },
     "node_modules/leven": {
@@ -16685,9 +16726,9 @@
       }
     },
     "node_modules/postcss": {
-      "version": "8.5.8",
-      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.8.tgz",
-      "integrity": "sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg==",
+      "version": "8.5.10",
+      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.10.tgz",
+      "integrity": "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==",
       "dev": true,
       "funding": [
         {
@@ -19155,12 +19196,12 @@
       }
     },
     "node_modules/react-router": {
-      "version": "6.30.3",
-      "resolved": "https://registry.npmjs.org/react-router/-/react-router-6.30.3.tgz",
-      "integrity": "sha512-XRnlbKMTmktBkjCLE8/XcZFlnHvr2Ltdr1eJX4idL55/9BbORzyZEaIkBFDhFGCEWBBItsVrDxwx3gnisMitdw==",
+      "version": "6.30.4",
+      "resolved": "https://registry.npmjs.org/react-router/-/react-router-6.30.4.tgz",
+      "integrity": "sha512-SVUsDe+DybHM/WmYKIVYhZh1o5Dcuf16yM6WjG02Q9XVFMZIJyHYhwrr6bFBXZkVP6z69kNkMyBCujt8FaFLJA==",
       "license": "MIT",
       "dependencies": {
-        "@remix-run/router": "1.23.2"
+        "@remix-run/router": "1.23.3"
       },
       "engines": {
         "node": ">=14.0.0"
@@ -19170,13 +19211,13 @@
       }
     },
     "node_modules/react-router-dom": {
-      "version": "6.30.3",
-      "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-6.30.3.tgz",
-      "integrity": "sha512-pxPcv1AczD4vso7G4Z3TKcvlxK7g7TNt3/FNGMhfqyntocvYKj+GCatfigGDjbLozC4baguJ0ReCigoDJXb0ag==",
+      "version": "6.30.4",
+      "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-6.30.4.tgz",
+      "integrity": "sha512-q4HvNl+mmDdkS0g+MqiBZNteQJCuimWoOyHMy4T/RQLAn9Z29+E91QXRaxOujeMl2HTzRSS0KFPd7lxX3PjV0Q==",
       "license": "MIT",
       "dependencies": {
-        "@remix-run/router": "1.23.2",
-        "react-router": "6.30.3"
+        "@remix-run/router": "1.23.3",
+        "react-router": "6.30.4"
       },
       "engines": {
         "node": ">=14.0.0"
@@ -19667,32 +19708,20 @@
       }
     },
     "node_modules/resolve-url-loader": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/resolve-url-loader/-/resolve-url-loader-4.0.0.tgz",
-      "integrity": "sha512-05VEMczVREcbtT7Bz+C+96eUO5HDNvdthIiMB34t7FcF8ehcu4wC0sSgPUubs3XW2Q3CNLJk/BJrCU9wVRymiA==",
+      "version": "5.0.0",
+      "resolved": "https://registry.npmjs.org/resolve-url-loader/-/resolve-url-loader-5.0.0.tgz",
+      "integrity": "sha512-uZtduh8/8srhBoMx//5bwqjQ+rfYOUq8zC9NrMUGtjBiGTtFJM42s58/36+hTqeqINcnYe08Nj3LkK9lW4N8Xg==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "adjust-sourcemap-loader": "^4.0.0",
         "convert-source-map": "^1.7.0",
         "loader-utils": "^2.0.0",
-        "postcss": "^7.0.35",
+        "postcss": "^8.2.14",
         "source-map": "0.6.1"
       },
       "engines": {
-        "node": ">=8.9"
-      },
-      "peerDependencies": {
-        "rework": "1.0.1",
-        "rework-visit": "1.0.0"
-      },
-      "peerDependenciesMeta": {
-        "rework": {
-          "optional": true
-        },
-        "rework-visit": {
-          "optional": true
-        }
+        "node": ">=12"
       }
     },
     "node_modules/resolve-url-loader/node_modules/convert-source-map": {
@@ -19702,31 +19731,6 @@
       "dev": true,
       "license": "MIT"
     },
-    "node_modules/resolve-url-loader/node_modules/picocolors": {
-      "version": "0.2.1",
-      "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-0.2.1.tgz",
-      "integrity": "sha512-cMlDqaLEqfSaW8Z7N5Jw+lyIW869EzT73/F5lhtY9cLGoVxSXznfgfXMO0Z5K0o0Q2TkTXq+0KFsdnSe3jDViA==",
-      "dev": true,
-      "license": "ISC"
-    },
-    "node_modules/resolve-url-loader/node_modules/postcss": {
-      "version": "7.0.39",
-      "resolved": "https://registry.npmjs.org/postcss/-/postcss-7.0.39.tgz",
-      "integrity": "sha512-yioayjNbHn6z1/Bywyb2Y4s3yvDAeXGOyxqD+LnVOinq6Mdmd++SW2wUNVzavyyHxd6+DxzWGIuosg6P1Rj8uA==",
-      "dev": true,
-      "license": "MIT",
-      "dependencies": {
-        "picocolors": "^0.2.1",
-        "source-map": "^0.6.1"
-      },
-      "engines": {
-        "node": ">=6.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/postcss/"
-      }
-    },
     "node_modules/resolve-url-loader/node_modules/source-map": {
       "version": "0.6.1",
       "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",
@@ -21191,6 +21195,20 @@
         "node": ">=4"
       }
     },
+    "node_modules/svgo/node_modules/js-yaml": {
+      "version": "3.15.0",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.15.0.tgz",
+      "integrity": "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog==",
+      "dev": true,
+      "license": "MIT",
+      "dependencies": {
+        "argparse": "^1.0.7",
+        "esprima": "^4.0.0"
+      },
+      "bin": {
+        "js-yaml": "bin/js-yaml.js"
+      }
+    },
     "node_modules/svgo/node_modules/nth-check": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-1.0.2.tgz",
@@ -21315,6 +21333,24 @@
         }
       }
     },
+    "node_modules/tailwindcss/node_modules/yaml": {
+      "version": "2.9.0",
+      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz",
+      "integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==",
+      "dev": true,
+      "license": "ISC",
+      "optional": true,
+      "peer": true,
+      "bin": {
+        "yaml": "bin.mjs"
+      },
+      "engines": {
+        "node": ">= 14.6"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/eemeli"
+      }
+    },
     "node_modules/tapable": {
       "version": "2.3.2",
       "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.2.tgz",
@@ -22332,9 +22368,9 @@
       }
     },
     "node_modules/webpack-dev-server/node_modules/ws": {
-      "version": "8.20.0",
-      "resolved": "https://registry.npmjs.org/ws/-/ws-8.20.0.tgz",
-      "integrity": "sha512-sAt8BhgNbzCtgGbt2OxmpuryO63ZoDk/sqaB/znQm94T4fCEsy/yV+7CdC1kJhOU9lboAEU7R3kquuycDoibVA==",
+      "version": "8.21.0",
+      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
+      "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -22429,9 +22465,9 @@
       }
     },
     "node_modules/websocket-driver": {
-      "version": "0.7.4",
-      "resolved": "https://registry.npmjs.org/websocket-driver/-/websocket-driver-0.7.4.tgz",
-      "integrity": "sha512-b17KeDIQVjvb0ssuSDF2cYXSg2iztliJ4B9WdsuB6J952qCPKmnVq4DyW5motImXHDC1cBT/1UezrJVsKw5zjg==",
+      "version": "0.7.5",
+      "resolved": "https://registry.npmjs.org/websocket-driver/-/websocket-driver-0.7.5.tgz",
+      "integrity": "sha512-ZL2+3c7kMBdIRCMz6l8jQMHyGVxj+UL+xVk74Ombiciboca8rHa15L86B19E5oh1pL9Ii/uj54gtsIrZGMo6zA==",
       "dev": true,
       "license": "Apache-2.0",
       "dependencies": {
@@ -23010,9 +23046,9 @@
       }
     },
     "node_modules/ws": {
-      "version": "7.5.10",
-      "resolved": "https://registry.npmjs.org/ws/-/ws-7.5.10.tgz",
-      "integrity": "sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==",
+      "version": "7.5.11",
+      "resolved": "https://registry.npmjs.org/ws/-/ws-7.5.11.tgz",
+      "integrity": "sha512-zS54Oen9bITtp7kp2XM3AydrCIq1D+HwJOuH+c+e4LfpL/lotP5osijd+UoMnxwAam1GN8R4KtLAyIrIcBNpiA==",
       "dev": true,
       "license": "MIT",
       "engines": {

frontend/package.json

@@ -1,12 +1,12 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.8.1",
+  "version": "1.11.1",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
-    "react-router-dom": "^6.8.0",
+    "react-router-dom": "^6.30.4",
     "axios": "^1.16.0",
     "antd": "^5.2.0",
     "@ant-design/icons": "^5.0.0",
@@ -30,6 +30,21 @@
     "@testing-library/user-event": "^14.4.3",
     "@babel/plugin-proposal-private-property-in-object": "^7.21.0"
   },
+  "overrides": {
+    "ws": "7.5.11",
+    "webpack-dev-server": { "ws": "8.21.0" },
+    "form-data": "4.0.6",
+    "jsdom": { "form-data": "3.0.5" },
+    "js-yaml": "3.15.0",
+    "eslint": { "js-yaml": "4.2.0" },
+    "@eslint/eslintrc": { "js-yaml": "4.2.0" },
+    "http-proxy-middleware": "2.0.10",
+    "launch-editor": "2.14.1",
+    "postcss": "8.5.10",
+    "resolve-url-loader": "5.0.0",
+    "@babel/core": "7.29.6",
+    "websocket-driver": "0.7.5"
+  },
   "scripts": {
     "start": "react-scripts start",
     "build": "react-scripts build",

frontend/src/App.css

@@ -209,4 +209,24 @@
 
 [data-theme='dark'] .agent-offline:hover > td {
   background-color: #321518 !important;
+}
+
+/* v1.11.0 — Request Log: failed exchanges (4xx/5xx and transport errors) are
+   tinted so a page of traffic reads at a glance. Defined for BOTH themes here
+   rather than as an inline style, so the dark variant is not forgotten (the
+   v1.10.2 regression class). */
+.request-log-error-row > td {
+  background-color: #fff2f0;
+}
+
+.request-log-error-row:hover > td {
+  background-color: #ffe7e5 !important;
+}
+
+[data-theme='dark'] .request-log-error-row > td {
+  background-color: #2a1215 !important;
+}
+
+[data-theme='dark'] .request-log-error-row:hover > td {
+  background-color: #321518 !important;
 }

frontend/src/App.js

@@ -23,7 +23,8 @@ import {
   BulbOutlined,
   BulbFilled,
   PlusOutlined,
-  ThunderboltOutlined
+  ThunderboltOutlined,
+  FileSearchOutlined
 } from '@ant-design/icons';
 
 import Dashboard from './components/DashboardV2';
@@ -49,6 +50,7 @@ import APIDocumentation from './components/APIDocumentation';
 import IPInventory from './components/IPInventory';
 import SiteWizard from './components/SiteWizard';
 import SiteDrafts from './components/SiteDrafts';
+import RequestLog from './components/RequestLog';  // v1.11.0 — request/response log
 import { AuthProvider, useAuth } from './contexts/AuthContext';
 import { ClusterProvider } from './contexts/ClusterContext';
 import { ThemeProvider, useTheme } from './contexts/ThemeContext';
@@ -183,6 +185,18 @@ const { Text } = Typography;
       icon: <SecurityScanOutlined />,
       label: <Link to="/security">Security</Link>,
     },
+    // v1.11.0 — sits directly above Settings on purpose: the Request Log page
+    // and its retention/capture policy (Settings -> Request Log) are the two
+    // halves of one feature, so they are adjacent in the sidebar.
+    {
+      key: '/request-log',
+      icon: <FileSearchOutlined />,
+      label: (
+        <Tooltip placement="right" title="Request Log — every API call in, and every HTTP call this backend made out (ACME, DNS, agents)">
+          <Link to="/request-log">Request Log</Link>
+        </Tooltip>
+      ),
+    },
     {
       key: '/settings',
       icon: <SettingOutlined />,
@@ -469,6 +483,9 @@ function AppContent() {
               <Route path="/security" element={<Security />} />
               <Route path="/agents" element={<AgentManagement />} />
               <Route path="/configuration" element={<Configuration />} />
+              {/* v1.11.0 — the page gates itself on requestlog.read; the route
+                  is unconditional like every other route in this app. */}
+              <Route path="/request-log" element={<RequestLog />} />
               <Route path="/api-docs" element={<APIDocumentation />} />
               <Route path="/pools" element={<PoolManagement />} />
               <Route path="/settings" element={<Settings />} />
@@ -484,12 +501,30 @@ function AppContent() {
 
 function ThemedApp() {
   const { isDarkMode } = useTheme();
+  const antdTheme = {
+    algorithm: isDarkMode ? theme.darkAlgorithm : theme.defaultAlgorithm,
+  };
+
+  // Ant Design 5: the STATIC message/notification/Modal.confirm APIs render into their own
+  // detached root, so they do not see this ConfigProvider and always fall back to the light
+  // algorithm — a confirm dialog came up white while the app was in dark mode. `holderRender`
+  // wraps that detached root in the same ConfigProvider, which fixes every static call in the
+  // app at once (12 components use Modal.confirm) instead of migrating each one to
+  // App.useApp(). In an effect rather than during render: ConfigProvider.config() mutates
+  // antd module state, and effects still run long before a user can click anything that opens
+  // a static modal. Re-registered on theme change so the toggle takes effect immediately.
+  React.useEffect(() => {
+    ConfigProvider.config({
+      holderRender: (children) => (
+        <ConfigProvider theme={{ algorithm: isDarkMode ? theme.darkAlgorithm : theme.defaultAlgorithm }}>
+          {children}
+        </ConfigProvider>
+      ),
+    });
+  }, [isDarkMode]);
+
   return (
-    <ConfigProvider
-      theme={{
-        algorithm: isDarkMode ? theme.darkAlgorithm : theme.defaultAlgorithm,
-      }}
-    >
+    <ConfigProvider theme={antdTheme}>
       <AuthProvider>
         <ClusterProvider>
           <ProgressProvider>

frontend/src/components/ACLRuleBuilder.js

@@ -53,19 +53,19 @@ const MATCH_TYPE_GROUPS = [
   { label: 'Advanced', options: MATCH_TYPES.filter(m => m.category === 'Advanced') },
 ];
 
-// Phase K Phase D follow-up (Bulgu #12 round 3) — the `-f <file>`
-// flag was removed from the visual builder because HAProxy OpenManager
-// does not provision pattern files onto the HAProxy node filesystem.
-// Allowing `-f` in the visual builder produced ACL rules that passed
-// every UI / Pydantic / heuristic check but ALWAYS failed HAProxy's
-// real `-c` parse at apply time with "failed to open pattern file".
-// Operators reported a multi-page wizard run ending at the Apply
-// Management red-badge for a footgun the UI made trivial to step on.
-// The Pydantic validators on the manual API + wizard reject `-f`
-// universally; the visual builder simply removes the option from the
-// dropdown so operators cannot author the unsupported state.
+// Issue #38 follow-up — `-f <file>` is back in the visual builder:
+// the Bulgu #12 removal (and the matching Pydantic rejects) assumed a
+// missing pattern file would surprise the operator at apply time, but
+// the agent runs `haproxy -c` before every reload so a missing file
+// fails safely (previous config keeps running), and bulk import plus
+// the free-form fields always accepted `-f`. Pattern files are
+// operator-managed host files, same policy as SPOE filter configs
+// (v1.8.8). The value field carries the file path (e.g. flag `-f`
+// + value `/etc/haproxy/blacklist.lst`); an informational note is
+// rendered on rules that use it.
 const FLAGS = [
   { value: '-i', label: '-i  (case insensitive)' },
+  { value: '-f', label: '-f  (pattern file on host)' },
   { value: '-m beg', label: '-m beg  (begins with)' },
   { value: '-m end', label: '-m end  (ends with)' },
   { value: '-m sub', label: '-m sub  (contains)' },
@@ -396,25 +396,23 @@ function ACLDefinitionCard({ rule, index, onChange, onDelete }) {
   };
   const isRaw = rule.raw !== undefined;
 
-  // Phase K Phase D follow-up (Bulgu #12 round 3) — surface `-f` flag
-  // usage inline. The Pydantic validator rejects the rule server-side,
-  // but operators benefit from seeing the error AS they type / when
-  // they re-open a draft that carries a `-f`-flagged rule (e.g. from
-  // a pre-fix draft). The error message matches the Pydantic error
-  // verbatim so support flows are consistent.
+  // Issue #38 follow-up — `-f <file>` pattern-file references are
+  // ACCEPTED now (the Bulgu #12 reject was removed server-side too).
+  // We still detect them, but only to render an informational note:
+  // the referenced file is operator-managed and must exist on every
+  // HAProxy host; a missing file fails safely at the agent's
+  // pre-reload `haproxy -c`.
   const rawHasFileFlag = isRaw && typeof rule.raw === 'string' && ACL_FILE_FLAG_PATTERN.test(rule.raw);
   const structuredHasFileFlag =
     !isRaw && Array.isArray(rule.flags) && rule.flags.includes('-f');
   const hasFileFlag = rawHasFileFlag || structuredHasFileFlag;
-  const cardStyleWithError = hasFileFlag
-    ? { ...ruleCardStyle, border: `1px solid ${token.colorError}` }
-    : ruleCardStyle;
+  const cardStyleWithError = ruleCardStyle;
   const FILE_FLAG_TOOLTIP =
-    "ACL pattern-file references (-f <file>) are not supported by "
-    + "HAProxy OpenManager: the product does not provision pattern "
-    + "files onto the HAProxy node filesystem, so the reference "
-    + "would fail at HAProxy reload time. Remove '-f' and use inline "
-    + "values instead.";
+    "This rule references a pattern file (-f <file>). The file must "
+    + "exist at that exact path on every HAProxy host in the cluster — "
+    + "HAProxy OpenManager does not create or distribute pattern files. "
+    + "A missing file fails safely at 'haproxy -c' (the previous config "
+    + "keeps running).";
 
   if (isRaw) {
     return (
@@ -427,11 +425,10 @@ function ACLDefinitionCard({ rule, index, onChange, onDelete }) {
                 onChange={(e) => onChange(index, { raw: e.target.value })}
                 placeholder="Raw ACL rule (e.g. my_acl path_beg /api)"
                 prefix={<Tag color="default" style={{ marginRight: 4 }}>RAW</Tag>}
-                status={hasFileFlag ? 'error' : undefined}
               />
             </Tooltip>
             {hasFileFlag && (
-              <Text type="danger" style={{ fontSize: 11, display: 'block', marginTop: 2 }}>
+              <Text type="secondary" style={{ fontSize: 11, display: 'block', marginTop: 2 }}>
                 {FILE_FLAG_TOOLTIP}
               </Text>
             )}
@@ -549,12 +546,11 @@ function ACLDefinitionCard({ rule, index, onChange, onDelete }) {
                 onChange={(e) => onChange(index, { ...rule, value: e.target.value })}
                 placeholder={matchDef?.placeholder || 'Value'}
                 size="small"
-                status={structuredHasFileFlag ? 'error' : undefined}
               />
             );
           })()}
           {structuredHasFileFlag && (
-            <Text type="danger" style={{ fontSize: 11, display: 'block', marginTop: 2 }}>
+            <Text type="secondary" style={{ fontSize: 11, display: 'block', marginTop: 2 }}>
               {FILE_FLAG_TOOLTIP}
             </Text>
           )}
@@ -891,11 +887,11 @@ export default function ACLRuleBuilder({ aclRules = [], useBackendRules = [], re
       .map(d => d.name);
   }, [aclDefs]);
 
-  // Phase K Phase D follow-up (Bulgu #12 round 3) — count rules that
-  // still carry the unsupported `-f <file>` flag. Surfaced as a
-  // section-level Alert so operators know the section as a whole
-  // has invalid rules even if individual cards / raw text would
-  // otherwise need scrolling to find them.
+  // Issue #38 follow-up — count rules that reference a `-f <file>`
+  // pattern file. Surfaced as a section-level informational Alert
+  // (non-blocking): the file is operator-managed and must exist on
+  // every HAProxy host; a missing file fails safely at the agent's
+  // pre-reload `haproxy -c`.
   const fileFlagRuleCount = useMemo(() => {
     let count = 0;
     for (const d of aclDefs) {
@@ -1153,19 +1149,19 @@ export default function ACLRuleBuilder({ aclRules = [], useBackendRules = [], re
           Define named conditions to match incoming requests by path, header, source IP, and more.
         </Text>
 
-        {/* Phase K Phase D follow-up (Bulgu #12 round 3) — section-
-            level warning when one or more rules still carry the
-            unsupported `-f <file>` pattern-file flag. Render as a
-            blocking-style Alert so the operator notices BEFORE
-            Submit. The Pydantic validator rejects the same shape
-            server-side; this is the up-front authoring guardrail. */}
+        {/* Issue #38 follow-up — section-level informational note
+            when one or more rules reference `-f <file>` pattern
+            files. Non-blocking: pattern files are operator-managed
+            host files (the Bulgu #12 reject was removed) and a
+            missing file fails safely at the agent's pre-reload
+            `haproxy -c`. */}
         {fileFlagRuleCount > 0 && (
           <Alert
-            type="error"
+            type="info"
             showIcon
             style={{ marginBottom: 8 }}
-            message={`${fileFlagRuleCount} ACL rule${fileFlagRuleCount === 1 ? '' : 's'} use the unsupported \`-f <file>\` flag`}
-            description="HAProxy OpenManager does not provision pattern files onto the HAProxy node filesystem, so any `-f /path/...` reference would fail HAProxy reload at apply time with 'failed to open pattern file'. Remove the `-f` flag and switch to inline values (e.g. `src 10.0.0.0/24` instead of `src -f /etc/haproxy/admins.lst`)."
+            message={`${fileFlagRuleCount} ACL rule${fileFlagRuleCount === 1 ? '' : 's'} reference a \`-f <file>\` pattern file`}
+            description="The referenced file must exist at that exact path on every HAProxy host in the cluster — HAProxy OpenManager does not create or distribute pattern files. A missing file fails safely at 'haproxy -c' (the previous config keeps running)."
           />
         )}
 

frontend/src/components/ACMEAutomation.js

@@ -107,8 +107,14 @@ const ACMEAutomation = () => {
   const [confirming, setConfirming] = useState(false);
   const regChallengeType = Form.useWatch('challenge_type', registerForm);
   const regDnsProvider = Form.useWatch('dns_provider', registerForm);
-  const wizardAccountId = Form.useWatch('account_id', wizardForm);
-  const wizardDomains = Form.useWatch('domains', wizardForm);
+  // `preserve: true` is load-bearing, not a nicety. Each wizard step renders only its own fields —
+  // the domain list on Domains, the account Select on Configuration — and a plain useWatch reports
+  // only fields that are currently REGISTERED, so every one of these read `undefined` from the
+  // Review step onward even though the values were still in the form store. That is what made
+  // Review (and the request it submits) silently fall back to the default ACME account, and it
+  // disabled the wildcard guard at exactly the step where Submit lives.
+  const wizardAccountId = Form.useWatch('account_id', { form: wizardForm, preserve: true });
+  const wizardDomains = Form.useWatch('domains', { form: wizardForm, preserve: true });
   const selectedDnsProvider = dnsProviders.find(p => p.name === regDnsProvider) || null;
 
   // Issue #35: per-account DNS credential management (view/replace/clear after creation).
@@ -217,7 +223,16 @@ const ACMEAutomation = () => {
   const pendingOrders = orders.filter(o =>
     o.status === 'pending' || o.status === 'processing' || o.status === 'ready' || isOrderStuck(o)
   );
-  const activeAccount = accounts.find(a => a.status === 'valid') || null;
+  // The account a request lands on when it carries no explicit account_id. This MUST match the
+  // backend, which takes `ORDER BY created_at DESC LIMIT 1` (routers/letsencrypt.py). The list
+  // arrives ORDER BY id, so picking the first valid entry would preview the OLDEST account — the
+  // opposite one. With two accounts of different challenge methods that made the wizard describe
+  // DNS-01 while the request would actually have gone to an HTTP-01 account.
+  const activeAccount = accounts.filter(a => a.status === 'valid').reduce((best, a) => {
+    if (!best) return a;
+    const delta = new Date(a.created_at) - new Date(best.created_at);
+    return delta > 0 || (delta === 0 && a.id > best.id) ? a : best;
+  }, null);
   const acmeAccount = activeAccount || (accounts.length > 0 ? accounts[accounts.length - 1] : null);
   const acmeEnabledClusters = clusters.filter(c => c.acme_enabled && c.is_active);
   // Issue #35: the cert wizard adapts to the selected account's challenge method.
@@ -260,18 +275,26 @@ const ACMEAutomation = () => {
         return;
       }
       setSubmitting(true);
-      // Resolve the chosen account's challenge method so DNS-01/wildcard requests are explicit.
-      // Use the same resolution as the wizard description (wizardAccount) so what the user reviewed
-      // matches what is sent.
-      const challengeType = wizardAccount?.challenge_type;  // 'http-01' | 'dns-01' | undefined
+      // Resolve the account ONCE and derive everything else from that single object. The id and the
+      // challenge method used to come from different places — account_id from the form store,
+      // challenge_type from wizardAccount — so whenever those two disagreed the request asked for
+      // DNS-01 validation on an HTTP-01 account and the backend answered "The selected ACME account
+      // has no DNS provider configured for DNS-01."
+      const selectedAccountId = values.account_id ?? wizardAccount?.id ?? null;
+      const account = accounts.find(a => a.id === selectedAccountId) || wizardAccount || null;
+      const challengeType = account?.challenge_type;  // 'http-01' | 'dns-01' | undefined
       // Manual DNS-01 can't auto-renew (the wizard shows the switch off+disabled). Send false to
       // match the displayed state rather than relying only on the backend to override it.
-      const autoRenew = wizardDnsManual ? false : (values.auto_renew !== false);
+      const isManualDns01 = challengeType === 'dns-01' && (account?.dns_provider || 'manual') === 'manual';
+      const autoRenew = isManualDns01 ? false : (values.auto_renew !== false);
       const res = await axios.post('/api/letsencrypt/certificates', {
         domains: values.domains,
         cluster_ids: values.cluster_ids || [],
         auto_renew: autoRenew,
-        account_id: values.account_id || null,
+        // Always explicit: sending the resolved id removes the frontend/backend "default account"
+        // guess, which disagreed (the UI previewed the oldest valid account, the backend used the
+        // newest) and made the Review step describe an account the request never went to.
+        account_id: account?.id ?? null,
         challenge_type: challengeType || undefined,
       });
       message.success(res.data?.message || 'Certificate request submitted');
@@ -1092,7 +1115,17 @@ const ACMEAutomation = () => {
             message="Prerequisite Check"
             description={
               <ul style={{ margin: 0, paddingLeft: 20 }}>
-                <li>ACME Account: {activeAccount ? <Tag color="success">Active ({activeAccount.email})</Tag> : <Tag color="error">No active account</Tag>}</li>
+                {/* The account the request will actually use — NOT `activeAccount`, which is only
+                    the default and would name a different account whenever the user picked one. */}
+                <li>ACME Account: {wizardAccount
+                  ? <Tag color={wizardAccount.status === 'valid' ? 'success' : 'error'}>
+                      {wizardAccount.email}{wizardAccount.status !== 'valid' ? ` (${wizardAccount.status})` : ''}
+                    </Tag>
+                  : <Tag color="error">No active account</Tag>}
+                  {wizardAccount && accounts.length > 1 && !wizardAccountId && (
+                    <Typography.Text type="secondary" style={{ fontSize: 12 }}> (default)</Typography.Text>
+                  )}
+                </li>
                 {wizardIsDns01 ? (
                   <>
                     <li>Challenge Method: <Tag>DNS-01</Tag> (TXT record; no port 80 / ACME routing needed)</li>
@@ -1324,8 +1357,11 @@ const ACMEAutomation = () => {
               Next
             </Button>
           )}
+          {/* Gate on the account the request will actually use, and on its status: with no valid
+              account wizardAccount falls back to the newest (deactivated) one, and the Select lists
+              deactivated accounts too, so a bare null-check would leave Submit enabled. */}
           {wizardStep === wizardSteps.length - 1 && (
-            <Button type="primary" onClick={handleRequestCert} loading={submitting} disabled={!activeAccount || wizardWildcardBlocked || wizardDns01Disabled || (!wizardIsDns01 && acmeEnabledClusters.length === 0)}>
+            <Button type="primary" onClick={handleRequestCert} loading={submitting} disabled={wizardAccount?.status !== 'valid' || wizardWildcardBlocked || wizardDns01Disabled || (!wizardIsDns01 && acmeEnabledClusters.length === 0)}>
               Submit Request
             </Button>
           )}

frontend/src/components/AgentManagement.js

@@ -2378,7 +2378,7 @@ const AgentManagement = () => {
                         const element = document.createElement('a');
                         const file = new Blob([installScript], { type: 'text/plain' });
                         element.href = URL.createObjectURL(file);
-                        element.download = `install-haproxy-agent-${selectedPlatform}.sh`;
+                        element.download = `install-agent-${selectedPlatform}.sh`;
                         document.body.appendChild(element);
                         element.click();
                         document.body.removeChild(element);
@@ -2516,7 +2516,7 @@ const AgentManagement = () => {
                                     const element = document.createElement('a');
                                     const file = new Blob([uninstallScript], { type: 'text/plain' });
                                     element.href = URL.createObjectURL(file);
-                                    element.download = `uninstall-haproxy-agent-${selectedPlatform}.sh`;
+                                    element.download = `uninstall-agent-${selectedPlatform}.sh`;
                                     document.body.appendChild(element);
                                     element.click();
                                     document.body.removeChild(element);
@@ -3147,7 +3147,7 @@ const AgentManagement = () => {
                           const element = document.createElement('a');
                           const file = new Blob([deleteUninstallScript], { type: 'text/plain' });
                           element.href = URL.createObjectURL(file);
-                          element.download = `uninstall-haproxy-agent-${agentToDelete.platform || 'linux'}.sh`;
+                          element.download = `uninstall-agent-${agentToDelete.platform || 'linux'}.sh`;
                           document.body.appendChild(element);
                           element.click();
                           document.body.removeChild(element);

frontend/src/components/ApplyManagement.js

@@ -1092,7 +1092,7 @@ const ApplyManagement = () => {
           style={{ 
             marginBottom: 24, 
             borderRadius: 8,
-            border: '1px solid #ffccc7',
+            border: `1px solid ${token.colorErrorBorder}`,
             boxShadow: '0 2px 8px rgba(255, 77, 79, 0.15)'
           }}
           message={
@@ -1348,7 +1348,7 @@ const ApplyManagement = () => {
                         HA / VIP Changes ({pendingChanges.vips.length})
                       </Title>
                       {pendingChanges.vips.map(item => (
-                        <div key={`vip-${item.id}`} style={{ display: 'flex', alignItems: 'center', gap: 8, padding: '8px 12px', marginBottom: 6, border: item.pending_delete ? '1px solid #ffccc7' : '1px solid #f0f0f0', borderRadius: 6, background: item.pending_delete ? '#fff1f0' : undefined }}>
+                        <div key={`vip-${item.id}`} style={{ display: 'flex', alignItems: 'center', gap: 8, padding: '8px 12px', marginBottom: 6, border: `1px solid ${item.pending_delete ? token.colorErrorBorder : token.colorBorderSecondary}`, borderRadius: 6, background: item.pending_delete ? token.colorErrorBg : undefined }}>
                           <CloudServerOutlined style={{ color: item.pending_delete ? '#cf1322' : '#13c2c2' }} />
                           <span style={{ fontWeight: 500 }}>{item.name}</span>
                           <Tag>{item.virtual_ip}/{item.prefix_length}</Tag>
@@ -1374,8 +1374,8 @@ const ApplyManagement = () => {
                         const isEnable = /^cluster-\d+-acme-enable-/.test(v.version_name);
                         return (
                           <div key={v.id} style={{
-                            padding: 10, border: '1px dashed #1890ff', borderRadius: 6, marginBottom: 8,
-                            display: 'flex', alignItems: 'center', justifyContent: 'space-between', backgroundColor: '#f0f8ff'
+                            padding: 10, border: `1px dashed ${token.colorPrimary}`, borderRadius: 6, marginBottom: 8,
+                            display: 'flex', alignItems: 'center', justifyContent: 'space-between', backgroundColor: token.colorInfoBg
                           }}>
                             <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
                             <span>
@@ -1419,7 +1419,7 @@ const ApplyManagement = () => {
                                     display: 'flex',
                                     alignItems: 'center',
                                     justifyContent: 'space-between',
-                                    backgroundColor: '#f0f8ff'
+                                    backgroundColor: token.colorInfoBg
                                   }}>
                                     <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
                                     <Tag color="orange">PENDING</Tag>
@@ -1443,7 +1443,7 @@ const ApplyManagement = () => {
                                     display: 'flex',
                                     alignItems: 'center',
                                     justifyContent: 'space-between',
-                                    backgroundColor: '#f6ffed'
+                                    backgroundColor: token.colorSuccessBg
                                   }}>
                                     <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
                                     <Tag color="orange">PENDING</Tag>
@@ -1518,9 +1518,13 @@ const ApplyManagement = () => {
               </Descriptions>
             </div>
 
-            {/* Pending Versions Section */}
+            {/* Pending Versions Section.
+                Theme tokens, not the light-mode literals #fffbe6/#ffe58f: in dark mode those
+                produced a cream panel with light text on it, so the version name, timestamp
+                and "View Change" link were unreadable. colorWarningBg/Border track the
+                algorithm, so the "pending" tint survives in both themes. */}
             {pendingVersions.length > 0 && (
-              <div style={{ marginBottom: 24, background: '#fffbe6', borderRadius: 8, border: '1px solid #ffe58f', padding: '16px 16px 8px' }}>
+              <div style={{ marginBottom: 24, background: token.colorWarningBg, borderRadius: 8, border: `1px solid ${token.colorWarningBorder}`, padding: '16px 16px 8px' }}>
                 <Title level={5} style={{ marginTop: 0 }}>
                   <ClockCircleOutlined style={{ marginRight: 8, color: '#faad14' }} />
                   Pending Changes ({pendingVersions.length})
@@ -1765,8 +1769,8 @@ const ApplyManagement = () => {
                             <div>
                               {/* Parsed suggestion */}
                               {agentSync.parsed_error?.suggestion && (
-                                <div style={{ marginBottom: 12, padding: '8px 12px', background: '#fff7e6', borderRadius: 4, border: '1px solid #ffd591' }}>
-                                  <Text strong style={{ color: '#ad4e00' }}>Recommendation: </Text>
+                                <div style={{ marginBottom: 12, padding: '8px 12px', background: token.colorWarningBg, borderRadius: 4, border: `1px solid ${token.colorWarningBorder}` }}>
+                                  <Text strong style={{ color: token.colorWarningText }}>Recommendation: </Text>
                                   <Text>{agentSync.parsed_error.suggestion}</Text>
                                 </div>
                               )}
@@ -1951,13 +1955,17 @@ const ApplyManagement = () => {
                     {diffData.changes && diffData.changes.length > 0 ? (
                       diffData.changes.map((change, index) => (
                         <div key={index} style={{ marginBottom: '4px' }}>
+                          {/* Token-based, not the light literals #f6ffed/#fff2f0: those stayed
+                              near-white in dark mode, so the added/removed rows glared against
+                              the dark diff panel around them. colorSuccessBg/colorErrorBg darken
+                              with the algorithm while keeping the green/red semantics. */}
                           {change.type === 'added' && (
-                            <div style={{ backgroundColor: '#f6ffed', color: '#52c41a', padding: '2px 8px', borderLeft: '3px solid #52c41a' }}>
+                            <div style={{ backgroundColor: token.colorSuccessBg, color: token.colorSuccessText, padding: '2px 8px', borderLeft: `3px solid ${token.colorSuccess}` }}>
                               + {change.line}
                             </div>
                           )}
                           {change.type === 'removed' && (
-                            <div style={{ backgroundColor: '#fff2f0', color: '#ff4d4f', padding: '2px 8px', borderLeft: '3px solid #ff4d4f' }}>
+                            <div style={{ backgroundColor: token.colorErrorBg, color: token.colorErrorText, padding: '2px 8px', borderLeft: `3px solid ${token.colorError}` }}>
                               - {change.line}
                             </div>
                           )}

frontend/src/components/BulkConfigImport.js

@@ -458,6 +458,8 @@ backend web-backend
           {record.request_headers && <Tag color="blue">Req Headers</Tag>}
           {record.response_headers && <Tag color="green">Resp Headers</Tag>}
           {record.tcp_request_rules && <Tag color="purple">TCP Rules</Tag>}
+          {record.filters && <Tag color="magenta">Filters</Tag>}
+          {record.log_format && <Tag color="geekblue">Log Format</Tag>}
           {record.acl_rules && record.acl_rules.length > 0 && <Tag color="orange">{record.acl_rules.length} ACLs</Tag>}
           {record.use_backend_rules && record.use_backend_rules.length > 0 && <Tag color="cyan">{record.use_backend_rules.length} Routes</Tag>}
         </Space>
@@ -1076,8 +1078,9 @@ backend web-backend
                         size="small"
                         expandable={{
                           expandedRowRender: (frontend) => {
-                            const hasDetails = frontend.request_headers || frontend.response_headers || 
-                                             frontend.options || frontend.tcp_request_rules || 
+                            const hasDetails = frontend.request_headers || frontend.response_headers ||
+                                             frontend.options || frontend.tcp_request_rules ||
+                                             frontend.filters || frontend.log_format ||
                                              (frontend.acl_rules && frontend.acl_rules.length > 0) ||
                                              (frontend.use_backend_rules && frontend.use_backend_rules.length > 0);
                             
@@ -1157,12 +1160,52 @@ backend web-backend
                                         </span>
                                       }
                                     >
-                                      <MultiLineDiffRenderer 
+                                      <MultiLineDiffRenderer
                                         value={frontend.tcp_request_rules}
                                         changeInfo={frontend._changes?.tcp_request_rules}
                                       />
                                     </Descriptions.Item>
                                   )}
+                                  {/* Issue #38: SPOE filters */}
+                                  {frontend.filters && (
+                                    <Descriptions.Item
+                                      label={
+                                        <span>
+                                          Filters (SPOE/WAF)
+                                          {frontend._changes?.filters && (
+                                            <Tag color="green" style={{ marginLeft: 8, fontSize: '10px' }}>
+                                              {frontend._changes.filters.old ? 'CHANGED' : 'NEW'}
+                                            </Tag>
+                                          )}
+                                        </span>
+                                      }
+                                    >
+                                      <MultiLineDiffRenderer
+                                        value={frontend.filters}
+                                        changeInfo={frontend._changes?.filters}
+                                      />
+                                    </Descriptions.Item>
+                                  )}
+                                  {/* Issue #38: frontend log-format */}
+                                  {frontend.log_format && (
+                                    <Descriptions.Item
+                                      label={
+                                        <span>
+                                          Log Format
+                                          {frontend._changes?.log_format && (
+                                            <Tag color="green" style={{ marginLeft: 8, fontSize: '10px' }}>
+                                              {frontend._changes.log_format.old ? 'CHANGED' : 'NEW'}
+                                            </Tag>
+                                          )}
+                                        </span>
+                                      }
+                                    >
+                                      <MultiLineDiffRenderer
+                                        value={frontend.log_format}
+                                        changeInfo={frontend._changes?.log_format}
+                                      />
+                                    </Descriptions.Item>
+                                  )}
                                   {frontend.acl_rules && frontend.acl_rules.length > 0 && (
                                     <Descriptions.Item label={`ACL Rules (${frontend.acl_rules.length})`}>
                                       {frontend.acl_rules.map((acl, idx) => (