fix(agent): adoption cannot hide a node; Config Import on fresh installs (v1.11.1)

Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.

DISCOVERY REPORTS THAT NEVER REACHED THE SERVER

The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.

  - the report is cached only on a 2xx;
  - GET /agents/{name}/keepalived-config now reports whether the server actually
    holds a discovery for that agent, and the cache may only suppress while it
    says yes - which is what lets nodes stuck from earlier releases recover on
    their own, with nobody touching them;
  - the flag is parsed with has() + tostring, not `// empty`: jq's alternative
    operator returns the alternative for **false** as well as null, so the naive
    form could not tell "no record" from "older backend" and the recovery would
    have been completely inert;
  - a 400/413/422 records the refusal so identical bytes are not re-posted
    forever - 4xx and 5xx agent calls are never sampled out of the request log,
    so an unattended loop would write a row carrying the whole config every
    cycle - while 401 and 404 keep retrying, because here they mean a token
    rotation or an agent row briefly absent, not a bad payload;
  - the CLEAR path had the same exit-code defect, where it left a stale row
    offering a managed node for adoption with nothing to ever retry it.

CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS

check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.

DETERMINISTIC CONFIG PATH

A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.

Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.

Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.

.env.template

@@ -17,6 +17,34 @@ REDIS_URL=redis://redis:6379
 # Change this to a strong random string in production
 SECRET_KEY=your-secret-key-change-this-in-production
 
+# ----------------------------------------------------------------------------
+# Optional per-purpose encryption keys.
+#
+# Every secret the application stores is encrypted at rest with Fernet. Each class
+# derives its own key, so rotating one never affects another. If a variable below is
+# unset, that class's key is derived from SECRET_KEY via HKDF — which works, but means
+# rotating SECRET_KEY makes the existing values of that class UNDECRYPTABLE. Set an
+# explicit key (urlsafe-base64, 32 bytes) in production if you want independent
+# rotation. Generate one with:
+#   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
+# ----------------------------------------------------------------------------
+
+# VRRP secrets for HA/VIP (Issue #27).
+# VIP_ENCRYPTION_KEY=
+
+# TOTP secrets for multi-factor authentication (Issue #18).
+# MFA_ENCRYPTION_KEY=
+
+# Per-account DNS provider credentials for ACME DNS-01 (Issue #35).
+# Rotating this without re-entering credentials makes DNS-01 renewals fail until the
+# affected accounts' credentials are re-saved in ACME Automation.
+# DNS_PROVIDER_ENCRYPTION_KEY=
+
+# Private keys of PENDING CSRs, held only until the signed certificate is imported
+# (Issue #53). Rotating this while CSRs are out for signature makes those CSRs
+# unusable — they must be deleted and re-created.
+# CSR_ENCRYPTION_KEY=
+
 # ============================================================================
 # PUBLIC URL CONFIGURATION
 # ============================================================================
@@ -31,6 +59,13 @@ PUBLIC_URL=http://localhost:8000
 
 # Management base URL (defaults to PUBLIC_URL if not set)
 # Override this if your management interface is on a different URL
+#
+# This is also the last-resort fallback for the ACME HTTP-01 challenge backend,
+# i.e. the address written into haproxy.cfg as `server _acme_mgmt <host>:<port>`.
+# That address is resolved BY HAPROXY, ON THE HAPROXY NODE. If HAProxy runs
+# anywhere other than this machine, localhost points at the wrong box and HTTP-01
+# validation fails while DNS-01 keeps working. Set a routable address, with the
+# port, e.g. MANAGEMENT_BASE_URL=http://10.90.1.4:8080
 MANAGEMENT_BASE_URL=http://localhost:8000
 
 # ============================================================================
@@ -54,6 +89,15 @@ AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
 # Config sync interval in seconds
 AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 
+# ============================================================================
+# BACKEND PERFORMANCE
+# ============================================================================
+# Number of uvicorn worker processes for the backend API (default: 1).
+# On multi-core hosts, setting this to the core count (e.g. 2) lets the API
+# use all cores. Safe to increase: background tasks are multi-replica safe
+# (the k8s deployment already runs 2+ replicas via HPA).
+UVICORN_WORKERS=1
+
 # ============================================================================
 # CORS CONFIGURATION
 # ============================================================================
@@ -62,6 +106,44 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 # Example: http://haproxy-manager.example.com,http://localhost:8080
 CORS_ORIGINS=
 
+# ============================================================================
+# REQUEST / RESPONSE LOG (v1.11.0)
+# ============================================================================
+# Records every inbound API call and every outbound HTTP call the backend makes
+# (ACME, DNS providers, agents) into the `request_logs` table, browsable under
+# "Request Log" in the UI.
+#
+# Only the four settings below are environment-level, because they decide
+# whether the middleware is registered at all and how much memory the writer
+# queue may hold. Everything an operator tunes day to day — retention windows,
+# body capture, sampling, excluded paths — lives in the database and is edited
+# in Settings -> Request Log.
+
+# Hard kill-switch. When false the logging middleware is NEVER added to the ASGI
+# stack and neither the writer nor the retention task starts: zero overhead, not
+# even a settings lookup. Requires a restart to change.
+# (The `enabled` toggle in Settings is the no-restart equivalent.)
+REQUEST_LOG_ENABLED=true
+
+# Per-worker in-process queue depth. When it fills, rows are DROPPED and counted
+# rather than blocking the request — the drop count is shown on the Request Log
+# page (per worker).
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Hard memory ceiling for that same queue, per worker. The row count above does
+# NOT bound memory on its own: `requestlog.max_body_bytes` is editable from
+# Settings up to 256 KB and a row can carry it twice, so at that ceiling a
+# 2000-row queue would hold ~1 GiB. Whichever limit is reached first stops the
+# queue. Raise this if you raise REQUEST_LOG_QUEUE_MAX.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Rows per batched INSERT. One connection is taken from the pool per batch, not
+# per request.
+REQUEST_LOG_BATCH_SIZE=100
+
+# Maximum wait before a partial batch is flushed, in milliseconds.
+REQUEST_LOG_FLUSH_MS=500
+
 # ============================================================================
 # FRONTEND CONFIGURATION (React)
 # ============================================================================

.github/workflows/docker-build.yml

@@ -7,6 +7,8 @@ on:
 jobs:
   build_and_push:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
 
     steps:
       - name: checkout
@@ -19,13 +21,17 @@ jobs:
       - name: read product version
         id: prodversion
         run: |
-          VERSION=$(jq -r .version version.json)
+          VERSION=$(jq -r .version backend/version.json)
           if [ -z "$VERSION" ] || [ "$VERSION" = "null" ]; then
-            echo "Failed to read product version from version.json" >&2
+            echo "Failed to read product version from backend/version.json" >&2
             exit 1
           fi
           echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
 
+      # version.json now lives at backend/version.json (inside the ./backend build
+      # context), so `COPY . .` bakes it into the image directly — no staging step
+      # is needed and the backend reports the correct version in every deployment,
+      # not just this workflow's builds.
       - name: set up qemu
         uses: docker/setup-qemu-action@v3
 
@@ -62,3 +68,30 @@ jobs:
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.version.outputs.TAG }}
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.prodversion.outputs.VERSION }}
 
+      # Keep the GitHub Releases/Tags in sync with version.json. The docker
+      # images above are tagged with the product version, but nothing here
+      # created the matching git tag, so the repo's Tags/Releases drifted
+      # behind (stuck at the last manually-created tag). After the images are
+      # pushed, cut a Release (which also creates the tag) for the current
+      # version.json, but only if one does not already exist, so re-runs
+      # without a version bump are a no-op.
+      - name: create github release from version.json
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          VERSION="${{ steps.prodversion.outputs.VERSION }}"
+          TAG="v${VERSION}"
+          RELEASE_NAME=$(jq -r '.releaseName // empty' backend/version.json)
+          if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "Release $TAG already exists, skipping."
+          else
+            TITLE="$TAG"
+            [ -n "$RELEASE_NAME" ] && TITLE="$TAG — $RELEASE_NAME"
+            gh release create "$TAG" \
+              --repo "$GITHUB_REPOSITORY" \
+              --target "$GITHUB_SHA" \
+              --title "$TITLE" \
+              --notes "Automated release for $TAG (from version.json)."
+            echo "Created release $TAG"
+          fi
+

.gitignore

@@ -39,9 +39,6 @@ venv.bak/
 *.sqlite
 *.sqlite3
 
-# Docker
-.dockerignore
-
 # IDE
 .vscode/
 .idea/

CONFIG.md

@@ -131,6 +131,66 @@ if (window.location) {
 }
 ```
 
+### REQUEST_LOG_ENABLED (v1.11.0)
+
+**Ne İşe Yarar**: Request/Response Log özelliğinin sert (hard) kill-switch'i. `false` yapıldığında
+loglama middleware'i ASGI zincirine **hiç eklenmez**, yazıcı ve retention görevleri başlatılmaz —
+yani sıfır ek yük, ayar okuması bile yapılmaz. Değişiklik için restart gerekir.
+
+**Örnekler**:
+```bash
+# Varsayılan: açık
+REQUEST_LOG_ENABLED=true
+
+# Tamamen kapat (ör. çok yüksek trafikli kurulum, veya regülasyon gereği)
+REQUEST_LOG_ENABLED=false
+```
+
+**Nasıl Kullanılır**:
+1. Restart gerektirmeden kapatmak isterseniz bunun yerine **Settings → Request Log → Enable request
+   log** anahtarını kullanın; o anında etkili olur.
+2. Retention süreleri, gövde (body) yakalama, örnekleme oranı ve hariç tutulan path'ler bu env
+   değişkeniyle değil, veritabanındaki `requestlog.*` ayarlarıyla yönetilir — arayüzden düzenlenir.
+3. Disk büyümesi asıl operasyonel konudur: sırasıyla `sample_rate`'i düşürün, `capture_get`'i
+   kapatın, `capture_bodies`'i kapatın, sonra `success_retention_days`'i kısaltın.
+
+### REQUEST_LOG_QUEUE_MAX / REQUEST_LOG_QUEUE_MAX_BYTES / REQUEST_LOG_BATCH_SIZE / REQUEST_LOG_FLUSH_MS (v1.11.0)
+
+**Ne İşe Yarar**: Log satırlarını yazan arka plan görevinin ayarları. Satırlar sınırlı bir kuyruğa
+konur ve toplu (batch) INSERT ile yazılır; böylece istek yolu asla veritabanını beklemez.
+
+**Örnekler**:
+```bash
+# Worker başına kuyruk derinliği. Dolduğunda satırlar DÜŞÜRÜLÜR (sayılır ve
+# Request Log sayfasında gösterilir), istek bloklanmaz.
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Aynı kuyruğun BAYT tavanı (worker başına). Satır sayısı tek başına belleği
+# sınırlamaz: `max_body_bytes` Settings'ten 256 KB'a kadar ayarlanabilir ve bir
+# satır bunu iki kez taşıyabilir, o tavanda 2000 satırlık kuyruk ~1 GiB tutar.
+# Hangi sınır önce dolarsa kuyruk orada durur.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Tek INSERT'te kaç satır yazılacağı (havuzdan istek başına değil, batch başına
+# bir bağlantı alınır)
+REQUEST_LOG_BATCH_SIZE=100
+
+# Yarım dolu bir batch'in en fazla ne kadar bekletileceği (ms)
+REQUEST_LOG_FLUSH_MS=500
+```
+
+**Nasıl Kullanılır**:
+1. Request Log sayfasında "rows dropped" uyarısı görüyorsanız önce `requestlog.max_body_bytes`
+   veya `sample_rate`'i düşürün. `REQUEST_LOG_QUEUE_MAX`'ı artırmak bu worker'ın tutabileceği
+   belleği de artırır; artıracaksanız `REQUEST_LOG_QUEUE_MAX_BYTES`'ı da birlikte artırın.
+2. Bu değerler worker başınadır — `UVICORN_WORKERS` arttıkça toplam bellek de o oranda artar.
+   `GET /api/request-logs/stats` içindeki sayaçlar da worker başınadır ve yanıtta öyle
+   etiketlenir; 4 worker'da gördüğünüz düşüş sayısı gerçeğin dörtte biridir.
+3. Büyük filolarda tek en etkili ayar `requestlog.capture_agent_success`'tir (varsayılan kapalı).
+   Açık olsaydı 200 düğümlük bir filo günde ~2M satır yazar ve 500.000 satır tavanına 6 saatte
+   ulaşırdı; yapılandırılmış 7 gün / 30 gün saklama o noktada birkaç saate iner. Başarısız ajan
+   çağrıları bu ayardan bağımsız olarak her zaman loglanır.
+
 ## 🚀 Deployment Senaryoları
 
 ### Docker Compose

LICENSE

@@ -1,22 +1,679 @@
-MIT License
+HAProxy OpenManager
+Copyright (C) 2025-2026  Taylan Bakırcıoğlu and HAProxy OpenManager Contributors
 
-Copyright (c) 2025 HAProxy OpenManager Contributors
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+---
 
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.

UPGRADE_GUIDE.md

@@ -1,3 +1,632 @@
+# Upgrade Notes — v1.11.1 (adoption cannot hide a node; Config Import on fresh installs)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. After deploying, sync the Linux agent script from **Agent Management** and let
+the agents upgrade, or none of this reaches the nodes.
+
+- **A node that never appeared under *Unmanaged keepalived detected* now recovers by itself.**
+  The agent caches the hash of its last discovery report and skips re-posting while it matches.
+  Delivery was judged by `curl`'s exit code, which is 0 for 5xx too, so a rejected report was
+  cached as delivered and a hand-maintained config — which never changes on its own — kept that
+  node hidden. The report is now cached only on a 2xx, and the config endpoint reports whether
+  the server actually holds a discovery for that agent, so the cache can only suppress while the
+  server agrees. **No access to the nodes is needed**; affected nodes reappear within one poll
+  cycle (~2.5 min) after the agents pick up the new script.
+- **Permanent refusals do not loop.** A 400, 413 or 422 means the payload itself is unacceptable,
+  so the refusal is recorded and the same bytes are not re-posted; fixing the file releases the
+  brake, because it is keyed to the content hash. **401 and 404 keep retrying** — in this system
+  they mean a token rotation or an agent row briefly absent while it re-registers, and braking on
+  them would have re-created the very failure above. This matters beyond noise: 4xx and 5xx agent
+  calls are never sampled out of the request log, so a loop would write a row carrying the whole
+  `keepalived.conf` every cycle on every affected node.
+- **The clear path had the same defect.** When a node becomes managed the agent tells the server
+  to drop the discovery; that too was judged by the exit code, so a rejected clear left a stale
+  row offering a **managed** node for adoption, with nothing to ever retry it.
+- **Config Import now works on a freshly installed agent.** `check_config_requests` uploads a
+  node's live `haproxy.cfg` when you ask for it. It was defined in the installer and in the
+  self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by
+  `type`, so on such a node the feature was a silent no-op: the request was made and nothing ever
+  arrived. Any agent that had self-upgraded at least once already had it, which is why it went
+  unnoticed. A freshly installed agent now polls the pending-requests endpoint once per cycle,
+  exactly as every upgraded agent already does — **no node running today changes behaviour**.
+- **The keepalived.conf path is resolved deterministically.** A pool may hold more than one
+  cluster and the join was unordered, so the path handed to an agent could differ between polls
+  whenever two clusters disagreed on it — the agent would inspect a file that is not there and the
+  node would never appear, intermittently. A customised path now wins over the shipped default,
+  then the lowest cluster id. With one cluster per pool, or when every cluster carries the
+  default, the value is byte-identical to before.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous behaviour, in which
+a rejected discovery report is never retried and Config Import is absent on fresh installs.
+
+---
+
+# Upgrade Notes — v1.11.0 (Unified request/response log)
+
+**Adds one new table and bumps `SCHEMA_VERSION` 11 → 12. The migration runs automatically on the
+first backend start.** No agent impact, no rendered-config change, no change to any existing API
+shape or response body.
+
+> **12, not 11.** The feature was developed against v1.10.3, where `SCHEMA_VERSION` was still 10, and
+> proposed 11. v1.10.4 took 11 in the meantime. Shipping it as 11 would have been silently inert:
+> `run_all_migrations()` returns early on `applied_version >= SCHEMA_VERSION`, so every database
+> already at 11 would have skipped the entire sequence and received neither `request_logs` nor the
+> `requestlog.*` permissions, while a fresh install would have received both. If you are coming from
+> a pre-release build that recorded 11 for this feature, no action is needed: the bump to 12 re-runs
+> the (idempotent) sequence and creates whatever is missing.
+
+- **New table `request_logs`** (BIGSERIAL primary key, 9 indexes). Created empty and starts filling
+  immediately. The shipped defaults keep 7 days of successful requests, 30 days of failed ones, and
+  at most 500 000 rows — whichever limit is reached first. Change any of it in
+  *Settings → Request Log*.
+- **Successful agent polls are NOT logged** (`requestlog.capture_agent_success`, default off).
+  Failed agent calls always are. This is what keeps the table's size a function of operator
+  activity rather than of how many nodes you run. Measured at 2 424 bytes/row on PostgreSQL 15
+  against the real schema and all nine indexes, with each agent issuing ~9 792 logged calls a day:
+
+  | fleet | if successful polls were logged | with the shipped default |
+  |---|---|---|
+  | 20 nodes  | 196k rows/day, 453 MB/day, row cap in 61 h | ~21k rows/day, 48 MB/day, cap in 24 d |
+  | 200 nodes | 2.0M rows/day, 4.5 GB/day, row cap in 6 h  | ~30k rows/day, 69 MB/day, cap in 17 d |
+  | 500 nodes | 4.9M rows/day, 11 GB/day, row cap in 2.5 h | ~44k rows/day, 103 MB/day, cap in 11 d |
+
+  The row cap always holds, but it holds by deleting — so without this default the configured
+  "7 days of successes, 30 days of failures" silently becomes a few hours of both, and the forensic
+  record the feature exists for is evicted by polling noise. Turn it on temporarily when debugging
+  a specific node, then turn it back off.
+- **Runtime cost is measured, not estimated.** The middleware adds 27.7 µs (p50) per request and
+  1.4 µs on an excluded path; redaction runs on the writer task, off the request path, at 18.8 µs
+  per row. At 500 nodes that is **0.096 % of one core** in total.
+- **New permissions `requestlog.read` and `requestlog.manage`.** Both are granted to `super_admin`
+  and `security_admin`; `operator` gets `requestlog.read` only; `viewer` gets neither, because
+  captured request/response bodies are a broader disclosure surface than the read-only configuration
+  views a viewer is meant to have. Custom roles can be granted either from **Users → Roles**.
+- **⚠️ Built-in roles are re-seeded to their defaults.** This is the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump, not something new in this release, but it bites here because this release
+  bumps: the version gate re-runs the whole sequence and
+  `update_system_roles_to_enterprise_rbac()` issues an unconditional
+  `UPDATE roles SET … permissions = <defaults> WHERE name = …` for the four **built-in** roles
+  (`super_admin`, `operator`, `security_admin`, `viewer`). **Any customisation you made to a
+  built-in role is reverted.** Roles you created yourself are untouched (the update matches on
+  name). To preserve customisation, export with `GET /api/roles` before upgrading and re-apply with
+  `PUT /api/roles/{id}`, or move the customisation into a custom role.
+- **Bodies are captured, redacted and capped at 8 KB.** Passwords, tokens, API keys, private-key
+  PEMs, JWT-shaped values, `Authorization` / `Cookie` headers, ACME JWS payloads and DNS-provider
+  credentials are never stored. Headers use an allowlist — anything not on it is dropped rather than
+  saved. Review *Settings → Request Log* before enabling body capture in a regulated environment;
+  `capture_bodies` can be turned off while still recording who called what, with what result.
+- **Excluded by default:** health checks, the API docs, the ACME HTTP-01 challenge endpoint (it
+  returns `key_authorization`), the agent heartbeat (the highest-volume POST in the system), static
+  assets, and the log viewer's own endpoints. The list is editable, except the log viewer itself,
+  which is a hard floor so the page cannot end up logging you reading it.
+- **Disk growth is the main operational consideration.** On a busy install, in order of bluntness:
+  leave `capture_agent_success` off (the single biggest lever on a large fleet), lower `sample_rate`
+  (errors are always kept at 100 %), turn off `capture_get`, turn off `capture_bodies`, or shorten
+  `success_retention_days`.
+- **`operator` can see agent rows.** A caller holding only `requestlog.read` sees their own inbound
+  requests plus the fleet's, and nothing belonging to another user. Agent rows are included because
+  an apply fails on the *node* and the node reports it over its own API key — scoping to own-rows
+  only would have hidden the diagnosis from exactly the role the grant exists for. Anonymous traffic
+  (failed logins and the usernames they carry, unauthenticated probes) is **not** agent traffic and
+  remains visible only to `requestlog.manage`.
+- **New environment variables**, all optional: `REQUEST_LOG_ENABLED` (default `true`),
+  `REQUEST_LOG_QUEUE_MAX` (2000 rows), `REQUEST_LOG_QUEUE_MAX_BYTES` (64 MiB),
+  `REQUEST_LOG_BATCH_SIZE` (100), `REQUEST_LOG_FLUSH_MS` (500). See `.env.template`.
+  `REQUEST_LOG_QUEUE_MAX_BYTES` is a hard memory ceiling per worker: the row count alone does not
+  bound memory, because `max_body_bytes` is operator-editable up to 256 KB and a row can carry it
+  twice — at the ceiling the default 2 000-row queue would hold ~1 GiB, which is the whole pod
+  limit. Whichever limit binds first stops the queue. If you raise `REQUEST_LOG_QUEUE_MAX`, raise
+  this with it.
+- **`GET /api/request-logs/stats` sink counters are per worker**, labelled as such in the response.
+  With `UVICORN_WORKERS > 1` each process keeps its own queue and its own drop counter.
+- **Clearing the exclude-path list does not mean "log everything."** An empty list falls back to the
+  shipped defaults, so the log viewer and the raw-body agent heartbeat stay excluded; the UI now
+  says so and shows what was actually applied.
+- **To disable entirely:** set `REQUEST_LOG_ENABLED=false` in the backend environment and restart —
+  the middleware is then not registered at all and costs nothing, not even a settings lookup. The
+  `enabled` toggle in Settings is the no-restart equivalent (it takes effect immediately).
+- **Default admin password is not reset** by this bump; user seeding is guarded by an existence
+  check, not an upsert.
+- **Rollback:** downgrade the backend image freely. `request_logs` is purely additive and is simply
+  ignored by v1.10.x. Drop the table manually if you want the space back:
+  `DROP TABLE IF EXISTS request_logs;`
+
+---
+
+# Upgrade Notes — v1.10.14 (a converged node keeps acknowledging)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** a VIP shows `SYNCING (0/n)` with an empty *Last ack* even though every member node
+  has the rendered `keepalived.conf` on disk, keepalived is running and the VIP is held.
+- **Cause:** the deploy report was sent only when the agent actually wrote the config. Once the
+  node matched, it took the idempotency early return every cycle and never reported again, so any
+  report lost in transit was never retried and the server's view stayed stale permanently.
+- **Fix:** the agent re-asserts its state on the idempotent path as well. One request per node
+  per poll cycle (~2.5 min); nothing is written and keepalived is not reloaded.
+- **Recovery is automatic.** A VIP stuck at SYNCING converges on the first poll after the agents
+  pick up the new script. No action on the nodes, no re-apply, no edit to force a rewrite.
+- **This is not new in 1.10.12.** The gap dates from the original HA/VIP work; it only became
+  visible when acknowledgements were dropped for an unrelated reason.
+
+**Rollback:** safe. Reverting restores the previous behaviour, in which a lost acknowledgement is
+never recovered.
+
+---
+
+# Upgrade Notes — v1.10.13 (deploy acknowledgements were dropped)
+
+**Backend only, no schema change.** No `SCHEMA_VERSION` bump, no agent change. If you deployed
+v1.10.12, deploy this one too.
+
+- **Regression in v1.10.12.** The takeover-retirement clause added to the keepalived status
+  endpoint reused a query placeholder for both an assignment and a comparison. PostgreSQL types a
+  placeholder per use, so it was deduced as `text` in one place and `character varying` in the
+  other, and asyncpg refused the statement outright.
+- **Symptom:** a VIP stayed at `SYNCING (0/n)` with an empty *Last ack* even though the agent log
+  showed `applied config for VIP <id>` on every member. Teardown acknowledgements were lost the
+  same way, so a deletion never showed as complete.
+- **Nothing was damaged.** The failure was on the write of the acknowledgement, not on the node.
+  Configs were deployed correctly throughout; only the reporting was lost. Existing VIPs converge
+  on the next poll once this is deployed, with no action on the nodes.
+- **Verified against a real PostgreSQL**, not by inspection: both statements execute, a matching
+  hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash`
+  leave it in place, and the acknowledgement is recorded in every case.
+
+**Rollback:** do not roll back to v1.10.12; roll back to v1.10.11 instead, which predates the
+clause entirely.
+
+---
+
+# Upgrade Notes — v1.10.12 (valid config rejected by its own warning)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** applying a VIP left it stuck at `SYNCING`, the node kept its previous config and
+  the agent logged only `config validation failed (keepalived -t)`.
+- **Cause:** the agent treated any non-zero exit from `keepalived -t` as invalid. keepalived's
+  config-test exit code does not separate fatal from benign: on 2.2.8 a clean config exits 0,
+  while `Truncating auth_pass to 8 characters` exits 5 and so do a missing `}` and an unknown
+  keyword. A VRRP password longer than eight characters was enough to block every apply, even on
+  a node whose own running config emits the same warning.
+- **Fix:** the gate judges the output instead. Known-benign messages are dropped and anything
+  left still fails, so it fails closed. Verified against real keepalived: the truncation warning
+  passes; a missing brace, an unknown keyword and a `SECURITY VIOLATION` are refused.
+- **Also:** the agent now reports what keepalived actually said, in its log and in the status the
+  HA/VIP page shows. The refusal was correct but unactionable without reproducing it by hand.
+- **The fail-safe itself is unchanged:** a config that genuinely fails validation is never
+  written and keepalived is never restarted.
+
+**Rollback:** safe. Reverting restores the stricter gate, which rejects valid configs whose
+password exceeds eight characters.
+
+---
+
+# Upgrade Notes — v1.10.11 (Adoptable tag names the real blocker)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The *Adoptable* tag and the disabled *Adopt* button were derived separately, so they could
+  name different problems. A pair blocked by a peer whose config could not be parsed showed
+  **MASTER missing**, because the unreadable node's `state MASTER` had not been counted — true,
+  but it sent the operator to the wrong node. Both now come from one ordered decision.
+- New label **blocked by peer** for a group held up by a node that references the same address
+  but cannot be taken over with it. Two MASTERs is now distinct from none.
+- Display only. The endpoint's checks and refusals are unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.10 (Adoption blockers listed once per instance)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The adoption panel merged every member's blocker list, so a two-node pair showed each shared
+  problem twice. The two files report different line numbers for the same directive, so exact
+  de-duplication did not collapse them. Blockers are now merged on the message with the leading
+  `line N:` ignored.
+- Display only. The endpoint already evaluated the combined set across all nodes, and what it
+  accepts or refuses is unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.9 (Adoption refuses to strand a node)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent change.
+
+- **Adoption will not leave a node behind.** v1.10.8 resolved the whole VRRP instance, but only
+  from nodes it could parse, that were enabled and that were in the same pool. Anything else fell
+  out of the set silently while its peers were rewritten. Adoption now refuses if any reported
+  `keepalived.conf` mentions the virtual address and is not among the nodes being taken over, and
+  says which node and why.
+- **The nodes must agree on the shared fields.** `prefix_length`, unicast/multicast mode, HAProxy
+  tracking and the VRRP password live on the VIP and are re-rendered onto every member, so one
+  node's value used to be imposed on the rest. A disagreement is now refused with both values
+  shown.
+- **The takeover authorisation is now retired on acknowledgement.** It is the permission to
+  overwrite a `keepalived.conf` that does not carry our marker. It was never cleared, so it stayed
+  valid for that exact file content indefinitely; restoring the pre-adoption file would have been
+  overwritten again without fresh approval. It is now dropped once the member acks our rendered
+  config, gated on the acked hash matching `applied_config_hash` so a failed deploy cannot strand
+  the VIP.
+- **Nothing to do on upgrade.** Existing adopted VIPs keep working; their authorisation is retired
+  on the next successful acknowledgement.
+
+**Rollback:** safe. No schema or data migration; reverting restores the previous (more permissive)
+adoption checks.
+
+---
+
+# Upgrade Notes — v1.10.8 (VIP adoption takes the whole VRRP instance)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact: nothing about what the agent reports or how it takes a
+config over changes.
+
+- **Adoption is now per VRRP instance, not per node.** Every node in the pool reporting the same
+  `virtual_router_id` and virtual address becomes a member of one VIP, each with the role,
+  priority and interface its own `keepalived.conf` declares, and each with its own one-shot
+  takeover hash. The panel lists one row per instance.
+- **Why this mattered:** single-node adoption could not produce a working pair. The BACKUP alone
+  failed apply, the MASTER alone left the peer unmanaged and the peer could not then be adopted
+  (VRID collision). On a **unicast** instance it was worse than inconvenient: the render drops
+  the unicast block when there are no peers, so the adopted node fell back to multicast while its
+  peer stayed unicast and both could hold the address.
+- **New refusals, each with the reason in the message:** the group does not have exactly one
+  MASTER; the nodes disagree on `advert_int`; a declared unicast peer is not among the nodes being
+  adopted; a node is already a member of a live VIP.
+- **Apply Management "View Change" now renders the adopt diff correctly.** It did not recognise
+  the `adopt` action and fell through to the generic HAProxy diff, which compared the staged
+  `keepalived.conf` against the cluster's previous `haproxy.cfg` and showed the whole HAProxy
+  config as removed. Alarming, but display-only — nothing was ever applied from that view.
+- **Rejecting an adoption is recoverable again.** It used to hide the node from the panel
+  permanently. Nothing clears `vip_discoveries.adopted_vip_id`, a VIP is only soft-deleted so the
+  column's `ON DELETE SET NULL` never fires, and the agent does not re-report a file whose hash
+  has not changed. Adoptability is now derived from whether the linked VIP is still active.
+- **If you adopted a VIP on 1.10.4-1.10.7**, check it before applying: it may have only one
+  member. Add the peer from the VIP's edit form, or reject the pending adoption and adopt again —
+  the node reappears in the panel under this release.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous single-node
+adoption behaviour.
+
+---
+
+# Upgrade Notes — v1.10.7 (HA / VIP follows the selected cluster)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact.
+
+- **The HA / VIP page ignored the cluster picker.** Both the VIP table and the *Unmanaged
+  keepalived detected* panel queried the whole fleet, so on an install with more than one
+  cluster the lists never changed when the selection did. Both now pass `cluster_id`, mapped to
+  the cluster's pool the same way `GET /api/vip?cluster_id=` already worked for Apply
+  Management.
+- **Behaviour change worth knowing:** the VIP table is now scoped to the selected cluster. It
+  used to show every VIP in the fleet. If you relied on the fleet-wide view, the API still
+  supports it — `GET /api/vip` and `GET /api/vip/discoveries` without `cluster_id` return
+  everything, unchanged.
+- **API compatibility:** `cluster_id` is optional on both endpoints. Existing integrations that
+  do not send it behave exactly as before.
+
+**Rollback:** safe. The change is a query parameter plus the page that sends it; reverting
+restores the fleet-wide lists and touches no data.
+
+---
+
+# Upgrade Notes — v1.10.6 (VIP adoption panel was unreachable)
+
+**One backend fix, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No API-shape change, no frontend change and zero agent impact.
+
+- **v1.10.4's adoption panel never appeared.** `GET /discoveries` was declared after
+  `GET /{vip_id}` in `routers/vip.py`. FastAPI matches routes in declaration order, so the
+  discovery list was routed into the get-one-VIP handler, which declares `vip_id: int` and
+  answered **422** before the real handler ran. The HA/VIP page treats any non-OK response as
+  "nothing to show", so the feature was invisible with no error in any log.
+- **Nothing was lost.** The agent side always worked: discoveries were reported and stored in
+  `vip_discoveries`. Deploy this backend and the rows appear immediately — no agent upgrade, no
+  re-sync of the agent script, no re-report needed.
+- **If you are upgrading straight from 1.10.3 or earlier**, follow the v1.10.4 notes below as
+  well: that release does bump `SCHEMA_VERSION` (10 → 11), which re-seeds the four built-in
+  roles, and its agent script has to reach the nodes before discovery starts.
+- **Regression guard.** A static source scan now fails the build if any literal API path in any
+  router is declared after a parameterised route that would swallow it. The whole router tree is
+  clean as of this release.
+
+**Rollback:** safe and immediate. The change is a route declaration order plus a test; reverting
+to 1.10.5 restores the previous (broken-panel) behaviour and touches no data.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.5 (HTTP-01 challenge backend on split deployments)
+
+**Bug fixes, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are **not**
+re-seeded. No API-shape change and zero agent impact.
+
+- **HTTP-01 could fail silently when HAProxy runs on different hosts than the management stack.**
+  The rendered config wrote `server _acme_mgmt <mgmt>:8080` from a value that defaults to
+  loopback — and HAProxy resolves that address **on the HAProxy node**, so it pointed at the wrong
+  box. Every check still reported success. The per-cluster `acme_backend_url` now has a UI field
+  (Cluster Management), changing it actually mints a config version, and the value is validated at
+  the write boundary.
+- **A config-generation failure could be pushed to agents as the cluster's whole `haproxy.cfg`.**
+  The generator reported failure by *returning* `# Error ...` instead of raising, and the apply
+  path hashed that comment and stored it as an APPLIED version. Both persisting call sites now
+  refuse with 422 and leave the running config in force. **This is worth knowing even if you never
+  touch ACME**, since any exception in the generator could trigger it.
+- **`frontends.mode` is nullable and was interpolated raw**, emitting a literal `mode None` that
+  HAProxy rejects — which fails the whole cluster config, not just that frontend. Normalised now.
+- **Cluster creation ignored the ACME fields**: a cluster created with ACME switched on came back
+  switched off, with no error.
+- **`docker-compose.yml` hardcoded `PUBLIC_URL` / `MANAGEMENT_BASE_URL`**, so a value in your
+  `.env` or host environment was silently ignored. They are interpolated now, with the previous
+  literals as defaults, so behaviour is unchanged unless you actually set them.
+- **Diagnostics stop over-reporting health.** The port-80 check now reads the body, so a reverse
+  proxy answering 200 with a web page is no longer counted as a working challenge endpoint. Every
+  new condition is a **warning, never a failure** — the Site Wizard blocks submit on a failing
+  check, so a new failing condition would have locked installs on upgrade day.
+- **Rollback:** downgrade freely. No schema or data change.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.4 (Adopt an existing keepalived VIP)
+
+**Additive, but this release DOES bump the schema — read the role warning below.** Nothing on any
+node changes until you adopt a VIP and apply it.
+
+- **Schema:** `SCHEMA_VERSION` bumps to `11`, so on first start the (idempotent) migration
+  sequence re-runs once and adds **one new table** (`vip_discoveries`) plus two additive columns
+  (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`). **No existing table is
+  altered**, no existing row changes, and the admin password is not reset.
+- **⚠️ Built-in roles are re-seeded to their defaults** — the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump. If you customised `super_admin` / `operator` / `security_admin` /
+  `viewer`, **re-apply those changes after upgrading**. (The three previous releases did not bump
+  the version, so this is the first re-seed since v1.9.0.) No new permission strings are
+  introduced: discovery and adoption are governed by the existing `vip.read` / `vip.create`.
+- **⚠️ The Linux agent script changed, and discovery does not start until nodes run it.** The
+  fallback latest Linux agent version moves `2.0.0` → `2.1.0`, so nodes will pull the new script
+  through the normal agent-upgrade path. The addition is **read-only**: the agent reads the
+  `keepalived.conf` it does not own and reports it, rate-limited to once per content change. It
+  writes nothing new to the node. Until a node has upgraded, it simply never appears under
+  *Unmanaged keepalived detected*.
+- **Nothing is taken over implicitly.** The agent still refuses to overwrite a `keepalived.conf`
+  that lacks OpenManager's ownership marker. Adoption authorises exactly **one** takeover of
+  exactly the file that was analysed, pinned to its md5: if the file changes between adoption and
+  Apply, the agent refuses again and reports `externally_managed` rather than clobbering your
+  edit. Re-adopt to pick up the current file.
+- **Adoption can refuse, on purpose.** It replaces the file with OpenManager's render, so anything
+  the renderer cannot reproduce would be destroyed. Those directives are listed as blockers —
+  `notify_*` failover hooks, `vrrp_sync_group`, LVS `virtual_server` sections, a second address in
+  one instance, a custom `track_script`, extra `global_defs`. You can accept that loss explicitly
+  with a tick, but a value that is *unknown* rather than lost (an absent `virtual_router_id`, or
+  an address with no prefix length) cannot be waived — the VRID is fatal to guess and the prefix
+  has to be supplied, because picking a netmask for a live VIP would change its routing.
+- **Multi-node VIPs need every node.** Adoption covers the node that reported. Its unicast peers
+  hold their own `keepalived.conf`, so adopt or add them as members before applying — otherwise
+  the render has no peers. The UI says so after a successful adopt.
+- **Secrets:** the reported config may contain the VRRP `auth_pass`. It is split at ingest — the
+  password is Fernet-encrypted into its own column (same key path as `vip_instances`,
+  `VIP_ENCRYPTION_KEY` falling back to a key derived from `SECRET_KEY`) and the stored copy of the
+  file has it masked, so nothing readable through the API, the UI preview or a DB dump carries it
+  in cleartext.
+- **Rollback:** downgrading to 1.10.3 leaves `vip_discoveries` as an unused table and the two new
+  columns unread; managed VIPs keep working. One caveat: a VIP adopted on 1.10.4 but **not yet
+  applied** loses its takeover authorisation on downgrade, so the node's original config stays in
+  place and the VIP sits PENDING — harmless, but re-adopt after upgrading again. Agents already on
+  script 2.1.0 keep reporting discoveries to an endpoint that no longer exists; the report fails
+  quietly and nothing on the node is affected.
+
+---
+
+# Upgrade Notes — v1.10.3 (Multi-account ACME wizard fix)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change, no
+environment variable, zero agent impact. Installations with a single ACME account behave exactly as
+before.
+
+- **What was broken:** with **more than one** ACME account registered, the *Request ACME
+  Certificate* wizard did not honour the account you selected. Choosing an HTTP-01 account still
+  submitted a DNS-01 request, which the API rejected with
+  `The selected ACME account has no DNS provider configured for DNS-01.` The *Review* step also
+  named the default account rather than the chosen one, so the mismatch was invisible before
+  submitting.
+- **Default account:** the wizard previously previewed the **oldest** valid account while the
+  backend uses the **newest** (`ORDER BY created_at DESC`). If you never picked an account
+  explicitly and have several, requests were already going to the newest one — only the preview was
+  wrong. The wizard now previews that same account, marks it `(default)`, and sends `account_id`
+  explicitly so the two can no longer diverge.
+- **Wildcard guard:** the client-side "wildcard requires a DNS-01 account" block silently stopped
+  applying on the *Review* step. Requests were still rejected by the backend, so nothing incorrect
+  was ever issued — you now get the warning before submitting instead of an error after.
+- **No action needed on existing certificates or orders.** Nothing about issuance, renewal or the
+  stored accounts changes; only how the wizard resolves which account a new request uses.
+- **Rollback:** downgrade freely. This release changes frontend behaviour only.
+
+---
+
+# Upgrade Notes — v1.10.2 (Dark mode fixes on Apply Management)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change,
+no environment variable, zero agent impact. Light mode is byte-identical: every colour swapped in
+this release resolves, under the default algorithm, to exactly the literal it replaced
+(`colorWarningBg` → `#fffbe6`, `colorSuccessBg` → `#f6ffed`, `colorErrorBg` → `#fff2f0`, …), so
+only dark mode changes.
+
+- **Apply Management panels** were painted with light-mode colour literals, so in dark mode the
+  "Pending Changes" box rendered as a cream panel with light text on it. Measured contrast was
+  **1.03:1** — effectively invisible. It is now **11.50:1**. The same class of bug affected the
+  diff rows in *View Change* (2.21:1 and 2.99:1, now 5.49:1 and 4.01:1), the ACME/pending version
+  panels, the VIP pending-delete row and the agent-error recommendation box.
+- **Static confirm dialogs came up white in dark mode.** In Ant Design 5 the static
+  `Modal.confirm` / `message` / `notification` APIs render into their own detached root and never
+  see the app's `ConfigProvider`, so they always used the light algorithm. This release registers
+  `ConfigProvider.config({ holderRender })` once at the app root, which fixes **every** static
+  dialog in the app (12 components use them), not just Apply Management.
+- **Rollback:** downgrade freely. This release changes rendering only.
+
+---
+
+# Upgrade Notes — v1.10.1 (CSR private key encrypted at rest)
+
+**Backward compatible.** Nothing to do on upgrade, and nothing changes for existing clusters,
+agents or certificates:
+
+- **Schema:** **no `SCHEMA_VERSION` bump and no migration.** The Fernet token replaces the PEM
+  inside the *existing* `ssl_csrs.private_key_pem` TEXT column. As in v1.10.0, this means the
+  four built-in roles are **not** re-seeded, so any customization of `super_admin` / `operator` /
+  `security_admin` / `viewer` survives.
+- **Existing pending CSRs keep working.** Rows written before this release hold a raw PEM and are
+  still read transparently, so a CSR that is already out for signature can be imported normally
+  after the upgrade. There is no data migration and no downtime step. Those rows stay plaintext
+  until they are imported (which NULLs the key) — if you want everything encrypted immediately,
+  delete and re-create any long-pending CSRs.
+- **Scope:** this covers the PENDING CSR key only. It is the one key in the system that sits idle
+  for the whole signing window and is never transmitted. `ssl_certificates.private_key_content`
+  and the ACME order keys are unchanged, because agents must receive those in plaintext on every
+  poll.
+- **Optional env:** `CSR_ENCRYPTION_KEY` (see `.env.template`). If unset, the key is derived from
+  `SECRET_KEY` via HKDF with its own info string, so it is independent of the VIP, MFA and DNS
+  provider keys.
+  - **⚠️ Rotating `SECRET_KEY` while `CSR_ENCRYPTION_KEY` is unset makes pending CSR keys
+    unrecoverable.** Import then fails with an explicit "delete this CSR and create a new one"
+    error rather than a misleading key-mismatch. Set an explicit `CSR_ENCRYPTION_KEY` if you
+    rotate `SECRET_KEY`. Certificates already imported are unaffected — their key lives on the
+    certificate row.
+- **API / UI / agents:** unchanged. No CSR endpoint ever returned the private key before or now,
+  and nothing about the CSR tab changes.
+- **Rollback:** the application downgrades cleanly — 1.10.0 starts normally against the same
+  database and every other feature is unaffected. The one casualty is a CSR **created on 1.10.1
+  and still pending**: 1.10.0 has no decrypt step, so it hands the Fernet token straight to the
+  key-pairing check. Measured on a real downgrade, the import then fails with
+  `HTTP 500 — Could not verify the certificate/key pair: key parse failed (encrypted?)`; it does
+  **not** silently pair the wrong key, and it does not corrupt anything. Import or delete CSRs
+  created on 1.10.1 before downgrading. Certificates already imported are unaffected, since their
+  key lives on the certificate row, and CSRs created before 1.10.1 are plaintext and still work.
+
+---
+
+# Upgrade Notes — v1.10.0 (GoDaddy DNS-01 provider)
+
+**Backward compatible & additive.** Nothing changes unless you select **GoDaddy** as an ACME
+account's DNS provider:
+
+- **Schema:** **no `SCHEMA_VERSION` bump.** The GoDaddy credentials (API Key + Secret) are stored
+  as two keys inside the *existing* encrypted
+  `letsencrypt_account_dns_credentials.credentials_encrypted` blob — no new table, no new column,
+  no migration.
+- **✅ Built-in roles are NOT re-seeded.** The re-seed warning in the v1.9.0 notes below is
+  triggered by a `SCHEMA_VERSION` bump. This release does not bump it, so any customization you
+  made to `super_admin` / `operator` / `security_admin` / `viewer` survives untouched.
+- **Permissions / API shape:** unchanged. `GET /api/letsencrypt/dns-providers` simply returns one
+  extra entry in its `providers` array; every request and response shape is identical, and the
+  credential form is rendered from that schema, so there is no frontend behaviour change either.
+- **Environment:** no new variable. GoDaddy credentials use the same Fernet-at-rest path as
+  Cloudflare (`DNS_PROVIDER_ENCRYPTION_KEY`, falling back to a key derived from `SECRET_KEY`).
+- **Agents:** zero agent changes. DNS-01 is invisible to agents; an issued certificate follows the
+  normal PENDING → Apply Management → agent pull pipeline exactly as before.
+- **Using it:** the API Key must be a **Production** key from `developer.godaddy.com/keys` (the
+  first key that dashboard issues is an OTE/test key and is rejected), the zone must be in the same
+  GoDaddy account, and that account needs at least one registered domain before GoDaddy permits DNS
+  API access. A Personal Access Token also works — paste it as the Key and leave the Secret blank.
+  Credentials are checked against the GoDaddy API before they are stored, so an invalid, OTE or
+  ineligible key fails at save time. Note the check is a **read**: a Personal Access Token that has
+  `domains.domain:read` but not `domains.dns:update` saves successfully and only fails at the first
+  publish, with a 403 in the order timeline.
+- **Rollback:** don't select GoDaddy. Existing Manual and Cloudflare accounts and all HTTP-01
+  issuance are untouched. **Downgrading after adopting GoDaddy is not a no-op**: on 1.9.0
+  `godaddy` is not a known provider, so any account still set to it degrades to the manual-confirm
+  path (in-flight DNS-01 orders wait for a confirmation nobody can give and expire after 48h, and
+  renewals stop), and the cleanup sweep marks published TXT records cleaned without removing them.
+  Before downgrading, switch affected accounts back to Manual or Cloudflare and let the reconcile
+  sweep remove outstanding `_acme-challenge` records first. The stored credential row itself is
+  inert — an encrypted blob for an unknown provider.
+
+---
+
+# Upgrade Notes — v1.9.0 (CSR creation)
+
+**Backward compatible & additive.** Upgrading to v1.9.0 changes nothing for existing
+clusters/agents until you create a CSR:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `10`, so on first start the (idempotent)
+  migration sequence re-runs once and adds **one new table** (`ssl_csrs`) plus its
+  indexes. **No existing table is altered**, existing rows are untouched, and the
+  **admin password is not reset** (the default-user seeding is guarded by an
+  existence check, not an upsert). No new permission strings are introduced — all
+  CSR endpoints are governed by the existing `ssl.create` / `ssl.read` /
+  `ssl.delete` permissions.
+- **⚠️ Built-in roles are re-seeded to their defaults (pre-existing behaviour of
+  every `SCHEMA_VERSION` bump — verified in a v1.8.10 → v1.9.0 upgrade drill).**
+  Because the version gate re-runs the whole sequence, `update_system_roles_to_enterprise_rbac()`
+  issues an unconditional `UPDATE roles SET … permissions = <defaults> WHERE name = …`
+  for the four **built-in** roles (`super_admin`, `operator`, `security_admin`,
+  `viewer`). **Any customization you made to a built-in role is reverted.** In the
+  drill, an `operator` role that had been narrowed by removing `apply.execute` and
+  `config.bulk_import` came back with both restored (57 → 59 permissions).
+  - **Roles you created yourself are NOT affected** — the re-seed matches on the four
+    built-in names only.
+  - This is not new in v1.9.0: it happens on every release that bumps
+    `SCHEMA_VERSION` (v1.7.0, v1.8.0, v1.8.8 …). It is documented as intentional at
+    `backend/database/migrations.py` (the "BUMP THIS … OR seeded/role data" note) —
+    the migration is treated as the authority on built-in-role contents.
+  - **If you have hardened a built-in role, do this:** export it before upgrading
+    (`GET /api/roles`), then re-apply your changes after the first start
+    (`PUT /api/roles/{id}`) — or, preferably, move your customization into a
+    purpose-made custom role, which survives every upgrade.
+- **Key storage:** CSR private keys are stored in the database like every other key
+  in the system (`ssl_certificates.private_key_content` and the ACME order keys).
+  The key is never returned by any CSR API endpoint, and after a successful import
+  the CSR row's key copy is set to NULL (the key then lives only on the certificate
+  row).
+- **Agents:** zero agent changes. Agents never read the new table; a CSR becomes
+  visible to agents only after its signed certificate is imported **and** applied via
+  Apply Management (the standard PENDING pipeline).
+- **Rollback:** simply don't use the CSR tab. The `ssl_csrs` table is inert when
+  empty; downgrading the application leaves it as an ignored extra table.
+
+---
+
+# Upgrade Notes — v1.7.0 (HA / VIP Keepalived management, Issue #27)
+
+**Backward compatible & opt-in.** Upgrading to v1.7.0 changes nothing for existing
+clusters/agents until you create a VIP:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `3`, so on first start the (idempotent)
+  migration sequence re-runs once and adds two **new** tables (`vip_instances`,
+  `vip_members`) plus an additive `vip_instances.applied_snapshot` column (enables
+  rejecting a pending VIP change and restoring the previous applied state). No existing
+  table is altered. Existing rows and the **admin password
+  are not reset** (default users are create-if-missing). The only data effect is that
+  the **four built-in system roles** (`super_admin`/`operator`/`security_admin`/`viewer`)
+  are re-seeded to their canonical permission sets **plus** the new `vip.*` permissions —
+  this is the long-standing behavior of the role seeder; **custom roles are untouched**.
+- **Agents:** the agent script gains an opt-in keepalived deploy that is a **no-op** on
+  any node without an applied VIP, and it **never overwrites a hand-managed
+  `/etc/keepalived/keepalived.conf`** (it reports "externally managed" instead).
+- **Scope:** VRRP VIPs target bare-metal / VMware / on-prem L2 networks. On AWS/Azure/GCP
+  the cloud fabric doesn't honor VRRP/gratuitous-ARP; the UI surfaces this. Ensure host
+  firewalls permit VRRP (IP protocol 112).
+- **Optional env:** `VIP_ENCRYPTION_KEY` (see `.env.template`) — if unset, the VRRP secret
+  encryption key is derived from `SECRET_KEY` (like MFA).
+
+No rollback steps are required to *disable* the feature: simply don't create VIPs (or
+delete them — agents tear down their managed keepalived on the next poll).
+
+---
+
 # Agent Upgrade Guide - Dashboard Stats Fix
 
 ## Problem

backend/Dockerfile

@@ -37,5 +37,13 @@ USER appuser
 # Expose port
 EXPOSE 8000
 
-# Run the application in production mode (without --reload)
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+# Run the application in production mode (without --reload).
+# UVICORN_WORKERS (default 1) opts into multiple worker processes on multi-core
+# hosts; with 1 worker uvicorn runs in-process, identical to the flagless CMD
+# this replaces. Falls back to WEB_CONCURRENCY when UVICORN_WORKERS is unset
+# because flagless uvicorn honored WEB_CONCURRENCY (uvicorn config.py) — this
+# keeps any deployment that relied on it byte-for-byte compatible. Background
+# tasks are multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as
+# already exercised by the k8s HPA deployment. `exec` keeps uvicorn as PID 1
+# so signal handling is unchanged.
+CMD ["sh", "-c", "exec uvicorn main:app --host 0.0.0.0 --port 8000 --workers ${UVICORN_WORKERS:-${WEB_CONCURRENCY:-1}}"]

backend/agent_notifications.py

@@ -25,31 +25,42 @@ async def notify_agents_config_change(cluster_id: int, version_name: str) -> Lis
         for agent in agents:
             try:
                 agent_url = f"http://{agent['ip_address']}:8081"  # Agent default port
-                
+
                 async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
                     payload = {
                         "cluster_id": cluster_id,
                         "version_name": version_name,
                         "action": "config_update"
                     }
-                    
-                    async with session.post(f"{agent_url}/api/config/update", json=payload) as response:
-                        if response.status == 200:
-                            results.append({
-                                'node': agent['name'],
-                                'success': True,
-                                'message': f'Configuration updated successfully',
-                                'version': version_name
-                            })
-                            logger.info(f"✅ Agent {agent['name']} notified successfully")
-                        else:
-                            error_text = await response.text()
-                            results.append({
-                                'node': agent['name'],
-                                'success': False,
-                                'error': f'HTTP {response.status}: {error_text}'
-                            })
-                            logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
+
+                    # v1.11.0: instrumented so the code stays correct if the push
+                    # architecture is ever reverted. Unreachable today — see the
+                    # unconditional early return above.
+                    from utils.http_instrumentation import outbound_span, TARGET_AGENT
+
+                    push_url = f"{agent_url}/api/config/update"
+                    async with outbound_span(
+                        target=TARGET_AGENT, method="POST", url=push_url, request_body=payload
+                    ) as span:
+                        async with session.post(push_url, json=payload) as response:
+                            if response.status == 200:
+                                span.set_response(response.status, getattr(response, "headers", None))
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': True,
+                                    'message': f'Configuration updated successfully',
+                                    'version': version_name
+                                })
+                                logger.info(f"✅ Agent {agent['name']} notified successfully")
+                            else:
+                                error_text = await response.text()
+                                span.set_response(response.status, getattr(response, "headers", None), error_text)
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': False,
+                                    'error': f'HTTP {response.status}: {error_text}'
+                                })
+                                logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
                             
             except asyncio.TimeoutError:
                 results.append({

backend/auth_middleware.py

@@ -1,4 +1,4 @@
-from fastapi import HTTPException, status
+from fastapi import HTTPException, status, Header
 from typing import Optional, Dict, Any
 from jose import jwt
 import logging
@@ -92,6 +92,18 @@ async def get_current_user_from_token(authorization: Optional[str] = None) -> Op
             detail="Authentication failed"
         )
 
+async def require_authenticated_user(authorization: Optional[str] = Header(None)) -> Dict[str, Any]:
+    """FastAPI dependency: require a valid operator JWT, else 401.
+
+    Reads the Authorization header itself, so it can be attached at router or
+    route level to gate operator/UI endpoints that must not be public:
+        APIRouter(..., dependencies=[Depends(require_authenticated_user)])
+        @router.get(..., dependencies=[Depends(require_authenticated_user)])
+    Any authenticated user passes (no fine-grained RBAC here) — this restores the
+    pre-existing "logged-in users only" expectation without changing role access.
+    """
+    return await get_current_user_from_token(authorization)
+
 async def get_current_user_from_token_no_exception(authorization: Optional[str] = None) -> Optional[Dict[str, Any]]:
     """
     Get current user from JWT token without raising HTTPException.

backend/config.py

@@ -37,4 +37,65 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS = 30
 
 # Entity snapshot enabled by default (rollback functionality)
 # Set to "false" only if you need to disable snapshot temporarily
-ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true" 
+ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true"
+
+
+# ---------------------------------------------------------------------------
+# v1.11.0 — unified request/response log
+# ---------------------------------------------------------------------------
+# These four are deliberately ENV-only (not database settings): they decide
+# whether the middleware is even registered and how much memory the writer
+# queue may hold, so they must be resolvable before the DB pool exists.
+# Everything the operator tunes at runtime (retention, body capture, sampling,
+# excluded paths) lives in `system_settings` under the `requestlog.` category
+# and is editable from Settings → Request Log.
+
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() not in ("0", "false", "no", "off", "")
+
+
+def _int_env(name: str, default: int, minimum: int, maximum: int) -> int:
+    """Read an int env var, clamped. A malformed value falls back to the
+    default rather than crashing the process at import time."""
+    raw = os.getenv(name)
+    if raw is None or not raw.strip():
+        return default
+    try:
+        value = int(raw.strip())
+    except (TypeError, ValueError):
+        return default
+    return max(minimum, min(maximum, value))
+
+
+# Hard kill-switch. When false the logging middleware is never added to the
+# ASGI stack and neither the writer nor the prune task is started — literally
+# zero overhead, not even a settings lookup.
+REQUEST_LOG_ENABLED = _bool_env("REQUEST_LOG_ENABLED", True)
+# Per-worker in-process queue depth. When full, rows are DROPPED (counted, and
+# reported through GET /api/request-logs/stats) — the request path never blocks
+# on the database.
+REQUEST_LOG_QUEUE_MAX = _int_env("REQUEST_LOG_QUEUE_MAX", 2000, 100, 100000)
+# HARD memory ceiling for the same queue, per worker. The row count above does
+# NOT bound memory on its own, because how much a row weighs is an operator
+# setting: `requestlog.max_body_bytes` is editable from Settings and its stated
+# ceiling is 256 KB, which a row can carry twice (request + response). Measured
+# on the real dataclass, the two limits multiply out to:
+#
+#     defaults (2 000 rows x 8 KB)          33.9 MiB   3.3% of the 1 GiB pod limit
+#     max_body_bytes at its 256 KB ceiling   1003 MiB   at the pod limit
+#     REQUEST_LOG_QUEUE_MAX at its ceiling   1695 MiB   over the pod limit
+#
+# Both of those are reachable from documented, in-range values, and the drop
+# warning used to advise raising the queue - so following the tool's own advice
+# could OOM the worker. Whichever limit is hit FIRST now stops the queue, so
+# memory stays bounded no matter what the other is set to.
+REQUEST_LOG_QUEUE_MAX_BYTES = _int_env(
+    "REQUEST_LOG_QUEUE_MAX_BYTES", 64 * 1024 * 1024, 1024 * 1024, 1024 * 1024 * 1024
+)
+# Rows per batched INSERT: one pool acquire per batch, not per request.
+REQUEST_LOG_BATCH_SIZE = _int_env("REQUEST_LOG_BATCH_SIZE", 100, 1, 1000)
+# Max wait before a partial batch is flushed (milliseconds).
+REQUEST_LOG_FLUSH_MS = _int_env("REQUEST_LOG_FLUSH_MS", 500, 50, 10000)

backend/database/migrations.py

@@ -50,9 +50,62 @@ async def ensure_agents_table():
         await conn.execute("ALTER TYPE config_status ADD VALUE IF NOT EXISTS 'DELETION';")
         logger.info("Ensured REJECTED and DELETION values exist in config_status enum.")
 
-        # First, create essential tables if they don't exist
-        await create_essential_tables(conn)
-        
+        # First, create essential tables if they don't exist.
+        #
+        # Rolling-restart resilience: create_essential_tables() runs idempotent
+        # CREATE ... IF NOT EXISTS statements on every startup. Its CREATE INDEX
+        # block needs a SHARE lock that conflicts with concurrent writes (e.g.
+        # agent heartbeats updating backend_servers/agents). During a redeploy a
+        # writer can hold that lock, so the DDL blocked for the full 60s
+        # command_timeout -> TimeoutError -> startup crash -> crash-loop.
+        #
+        # Fix: fail fast on locks (lock_timeout), retry briefly, and on
+        # persistent contention SKIP the idempotent bootstrap and continue — on
+        # an established DB the objects already exist; a fresh DB has no writers
+        # so the first attempt always succeeds. lock_timeout is scoped to this
+        # call and RESET afterwards, so every other migration below keeps its
+        # original (wait-indefinitely) behavior. Non-lock errors still propagate
+        # (genuine schema problems must NOT be masked).
+        import asyncio as _asyncio
+        _lock_excs = (_asyncio.TimeoutError,)
+        try:
+            import asyncpg as _asyncpg
+            _lock_excs = _lock_excs + (
+                _asyncpg.exceptions.LockNotAvailableError,
+                _asyncpg.exceptions.QueryCanceledError,
+            )
+        except Exception:
+            pass
+        try:
+            await conn.execute("SET lock_timeout = '10s'")
+        except Exception:
+            pass
+        try:
+            for _attempt in range(1, 4):
+                try:
+                    await create_essential_tables(conn)
+                    break
+                except _lock_excs as _lock_err:
+                    if _attempt < 3:
+                        logger.warning(
+                            f"create_essential_tables: lock contention "
+                            f"(attempt {_attempt}/3), retrying in 3s "
+                            f"({type(_lock_err).__name__})"
+                        )
+                        await _asyncio.sleep(3)
+                    else:
+                        logger.warning(
+                            "create_essential_tables: persistent lock contention; "
+                            "skipping idempotent schema bootstrap and continuing "
+                            "startup (objects already exist on an established DB) "
+                            f"({type(_lock_err).__name__})"
+                        )
+        finally:
+            try:
+                await conn.execute("RESET lock_timeout")
+            except Exception:
+                pass
+
         # Ensure status column exists in config_versions table
         status_column_exists = await conn.fetchval("""
             SELECT 1 FROM information_schema.columns 
@@ -141,7 +194,14 @@ async def ensure_agents_table():
             'use_backend_rules': "ALTER TABLE frontends ADD COLUMN use_backend_rules JSONB DEFAULT '[]'::jsonb;",
             'request_headers': "ALTER TABLE frontends ADD COLUMN request_headers TEXT;",
             'response_headers': "ALTER TABLE frontends ADD COLUMN response_headers TEXT;",
-            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;"
+            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;",
+            # Issue #38: SPOE filter directives (e.g. Coraza WAF) and frontend
+            # log-format were silently dropped on bulk-import / manual edit
+            # because the parser recognised only a fixed set of directives.
+            # These nullable TEXT columns persist them verbatim (multi-line for
+            # `filters`), mirroring the request_headers/options passthrough.
+            'log_format': "ALTER TABLE frontends ADD COLUMN log_format TEXT;",
+            'filters': "ALTER TABLE frontends ADD COLUMN filters TEXT;"
         }
         
         for col, query in frontend_columns.items():
@@ -431,6 +491,7 @@ async def ensure_agents_table():
                 connection_type VARCHAR(50) DEFAULT 'agent',
                 stats_socket_path VARCHAR(500) DEFAULT '/run/haproxy/admin.sock',
                 haproxy_config_path VARCHAR(500) DEFAULT '/etc/haproxy/haproxy.cfg',
+                keepalived_config_path VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf',
                 pool_id INTEGER REFERENCES haproxy_cluster_pools(id) ON DELETE SET NULL,
                 haproxy_user VARCHAR(255) DEFAULT 'haproxy',
                 haproxy_group VARCHAR(255) DEFAULT 'haproxy',
@@ -1298,11 +1359,16 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk', 'apply.emergency',
                     'agents.read', 'agents.create', 'agents.update', 'agents.delete', 'agents.script', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.create', 'clusters.update', 'clusters.delete', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.upload', 'config.backup', 'config.restore', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'users.read', 'users.create', 'users.update', 'users.delete', 'users.password', 'users.roles',
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health', 'statistics.export',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — request/response log. `read` browses the log,
+                    # `manage` edits retention/capture settings and triggers a
+                    # manual purge.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.update', 'settings.system', 'settings.security',
                     'system.restart', 'system.logs', 'system.database', 'system.services', 'system.emergency'
                 ]
@@ -1319,9 +1385,14 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk',
                     'agents.read', 'agents.update', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
-                    'activity.read'
+                    'activity.read',
+                    # v1.11.0 — operators debug failing applies and ACME orders,
+                    # so they get read access to the request log; retention and
+                    # purge stay with the admins.
+                    'requestlog.read'
                 ]
             },
             'security_admin': {
@@ -1336,12 +1407,20 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history',
                     'agents.read', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — the request log is a security-forensics surface,
+                    # so the security admin gets both read and retention control.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.security'
                 ]
             },
+            # NOTE (v1.11.0): `viewer` deliberately gets NEITHER requestlog
+            # permission. Even redacted, captured request/response bodies are a
+            # far broader disclosure surface than the read-only configuration
+            # views a viewer is meant to have.
             'viewer': {
                 'display_name': 'Viewer',
                 'description': 'Read-only access to view configurations, statistics, and monitor system status',
@@ -1354,6 +1433,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.history',
                     'agents.read',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read',
@@ -1639,13 +1719,159 @@ async def ensure_agent_activity_logs_table():
             await close_database_connection(conn)
         # Don't raise - this is not critical for system operation
 
+# Schema-version gate for the migration runner.
+#
+# >>> BUMP THIS whenever you add/modify ANY step in _run_all_migrations_inner()
+# >>> that changes the schema (table/column/index/constraint) OR seeded/role data
+# >>> (e.g. update_system_roles_to_enterprise_rbac). Otherwise the new step will
+# >>> NOT run on databases already marked at the current version.
+#
+# When the DB already records >= this version, run_all_migrations() skips the
+# whole (lock-heavy) idempotent sequence, so redeploys/scale-ups issue NO DDL and
+# a concurrently-serving replica's traffic cannot block ALTER / CREATE INDEX (the
+# rolling-deploy startup crash that motivated this gate).
+#
+# Backward compatibility (the product runs at many versions across companies):
+#  - First start on this code: no marker -> applied_version is NULL -> the FULL
+#    sequence runs (upgrades any prior version), THEN the marker is written. So
+#    upgrading from any older version is unaffected.
+#  - The marker is written ONLY after _run_all_migrations_inner() completes with
+#    no exception, so an interrupted/failed migration never marks an incomplete
+#    schema as done — the next start retries.
+#  - Behavior change vs the historical "re-run every idempotent ensure_* on every
+#    start": once marked, same-version restarts no longer re-run (and therefore no
+#    longer auto-repair manual drift). To force a re-run, bump SCHEMA_VERSION or
+#    delete the schema_migrations row.
+#
+# v1.7.0 (Issue #27 — HA/VIP Keepalived management): bumped 1 -> 2 so the new
+# additive ensure_vip_tables() step (two brand-new tables) actually runs on
+# databases already marked at version 1. The whole re-run is idempotent.
+# v1.7.0 self-review: bumped 2 -> 3 so the additive `applied_snapshot` column on
+# vip_instances (enables VIP reject/restore-to-previous) lands on DBs marked at 2.
+# v1.7.0 self-review: bumped 3 -> 4 for the additive `keepalived_config_path` column on
+# haproxy_clusters (cluster-driven keepalived.conf path, like haproxy_config_path).
+# v1.7.0 self-review: bumped 4 -> 5 to drop the table-level UNIQUE on vip_instances.name
+# and replace it with a partial unique index (active rows only), so a soft-deleted VIP's
+# name is reusable — consistent with the address/VRID partial indexes. Idempotent re-run.
+# v1.7.2: bumped 5 -> 6 for the additive `purge_on_teardown` column on vip_instances
+# (opt-in "also uninstall the keepalived package on delete"; default FALSE keeps the safe
+# graceful-teardown behaviour). Additive + idempotent.
+# v1.7.2: bumped 6 -> 7 for the additive `pending_delete` column on vip_instances
+# (approval-gated VIP deletion: a delete is staged for Apply Management and the VIP keeps
+# running until APPROVED, so an agent never tears down without explicit human approval).
+# v1.8.0 (Issue #35 — ACME DNS-01 challenge support): bumped 7 -> 8 for additive DNS-01
+# columns on letsencrypt_accounts/letsencrypt_orders/acme_challenges and the brand-new
+# letsencrypt_account_dns_credentials table (ensure_letsencrypt_dns_credentials step).
+# All additive + idempotent; default challenge_type 'http-01' keeps existing flows byte-identical.
+# v1.8.8 (Issue #38 — SPOE filter + frontend log-format): bumped 8 -> 9 for the additive
+# `log_format` + `filters` TEXT columns on `frontends` (frontend_columns loop). Without this
+# bump, already-deployed databases (version >= 8) skip the whole migration run and never gain
+# the columns, so the frontends SELECT/INSERT would fail. Additive + idempotent + nullable;
+# existing rows stay NULL and render byte-identical.
+# v1.9.0 (CSR creation): bumped 9 -> 10 for the brand-new `ssl_csrs` table
+# (ensure_ssl_csrs_table step). Holds a locally generated private key + CSR PEM
+# until the operator imports the CA-signed certificate; the import creates a
+# normal ssl_certificates row and NULLs the key copy here. Additive + idempotent;
+# no existing table is altered, agents never read this table.
+# v1.10.4 (VIP adoption): bumped 10 -> 11 for the new `vip_discoveries` table plus two
+# additive columns (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`).
+# Holds the keepalived.conf an agent found already on a node so an existing VIP can be
+# adopted instead of retyped. Additive + idempotent; no existing table is altered and no
+# existing row changes. NOTE for the upgrade notes: a SCHEMA_VERSION bump re-seeds the four
+# built-in roles to their defaults, so role customizations are lost on this upgrade.
+# v1.11.0 (unified request/response log): bumped 11 -> 12 for the brand-new
+# `request_logs` table (ensure_request_logs_table), its retention-settings seed
+# (ensure_request_log_settings), and the new `requestlog.read` /
+# `requestlog.manage` permissions added to the built-in roles in
+# update_system_roles_to_enterprise_rbac().
+#
+# 12, NOT 11. The feature branch was cut when this constant was still 10 and
+# proposed 11, but 11 was taken in the meantime by v1.10.4 (vip_discoveries)
+# above. Landing it as 11 would be silently inert: run_all_migrations() returns
+# early on `applied_version >= SCHEMA_VERSION`, so every database already at 11
+# would skip the whole sequence and get neither the table nor the permissions,
+# while a fresh install would get both. Same class of bug the bump exists to
+# prevent, one number later.
+#
+# Additive + idempotent; no existing table is altered, agents never read this
+# table. Same caveat as v1.10.4: this bump re-seeds the four built-in roles to
+# their defaults, so export role customizations before upgrading.
+SCHEMA_VERSION = 12
+
+
 async def run_all_migrations():
-    """Run all database migrations"""
+    """Run all database migrations.
+
+    Hardened for multiple backend replicas / rolling deploys:
+    - A session-level advisory lock serializes the run so only one pod migrates
+      at a time (others wait, then hit the version gate and skip). It is
+      session-scoped, so it auto-releases if a pod dies mid-migration.
+    - A schema-version marker (schema_migrations) gates the run: when the DB is
+      already at SCHEMA_VERSION the whole idempotent sequence is skipped, so no
+      DDL is issued and a serving replica's traffic can't block it.
+    If the advisory lock or marker can't be used, we fall back to running the
+    (idempotent) migrations rather than crashing startup.
+    """
     logger.info("Starting database migrations...")
-    
+    MIGRATION_ADVISORY_LOCK_KEY = 1836016242  # single-key advisory space ("migr"); distinct from the (ns,id) locks used elsewhere
+    lock_conn = None
+    lock_acquired = False
+    try:
+        lock_conn = await get_database_connection()
+        try:
+            await lock_conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            lock_acquired = True
+            logger.info("Acquired migration advisory lock (migrations serialized across pods)")
+        except Exception as _lock_e:
+            logger.warning(f"Could not acquire migration advisory lock; proceeding (migrations are idempotent): {_lock_e}")
+
+        # Schema-version gate: skip the lock-heavy sequence if the DB is current.
+        applied_version = None
+        try:
+            await lock_conn.execute("""
+                CREATE TABLE IF NOT EXISTS schema_migrations (
+                    id INTEGER PRIMARY KEY DEFAULT 1,
+                    version INTEGER NOT NULL,
+                    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                    CONSTRAINT schema_migrations_singleton CHECK (id = 1)
+                )
+            """)
+            applied_version = await lock_conn.fetchval("SELECT version FROM schema_migrations WHERE id = 1")
+        except Exception as _mk_e:
+            logger.warning(f"schema_migrations marker unavailable; running full migrations: {_mk_e}")
+            applied_version = None
+
+        if applied_version is not None and applied_version >= SCHEMA_VERSION:
+            logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
+            return
+
+        await _run_all_migrations_inner()
+
+        try:
+            await lock_conn.execute("""
+                INSERT INTO schema_migrations (id, version, applied_at)
+                VALUES (1, $1, CURRENT_TIMESTAMP)
+                ON CONFLICT (id) DO UPDATE SET version = EXCLUDED.version, applied_at = EXCLUDED.applied_at
+            """, SCHEMA_VERSION)
+            logger.info(f"Recorded schema version {SCHEMA_VERSION} in schema_migrations.")
+        except Exception as _wr_e:
+            logger.warning(f"Could not record schema version marker (migrations still applied): {_wr_e}")
+    finally:
+        if lock_acquired and lock_conn is not None:
+            try:
+                await lock_conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            except Exception:
+                pass
+        if lock_conn is not None:
+            await close_database_connection(lock_conn)
+
+
+async def _run_all_migrations_inner():
+    """The full idempotent migration sequence. Runs under the migration advisory
+    lock and is gated by the schema-version marker in run_all_migrations()."""
     # First, ensure basic database schema exists
     await run_init_sql()
-    
+
     # Then run additional migrations
     await ensure_agents_table()
     await ensure_config_versions_metadata_column()
@@ -1684,6 +1910,9 @@ async def run_all_migrations():
     await ensure_system_settings_table()
     await ensure_acme_tables()
     await ensure_acme_columns_on_existing_tables()
+    # Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+    # MUST run after ensure_acme_tables() (FK references letsencrypt_accounts).
+    await ensure_letsencrypt_dns_credentials()
     # Issue #11 cleanup: must run AFTER acme_tables/columns to ensure FK refs exist
     await cleanup_orphan_acme_challenge_backend()
     # v1.5.0 Feature A (ACME diagnostics) + Feature B (site wizard)
@@ -1700,8 +1929,485 @@ async def run_all_migrations():
     # (cluster_id, bind_address, bind_port) WHERE is_active.
     await ensure_frontends_bind_unique_constraint()
 
+    # Issue #18 — TOTP MFA (v1.6.0): additive columns + 3 new tables
+    await ensure_mfa_columns()
+
+    # Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
+    # MUST run after its FK targets (haproxy_cluster_pools/agents/users), all created above.
+    await ensure_vip_tables()
+
+    # v1.9.0 — CSR creation: brand-new ssl_csrs table. FK-references
+    # ssl_certificates/users, both created above.
+    await ensure_ssl_csrs_table()
+
+    # v1.11.0 — unified request/response log: brand-new request_logs table
+    # (no FK targets) plus the seed for its operator-tunable retention
+    # settings. Both are additive and idempotent.
+    await ensure_request_logs_table()
+    await ensure_request_log_settings()
+
     logger.info("Database migrations completed successfully.")
 
+
+async def ensure_ssl_csrs_table():
+    """v1.9.0 — CSR (Certificate Signing Request) creation. Additive only:
+    one brand-new table (ssl_csrs) + indexes. No ALTER of any existing table,
+    so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (ssl_certificates, users)
+    are created earlier in the sequence.
+
+    A CSR row holds a locally generated private key + CSR PEM until the
+    operator imports the CA-signed certificate. The import creates a normal
+    ssl_certificates row (source='csr', last_config_status='PENDING') and
+    NULLs the private_key_pem copy here — the key then lives only on the
+    certificate row, like every other key in the system. Agents never read
+    this table: the agent SSL delivery endpoint selects from
+    ssl_certificates only, so a pending CSR can never leak to an agent.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS ssl_csrs (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(100) NOT NULL,
+                common_name VARCHAR(253) NOT NULL,
+                subject JSONB NOT NULL DEFAULT '{}'::jsonb,
+                sans JSONB NOT NULL DEFAULT '[]'::jsonb,
+                key_algorithm VARCHAR(20) NOT NULL DEFAULT 'rsa-2048',
+                csr_pem TEXT NOT NULL,
+                private_key_pem TEXT,
+                status VARCHAR(20) NOT NULL DEFAULT 'pending',
+                ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL,
+                completed_at TIMESTAMP,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT ssl_csrs_status_check CHECK (status IN ('pending', 'completed'))
+            );
+        """)
+
+        # Only PENDING CSRs reserve their name: the name becomes the
+        # ssl_certificates.name (and thus /etc/ssl/haproxy/{name}.pem on every
+        # agent) at import time, so two open CSRs must not target the same
+        # cert name. Completed CSRs are history and may share a name across
+        # reissues — mirrors the uq_vip_name_active partial-index rationale.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_ssl_csrs_name_pending ON ssl_csrs(name) WHERE status = 'pending';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_status ON ssl_csrs(status);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_cert ON ssl_csrs(ssl_certificate_id);"
+        )
+
+        logger.info("ssl_csrs table ensured (v1.9.0 CSR creation)")
+    except Exception as e:
+        logger.error(f"Error ensuring ssl_csrs table: {e}")
+        # Re-raise (ensure_ssl_cluster_junction_table precedent): this step is
+        # part of the SCHEMA_VERSION=10 bump, and run_all_migrations() records
+        # the marker only after the inner sequence completes cleanly. Swallowing
+        # a failure here would stamp version 10 with no ssl_csrs table, and the
+        # version gate would then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_request_logs_table():
+    """v1.11.0 — unified inbound/outbound request/response log.
+
+    Additive only: one brand-new table (request_logs) + indexes. No ALTER of
+    any existing table; agents never read this table.
+
+    Deliberately has NO foreign key on user_id. This is the highest-volume
+    table in the system — one row per API call — and per-insert FK validation
+    is not worth it here; `username` is a denormalized snapshot so a row stays
+    readable after the user who made the request is deleted. That is also the
+    correct audit semantics: the record should outlive the account.
+
+    Fully idempotent (CREATE TABLE/INDEX IF NOT EXISTS). Uses only PostgreSQL
+    9.5+ features (BIGSERIAL, JSONB, partial indexes, varchar_pattern_ops) so
+    there is no server-version floor beyond what the rest of the schema needs.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS request_logs (
+                id                  BIGSERIAL PRIMARY KEY,
+                request_id          VARCHAR(64) NOT NULL,
+                direction           VARCHAR(8)  NOT NULL,
+                target              VARCHAR(32),
+                method              VARCHAR(10) NOT NULL,
+                url                 TEXT        NOT NULL,
+                path                VARCHAR(512),
+                query_params        JSONB,
+                status_code         INTEGER,
+                status_class        SMALLINT    NOT NULL DEFAULT 0,
+                duration_ms         INTEGER     NOT NULL DEFAULT 0,
+                user_id             INTEGER,
+                username            VARCHAR(50),
+                client_ip           INET,
+                user_agent          TEXT,
+                request_headers     JSONB,
+                request_body        JSONB,
+                request_body_bytes  INTEGER     NOT NULL DEFAULT 0,
+                response_headers    JSONB,
+                response_body       JSONB,
+                response_body_bytes INTEGER     NOT NULL DEFAULT 0,
+                error               TEXT,
+                truncated           BOOLEAN     NOT NULL DEFAULT FALSE,
+                created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                CONSTRAINT request_logs_direction_check
+                    CHECK (direction IN ('inbound', 'outbound'))
+            );
+        """)
+
+        # Indexes run UNCONDITIONALLY on every startup, not only on first
+        # creation (the R16-2 rule established for acme_order_events): an older
+        # deploy that raced ahead of an index would otherwise be stuck doing
+        # sequential scans forever. All are IF NOT EXISTS, so re-running is free.
+
+        # --- read paths: the filters the log viewer actually issues ---
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_created_at "
+            "ON request_logs(created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_dir_created "
+            "ON request_logs(direction, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_status_created "
+            "ON request_logs(status_class, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_user_created "
+            "ON request_logs(user_id, created_at DESC) WHERE user_id IS NOT NULL;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_target_created "
+            "ON request_logs(target, created_at DESC) WHERE target IS NOT NULL;"
+        )
+        # Correlates one inbound row with the outbound calls it caused — this is
+        # what makes "which request went where" readable as a single trace.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_request_id "
+            "ON request_logs(request_id);"
+        )
+        # Prefix search on path (LIKE 'x%') needs pattern_ops to be usable under
+        # a non-C collation.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_path_prefix "
+            "ON request_logs(path varchar_pattern_ops);"
+        )
+
+        # --- prune paths: the TTL delete is split by outcome, so a plain
+        # (status_class, created_at) index would still range-scan the half it
+        # is not interested in.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_ok "
+            "ON request_logs(created_at) WHERE status_class BETWEEN 1 AND 3;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_err "
+            "ON request_logs(created_at) WHERE status_class = 0 OR status_class >= 4;"
+        )
+
+        logger.info("request_logs table ensured (v1.11.0 request/response log)")
+    except Exception as e:
+        logger.error(f"Error ensuring request_logs table: {e}")
+        # Re-raise (ssl_csrs precedent): this step is part of the
+        # SCHEMA_VERSION=11 bump and the version marker is written only after
+        # the inner sequence completes cleanly. Swallowing here would stamp
+        # version 11 with no request_logs table, and the version gate would
+        # then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_request_log_settings():
+    """v1.11.0 — seed the request/response-log retention defaults.
+
+    Runs UNCONDITIONALLY rather than inside an `if not table_exists:` branch,
+    so an install that already has `system_settings` picks the rows up too.
+    ON CONFLICT DO NOTHING means an operator's tuning is never overwritten by a
+    later upgrade.
+
+    Defaults are mirrored in utils/request_log_settings.py; the pair is pinned
+    by backend/tests/test_request_log_settings.py so they cannot drift apart.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            INSERT INTO system_settings (key, value, category, description) VALUES
+            ('requestlog.enabled', 'true', 'requestlog', 'Master switch for the request/response log'),
+            ('requestlog.capture_inbound', 'true', 'requestlog', 'Log inbound API calls'),
+            ('requestlog.capture_outbound', 'true', 'requestlog', 'Log outbound HTTP calls made by the backend'),
+            ('requestlog.capture_bodies', 'true', 'requestlog', 'Capture redacted, size-capped request/response bodies'),
+            ('requestlog.capture_get', 'true', 'requestlog', 'Log inbound GET requests'),
+            ('requestlog.capture_agent_success', 'false', 'requestlog', 'Log SUCCESSFUL agent polls too (failures are always logged); off by default because the row rate scales with fleet size, not operator activity'),
+            ('requestlog.max_body_bytes', '8192', 'requestlog', 'Per-body capture cap in bytes'),
+            ('requestlog.sample_rate', '1.0', 'requestlog', 'Sampling rate for successful inbound requests (errors always 1.0)'),
+            ('requestlog.exclude_paths', '["/api/request-logs","/api/health","/api/docs","/api/redoc","/api/openapi.json","/.well-known/acme-challenge","/api/agents/heartbeat","/static","/favicon.ico"]', 'requestlog', 'Path prefixes that are never logged'),
+            ('requestlog.success_retention_days', '7', 'requestlog', 'Retention for 1xx/2xx/3xx rows, in days'),
+            ('requestlog.error_retention_days', '30', 'requestlog', 'Retention for 4xx/5xx/transport-error rows, in days'),
+            ('requestlog.max_rows', '500000', 'requestlog', 'Hard row cap; oldest rows are pruned beyond this'),
+            ('requestlog.prune_interval_minutes', '60', 'requestlog', 'Minimum interval between retention prune passes')
+            ON CONFLICT (key) DO NOTHING
+        """)
+        logger.info("request_log retention settings seeded (v1.11.0)")
+    except Exception as e:
+        logger.error(f"Error seeding request_log settings: {e}")
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_mfa_columns():
+    """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
+
+    All operations are idempotent (ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS).
+    Default behavior preserved: every existing user gets mfa_enabled=FALSE, so login
+    flow is byte-identical for accounts that don't opt in.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            ALTER TABLE users
+                ADD COLUMN IF NOT EXISTS mfa_enabled BOOLEAN DEFAULT FALSE NOT NULL,
+                ADD COLUMN IF NOT EXISTS mfa_method VARCHAR(20),
+                ADD COLUMN IF NOT EXISTS mfa_secret_encrypted TEXT,
+                ADD COLUMN IF NOT EXISTS mfa_enrolled_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_totp_step BIGINT;
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_backup_codes (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                code_hash VARCHAR(255) NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute("""
+            CREATE INDEX IF NOT EXISTS idx_mfa_backup_codes_user
+              ON mfa_backup_codes(user_id);
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_logins (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                challenge_token VARCHAR(64) UNIQUE NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                ip_address INET
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_token ON mfa_pending_logins(challenge_token);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_expires ON mfa_pending_logins(expires_at);"
+        )
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_enrollments (
+                user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+                secret_encrypted TEXT NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_enroll_expires ON mfa_pending_enrollments(expires_at);"
+        )
+
+        logger.info("✅ MFA migration completed (Issue #18 — Phase 1)")
+    except Exception as e:
+        logger.error(f"Failed to ensure MFA columns: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_user_activity_logs_table
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+async def ensure_vip_tables():
+    """Issue #27 — HA/VIP (Keepalived) management (v1.7.0). Additive only:
+    two brand-new tables (vip_instances, vip_members) + indexes. No ALTER of any
+    existing table, so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (haproxy_cluster_pools, agents,
+    users) are created earlier in the sequence — this function is registered LAST.
+
+    Backward-compat: a cluster/agent with no VIP row is unaffected; the agent
+    delivery endpoint returns 'not_configured' for every node without a membership.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Cluster-driven keepalived.conf path (mirrors haproxy_config_path): additive +
+        # idempotent, with a universal default so operators need set nothing. The agent
+        # pulls this from its cluster, exactly like the HAProxy paths.
+        await conn.execute(
+            "ALTER TABLE haproxy_clusters ADD COLUMN IF NOT EXISTS keepalived_config_path "
+            "VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf';")
+
+        # VIP instance: one row per virtual IP (one VRRP group), anchored to a pool.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_instances (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(255) NOT NULL,
+                description TEXT,
+                pool_id INTEGER NOT NULL REFERENCES haproxy_cluster_pools(id) ON DELETE CASCADE,
+                virtual_ip VARCHAR(45) NOT NULL,
+                prefix_length INTEGER NOT NULL DEFAULT 24,
+                virtual_router_id INTEGER NOT NULL,
+                advert_int INTEGER NOT NULL DEFAULT 1,
+                auth_pass_encrypted TEXT,
+                use_unicast BOOLEAN NOT NULL DEFAULT TRUE,
+                track_haproxy BOOLEAN NOT NULL DEFAULT TRUE,
+                is_active BOOLEAN NOT NULL DEFAULT TRUE,
+                last_config_status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
+                applied_snapshot JSONB,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_vrid_range CHECK (virtual_router_id BETWEEN 1 AND 255)
+            );
+        """)
+        # Additive (idempotent) for DBs that created vip_instances before applied_snapshot
+        # existed (v1.7.0 self-review): holds the field-level state as of the last Apply so
+        # a pending edit can be rejected and fully reverted to the previous applied state.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS applied_snapshot JSONB;")
+        # Opt-in package removal (v1.7.2): when an operator deletes a VIP and explicitly ticks
+        # "also uninstall keepalived from the node(s)", we set this flag so the teardown
+        # delivery tells the agent to purge the OS package. Default FALSE = the safe enterprise
+        # default (stop+disable+remove our config, but KEEP the package). Additive + idempotent;
+        # a node we never managed stays untouched regardless.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Approval-gated deletion (v1.7.2): deleting a RUNNING VIP from the UI does NOT take
+        # effect immediately — it sets pending_delete=TRUE and stages a vip-*-delete version
+        # for Apply Management. The VIP stays is_active=TRUE (agents keep serving it, NOTHING
+        # is torn down) until the operator APPROVES; only then does apply flip is_active=FALSE
+        # and the agents tear down. Reject clears the flag and the VIP keeps running untouched.
+        # This guarantees an agent never tears a VIP down without an explicit human approval —
+        # protecting production. Additive + idempotent.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS pending_delete BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Uniqueness as PARTIAL indexes on active rows so a soft-deleted VIP frees its
+        # name/address/VRID for immediate reuse (a table-level UNIQUE would keep blocking it).
+        # NAME (v1.7.0 self-review): the original CREATE used a table-level UNIQUE on name,
+        # which left a soft-deleted VIP's name blocked (you couldn't re-create a VIP with the
+        # same name) — inconsistent with addr/VRID. Drop that constraint and use a partial
+        # index instead. Idempotent: no-op on a fresh table (no inline UNIQUE) and on re-run.
+        await conn.execute("ALTER TABLE vip_instances DROP CONSTRAINT IF EXISTS vip_instances_name_key;")
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_name_active ON vip_instances(name) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_addr_active ON vip_instances(virtual_ip) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_vrid_active ON vip_instances(pool_id, virtual_router_id) WHERE is_active=TRUE;"
+        )
+
+        # Per-node membership: which agents participate + their VRRP role/priority,
+        # the applied (delivered) config snapshot, and the agent's deploy ack.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_members (
+                id SERIAL PRIMARY KEY,
+                vip_id INTEGER NOT NULL REFERENCES vip_instances(id) ON DELETE CASCADE,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                network_interface VARCHAR(64) NOT NULL,
+                role VARCHAR(10) NOT NULL DEFAULT 'BACKUP',
+                priority INTEGER NOT NULL DEFAULT 100,
+                applied_config_content TEXT,
+                applied_config_hash VARCHAR(64),
+                last_deploy_state VARCHAR(24),
+                last_deploy_message TEXT,
+                last_deploy_hash VARCHAR(64),
+                last_deploy_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_member_role CHECK (role IN ('MASTER','BACKUP')),
+                CONSTRAINT vip_member_priority_range CHECK (priority BETWEEN 1 AND 254),
+                CONSTRAINT vip_member_unique UNIQUE (vip_id, agent_id)
+            );
+        """)
+        # Last line of defense against split-brain: at most one MASTER per VIP.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_one_master ON vip_members(vip_id) WHERE role='MASTER';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_vip ON vip_members(vip_id);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_agent ON vip_members(agent_id);"
+        )
+
+        # ── v1.10.4 — VIP adoption: what the agent found already on the node ──────────
+        # A node with a hand-maintained keepalived.conf reports it here so an existing VIP can
+        # be adopted instead of retyped. One row per agent (the file is per-node); the agent
+        # only reports a config it does NOT own, and only when the content changed.
+        #
+        # SECRETS: `raw_config` is stored MASKED (auth_pass replaced) because it is served to
+        # the UI. The real VRRP password is Fernet-encrypted in auth_pass_encrypted, mirroring
+        # vip_instances, so adoption can carry it into the managed VIP without it ever being
+        # readable through the API or a DB dump. `analysis` is the parser output with auth_pass
+        # stripped out.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_discoveries (
+                id SERIAL PRIMARY KEY,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                config_path VARCHAR(500) NOT NULL,
+                config_hash VARCHAR(64) NOT NULL,
+                is_managed BOOLEAN NOT NULL DEFAULT FALSE,
+                raw_config_masked TEXT,
+                auth_pass_encrypted TEXT,
+                analysis JSONB,
+                parse_error TEXT,
+                adopted_vip_id INTEGER REFERENCES vip_instances(id) ON DELETE SET NULL,
+                reported_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_discovery_agent_unique UNIQUE (agent_id)
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_discoveries_agent ON vip_discoveries(agent_id);"
+        )
+        # Adoption provenance + the one-shot takeover authorisation. The agent refuses to
+        # overwrite a keepalived.conf that lacks our ownership marker, which is exactly the
+        # guard adoption has to pass. Rather than weaken it, an adopted VIP carries the hash of
+        # the file we analysed: the agent takes over ONLY if the file on disk still hashes to
+        # that value, so a config that changed after adoption is never clobbered.
+        await conn.execute(
+            "ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS adopted_at TIMESTAMP;")
+        await conn.execute(
+            "ALTER TABLE vip_members ADD COLUMN IF NOT EXISTS takeover_expected_hash VARCHAR(64);")
+
+        logger.info("✅ VIP tables ensured (Issue #27 — HA/VIP Keepalived management)")
+    except Exception as e:
+        logger.error(f"Failed to ensure VIP tables: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_mfa_columns
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 async def add_ssl_certificate_id_to_backend_servers():
     """Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
     conn = None
@@ -1836,19 +2542,19 @@ async def create_initial_system_data(conn):
                 'name': 'super_admin',
                 'display_name': 'Super Administrator', 
                 'description': 'Full system access with all permissions',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'operator',
                 'display_name': 'Operator',
                 'description': 'Daily operational access for managing HAProxy configurations',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request","requestlog.read"]
             },
             {
                 'name': 'security_admin',
                 'display_name': 'Security Administrator',
                 'description': 'Security-focused access for WAF rules and SSL certificates',
-                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'viewer',
@@ -3278,6 +3984,23 @@ async def ensure_acme_columns_on_existing_tables():
             ('last_attempt_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS last_attempt_at TIMESTAMPTZ"),
             # Commit 3a: track auto-completion task lock/poll timestamps for atomic claim across replicas
             ('orders_updated_at_idx', "CREATE INDEX IF NOT EXISTS idx_letsencrypt_orders_status_updated ON letsencrypt_orders(status, updated_at) WHERE status = 'valid' AND ssl_certificate_id IS NULL"),
+            # Issue #35 (v1.8.0 — ACME DNS-01): per-account challenge method + DNS provider selection
+            ('acct_challenge_type', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('acct_dns_provider', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS dns_provider VARCHAR(50)"),
+            # per-order challenge method + bounded DNS-01 retry chain (dns01_parent_order_id is a PLAIN INTEGER, not a FK,
+            # to avoid a self-referential cascade interacting with account/order bulk DELETEs)
+            ('order_challenge_type', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('order_dns01_attempts', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_attempts INTEGER DEFAULT 0"),
+            ('order_dns01_last_attempt_at', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_last_attempt_at TIMESTAMPTZ"),
+            ('order_dns01_parent_order_id', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_parent_order_id INTEGER"),
+            ('order_dns01_retry_claimed', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_retry_claimed BOOLEAN DEFAULT FALSE"),
+            # per-challenge DNS-01 lifecycle state
+            ('chal_challenge_type', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('chal_dns_txt_value', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_txt_value TEXT"),
+            ('chal_dns_record_published', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_published BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_record_cleaned', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_cleaned BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_published_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_published_at TIMESTAMPTZ"),
+            ('chal_manual_confirm_deadline', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS manual_confirm_deadline TIMESTAMPTZ"),
         ]:
             try:
                 await conn.execute(sql)
@@ -3293,6 +4016,34 @@ async def ensure_acme_columns_on_existing_tables():
         logger.error(f"Error adding ACME columns: {e}")
 
 
+async def ensure_letsencrypt_dns_credentials():
+    """Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+
+    Idempotent (CREATE TABLE IF NOT EXISTS). FK to letsencrypt_accounts (created earlier by
+    ensure_acme_tables). Credentials are Fernet-encrypted at rest (backend/utils/dns_credentials.py);
+    only the provider name + timestamps are ever surfaced to the API.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS letsencrypt_account_dns_credentials (
+                id SERIAL PRIMARY KEY,
+                account_id INTEGER NOT NULL UNIQUE REFERENCES letsencrypt_accounts(id) ON DELETE CASCADE,
+                dns_provider VARCHAR(50) NOT NULL,
+                credentials_encrypted TEXT NOT NULL,
+                created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
+            )
+        """)
+        logger.info("Ensured letsencrypt_account_dns_credentials table")
+        await close_database_connection(conn)
+    except Exception as e:
+        if conn:
+            await close_database_connection(conn)
+        logger.error(f"Error ensuring letsencrypt_account_dns_credentials: {e}")
+
+
 async def cleanup_orphan_acme_challenge_backend():
     """
     Issue #11: One-time cleanup of orphan `_acme_challenge_backend` rows that may

backend/haproxy_client.py

@@ -61,14 +61,27 @@ class HAProxyClient:
             if self.stats_username and self.stats_password:
                 auth = aiohttp.BasicAuth(self.stats_username, self.stats_password)
             
+            # v1.11.0: instrumented for completeness. NOTE the CSV body is
+            # deliberately NOT handed to the span — a full stats dump is large,
+            # changes every poll, and has no diagnostic value in an audit row;
+            # status + duration is what matters. `auth` is likewise never logged:
+            # aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+            # password.
+            from utils.http_instrumentation import outbound_span, TARGET_HAPROXY_STATS
+
             async with aiohttp.ClientSession() as session:
-                async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
-                    if response.status == 200:
-                        csv_data = await response.text()
-                        return self._parse_csv_stats(csv_data)
-                    else:
-                        logger.warning(f"HTTP stats request failed with status {response.status}")
-                        return self._get_fallback_stats()
+                async with outbound_span(
+                    target=TARGET_HAPROXY_STATS, method="GET", url=url,
+                    capture_body=False, capture_response_body=False,
+                ) as span:
+                    async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
+                        span.set_response(response.status, getattr(response, "headers", None))
+                        if response.status == 200:
+                            csv_data = await response.text()
+                            return self._parse_csv_stats(csv_data)
+                        else:
+                            logger.warning(f"HTTP stats request failed with status {response.status}")
+                            return self._get_fallback_stats()
         except Exception as e:
             logger.error(f"HTTP stats request failed: {e}")
             return self._get_fallback_stats()

backend/main.py

@@ -8,8 +8,13 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.5.1", "releaseName": "Round-23 + Round-24 audit follow-ups", "releaseDate": "2026-05-13"}
-for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
+# Single source of truth: backend/version.json, which sits next to this module and is baked into
+# every image by `COPY . .` (build context ./backend) — no pipeline staging needed. The literal
+# below is only a last-resort "file missing" marker; it is deliberately NOT a real version so it can
+# never silently drift out of sync (this exact drift showed a stale version after v1.8.5/v1.8.6).
+# Keep the canonical version ONLY in backend/version.json — test_version_consistency.py enforces it.
+_version_info = {"version": "unknown", "releaseName": "unknown", "releaseDate": ""}
+for _vpath in [os.path.join(os.path.dirname(__file__), "version.json"), "/app/version.json"]:
     try:
         with open(_vpath) as _vf:
             _version_info = json.load(_vf)
@@ -24,7 +29,7 @@ for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), ".."
 
 # Import configurations and database
 
-from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL
+from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL, REQUEST_LOG_ENABLED
 from database.connection import redis_client, get_database_connection, close_database_connection, init_database_pool, close_database_pool
 from database.migrations import run_all_migrations
 
@@ -40,6 +45,10 @@ from routers.settings import router as settings_router
 from routers.letsencrypt import router as letsencrypt_router
 from routers.acme_diagnostics import router as acme_diagnostics_router
 from routers.site_wizard import router as site_wizard_router
+from routers.mfa import router as mfa_router
+from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
+from routers.csr import router as csr_router  # v1.9.0 — CSR creation (in-app key+CSR generation, signed-cert import)
+from routers.request_logs import router as request_logs_router  # v1.11.0 — unified request/response log
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -48,6 +57,10 @@ from middleware.error_handler import (
     GlobalExceptionHandler, get_error_statistics
 )
 from middleware.activity_logger import log_activity_middleware
+from middleware.request_logger import RequestResponseLogMiddleware  # v1.11.0
+from utils.http_instrumentation import begin_background_trace  # v1.11.0
+from utils.request_log_settings import refresh_config as refresh_request_log_config
+from utils.request_log_sink import request_log_sink
 
 # Setup structured logging
 logger = setup_production_logging(LOG_LEVEL)
@@ -224,6 +237,8 @@ Most operations are **cluster-scoped**:
 async def monitor_agent_status():
     """Background task to monitor agent status and mark offline agents"""
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("agent_status_monitor")
         try:
             conn = await get_database_connection()
             
@@ -268,6 +283,10 @@ async def complete_pending_acme_orders():
     """
     await asyncio.sleep(60)
     while True:
+        # v1.11.0: one correlation id per TICK, so the outbound rows for this
+        # pass group together and do not merge with every other pass this
+        # process has ever run.
+        begin_background_trace("acme_complete_orders")
         try:
             conn_check = await get_database_connection()
             try:
@@ -302,6 +321,21 @@ async def complete_pending_acme_orders():
                         WHERE (
                             status IN ('pending', 'processing', 'ready')
                             OR (status = 'valid' AND ssl_certificate_id IS NULL)
+                            -- Issue #35: bounded DNS-01 retry. ONLY dns-01 invalids with remaining
+                            -- budget + elapsed backoff are claimed; http-01 invalids are NEVER matched
+                            -- (their existing skip-and-log is preserved).
+                            OR (
+                                status = 'invalid' AND challenge_type = 'dns-01'
+                                AND ssl_certificate_id IS NULL
+                                AND COALESCE(dns01_retry_claimed, FALSE) = FALSE
+                                AND COALESCE(dns01_attempts, 0) < 3
+                                AND (
+                                    dns01_last_attempt_at IS NULL
+                                    OR dns01_last_attempt_at < NOW() - (
+                                        (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
+                                        || ' minutes')::INTERVAL
+                                )
+                            )
                         )
                         AND created_at > NOW() - INTERVAL '7 days'
                         AND (updated_at IS NULL OR updated_at < NOW() - INTERVAL '30 seconds')
@@ -335,9 +369,17 @@ async def complete_pending_acme_orders():
                 continue
 
             logger.info(f"[ACME-COMPLETE] Claimed {len(claimed_ids)} order(s) for completion: {claimed_ids}")
-            
+
+            from services.dns01_orchestrator import (
+                advance_dns01_order, retry_invalid_dns01, reconcile_dns01_cleanup,
+            )
+
             for oid in claimed_ids:
                 try:
+                    # Issue #35: advance the DNS-01 publish->confirm->respond state machine for
+                    # pending dns-01 orders (no-op for http-01 or non-pending orders).
+                    await advance_dns01_order(oid)
+
                     status_info = await acme_svc.check_order_status(oid)
                     current_status = status_info.get('status')
 
@@ -350,12 +392,21 @@ async def complete_pending_acme_orders():
                         result = await _complete_certificate(oid)
                         logger.info(f"[ACME-COMPLETE] Order {oid} completed - {result.get('message', '')}")
                     elif current_status == 'invalid':
-                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid, skipping")
+                        # Issue #35: bounded DNS-01 fresh-order retry (no-op for http-01).
+                        await retry_invalid_dns01(oid)
+                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid")
                     elif current_status in ('pending', 'processing'):
                         logger.info(f"[ACME-COMPLETE] Order {oid} still {current_status}, will retry next cycle")
                 except Exception as poll_err:
                     logger.error(f"[ACME-COMPLETE] Failed to complete order {oid}: {poll_err}")
 
+            # Issue #35: best-effort cleanup of TXT records left published on terminal orders
+            # (covers a failed cleanup or the kill-switch being flipped off). NOT gated by the switch.
+            try:
+                await reconcile_dns01_cleanup()
+            except Exception as rec_err:
+                logger.debug(f"[ACME-COMPLETE] DNS-01 reconcile skipped: {rec_err}")
+
             # NOTE: v1.5.0 wizard-staged processing now runs BEFORE the
             # claimed_ids early-continue above (Bulgu #2 fix), so it executes
             # every cycle regardless of pending/processing volume.
@@ -605,6 +656,8 @@ async def check_letsencrypt_renewals():
     """
     await asyncio.sleep(120)
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("acme_renewals")
         conn = None
         try:
             conn = await get_database_connection()
@@ -671,7 +724,9 @@ async def check_letsencrypt_renewals():
                     skip = False
                     try:
                         order = await conn2.fetchrow(
-                            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1",
+                            "SELECT o.account_id, o.domains, o.cluster_ids, o.challenge_type, a.dns_provider "
+                            "FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id "
+                            "WHERE o.id = $1",
                             order_id
                         )
                         if order:
@@ -687,15 +742,37 @@ async def check_letsencrypt_renewals():
                             if existing:
                                 logger.info(f"[ACME-RENEWAL] Skipping cert {cert['id']} - order {existing['id']} already in progress")
                                 skip = True
+                            elif (order['challenge_type'] == 'dns-01'):
+                                # Issue #35: manual DNS-01 cannot auto-renew unattended; and for an
+                                # automated provider, don't re-mint hourly if a recent retry chain already
+                                # exhausted its budget (avoids tripping the CA new-order rate limit).
+                                if (order['dns_provider'] or 'manual') == 'manual':
+                                    logger.warning(f"[ACME-RENEWAL] cert {cert['id']} uses manual DNS-01; cannot auto-renew unattended (publish the TXT and renew manually)")
+                                    skip = True
+                                else:
+                                    exhausted = await conn2.fetchrow("""
+                                        SELECT id FROM letsencrypt_orders
+                                        WHERE domains::text = $1::text AND challenge_type = 'dns-01'
+                                        AND status = 'invalid' AND COALESCE(dns01_attempts, 0) >= 3
+                                        AND created_at > NOW() - INTERVAL '24 hours'
+                                        LIMIT 1
+                                    """, json.dumps(domains))
+                                    if exhausted:
+                                        logger.warning(f"[ACME-RENEWAL] cert {cert['id']} DNS-01 renewal recently failed (check DNS); skipping re-mint for 24h")
+                                        skip = True
                     finally:
                         await close_database_connection(conn2)
 
                     if not order or skip:
                         continue
 
-                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids)
-                    await acme_svc.respond_to_challenges(new_order['order_id'])
-                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} for cert {cert['id']} ({cert['name']})")
+                    challenge_type = order['challenge_type'] or 'http-01'
+                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids, challenge_type=challenge_type)
+                    # http-01 responds immediately (token served continuously); dns-01 is driven by the
+                    # orchestrator AFTER the TXT is published (never respond before publish).
+                    if challenge_type != 'dns-01':
+                        await acme_svc.respond_to_challenges(new_order['order_id'])
+                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} ({challenge_type}) for cert {cert['id']} ({cert['name']})")
                 except Exception as cert_err:
                     logger.error(f"[ACME-RENEWAL] Failed to initiate renewal for cert {cert['id']}: {cert_err}")
 
@@ -778,6 +855,41 @@ async def cleanup_stuck_agent_upgrades():
         # Wait 120 seconds (2 minutes) before next check
         await asyncio.sleep(120)
 
+async def prune_request_logs_loop():
+    """v1.11.0 — retention prune for `request_logs`.
+
+    Kept independent of the ACME prune loop on purpose: that one is gated on
+    the `letsencrypt_orders` table existing, which would silently disable this
+    prune on an install that never uses ACME.
+
+    The 5-minute tick is only a heartbeat — the real gate is the DB watermark
+    plus `requestlog.prune_interval_minutes`, so N replicas ticking every 5
+    minutes still produce one pass per configured interval.
+    """
+    # Stagger past startup so migrations and the first request burst are done.
+    await asyncio.sleep(180)
+
+    while True:
+        try:
+            conn = await get_database_connection()
+            try:
+                table_exists = await conn.fetchval("""
+                    SELECT EXISTS (
+                        SELECT 1 FROM information_schema.tables
+                        WHERE table_name = 'request_logs'
+                    )
+                """)
+            finally:
+                await close_database_connection(conn)
+
+            if table_exists:
+                from utils.request_log_prune import prune_request_logs_if_due
+                await prune_request_logs_if_due()
+        except Exception as e:
+            logger.error(f"Error in request_logs prune loop: {e}")
+
+        await asyncio.sleep(300)
+
 # Production middleware stack (order matters!)
 app.add_middleware(PerformanceMonitoringMiddleware, slow_request_threshold_ms=1000)
 app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/docs", "/redoc"])
@@ -785,15 +897,35 @@ app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/do
 # Activity logging middleware - must be before CORS
 app.middleware("http")(log_activity_middleware)
 
-# CORS middleware  
+# CORS middleware
 app.add_middleware(
     CORSMiddleware,
     allow_origins=CORS_ORIGINS,
     allow_credentials=True,
     allow_methods=["*"],
     allow_headers=["*"],
+    # v1.11.0: without an explicit expose list, browser JS on a cross-origin
+    # deployment cannot read ANY of these — so an operator could see the
+    # X-Request-ID in devtools but the app could never quote it back. Same-origin
+    # (nginx) deployments were already fine; this fixes the split-origin case.
+    expose_headers=["X-Correlation-ID", "X-Response-Time", "X-Request-ID"],
 )
 
+# v1.11.0 — unified request/response log.
+#
+# MUST be the LAST add_middleware call: Starlette inserts each new middleware at
+# index 0, so the last registration ends up OUTERMOST. Outermost is what we want:
+#   (a) we see the exact status/headers/body the client receives, including the
+#       JSONResponse that RequestLoggingMiddleware fabricates from an exception
+#       it swallowed, and
+#   (b) we seed correlation_id_context BEFORE RequestLoggingMiddleware calls
+#       get_correlation_id(), so X-Correlation-ID matches request_logs.request_id.
+#
+# REQUEST_LOG_ENABLED=false keeps it out of the ASGI stack entirely — not a
+# runtime branch, genuinely zero overhead.
+if REQUEST_LOG_ENABLED:
+    app.add_middleware(RequestResponseLogMiddleware)
+
 # Global exception handlers
 from fastapi.exceptions import RequestValidationError
 from starlette.exceptions import HTTPException as StarletteHTTPException
@@ -820,6 +952,7 @@ app.include_router(config_router)  # Configuration management
 app.include_router(maintenance_router, prefix="/api", tags=["maintenance"])  # Database cleanup & maintenance
 app.include_router(auth_router)
 app.include_router(user_router)
+app.include_router(mfa_router)
 app.include_router(frontend_router)
 app.include_router(backend_router)
 app.include_router(cluster_router)
@@ -828,12 +961,15 @@ app.include_router(dashboard_stats_router)  # HAProxy stats dashboard
 app.include_router(agent_router)
 app.include_router(waf_router)
 app.include_router(ssl_router)
+app.include_router(csr_router)  # v1.9.0: CSR creation (in-app key+CSR generation, signed-cert import)
+app.include_router(request_logs_router)  # v1.11.0: unified request/response log
 app.include_router(security_router)
 app.include_router(configuration_router)
 app.include_router(settings_router)
 app.include_router(letsencrypt_router)
 app.include_router(acme_diagnostics_router)  # v1.5.0 Issue #13: ACME Diagnostic Panel
 app.include_router(site_wizard_router)  # v1.5.0 Issue #14: New Site Setup Wizard
+app.include_router(vip_router)  # v1.7.0 Issue #27: HA/VIP (Keepalived) management
 
 
 # Legacy URL alias: /api/proxied-hosts/* → 308 redirect to /api/sites/*.
@@ -964,6 +1100,17 @@ async def startup_event():
         # Decoupled from auto_renew_enabled flag so user-initiated orders also complete.
         asyncio.create_task(complete_pending_acme_orders())
         logger.info("ACME order auto-completion task started (60s checks, replica-safe)")
+
+        # v1.11.0 — request/response log: load the operator's capture/retention
+        # policy, then start the batching writer and the retention prune.
+        # Guarded by the env kill-switch so a deployment that turned the log off
+        # pays for neither task.
+        if REQUEST_LOG_ENABLED:
+            await refresh_request_log_config()
+            asyncio.create_task(request_log_sink.run())
+            logger.info("Request/response log sink started (batching writer)")
+            asyncio.create_task(prune_request_logs_loop())
+            logger.info("Request/response log retention prune task started")
         
         # Create test activity log entry to verify system is working
         try:
@@ -992,6 +1139,16 @@ async def shutdown_event():
     """Cleanup on shutdown"""
     logger.info("HAProxy OpenManager API shutting down...")
 
+    # v1.11.0: flush queued request-log rows FIRST. The sink's writer is a
+    # `while True` loop, so it can never satisfy the asyncio.wait below — the
+    # rows still sitting in its queue would be lost when the pool closes.
+    try:
+        flushed = await request_log_sink.flush(timeout=3.0)
+        if flushed:
+            logger.info(f"Flushed {flushed} queued request-log row(s)")
+    except Exception as flush_err:
+        logger.warning(f"request-log flush skipped: {flush_err}")
+
     # R18c audit fix (round 3 #5): drain pending fire-and-forget
     # background tasks BEFORE closing the DB pool. The audit
     # logger middleware (`activity_logger.py`) and the wizard
@@ -1044,14 +1201,24 @@ async def get_version():
     return _version_info
 
 @app.get("/.well-known/acme-challenge/{token}")
-async def serve_acme_challenge(token: str):
+async def serve_acme_challenge(token: str, request: Request):
     """Serve ACME HTTP-01 challenge token. Public endpoint, no auth required."""
-    logger.info(f"ACME-CHALLENGE: Incoming request for token={token[:32]}...")
+    # Log who reached us. When HTTP-01 fails, the first question is always "did the
+    # request get here at all?" — and the answer separates a broken challenge-backend
+    # address (nothing arrives) from a wrong response (arrives, wrong body). The peer
+    # is normally the HAProxy node; X-Forwarded-For carries the CA when the frontend
+    # sets `option forwardfor`.
+    _peer = request.client.host if request.client else 'unknown'
+    _xff = request.headers.get('x-forwarded-for') or '-'
+    logger.info(
+        f"ACME-CHALLENGE: Incoming request for token={token[:32]}... "
+        f"peer={_peer} xff={_xff} host={request.headers.get('host') or '-'}"
+    )
     conn = None
     try:
         conn = await get_database_connection()
         row = await conn.fetchrow(
-            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) LIMIT 1",
+            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) AND (challenge_type = 'http-01' OR challenge_type IS NULL) LIMIT 1",
             token
         )
         if row:

backend/middleware/error_handler.py

@@ -126,10 +126,26 @@ class GlobalExceptionHandler:
             except Exception as body_error:
                 logger.debug(f"Could not extract raw body for debugging: {body_error}")
         
-        # Enhanced log message for agent heartbeats
-        log_message = f"Validation error: {str(exc)}"
+        # Build a sanitized log summary. The raw `str(exc)` from Pydantic
+        # contains the user-supplied `input` value for each failed field —
+        # which leaks secrets like TOTP codes, backup codes, mfa_token, and
+        # passwords to plaintext logs. Use only field NAMES + types here;
+        # `validation_details` (already sanitized to {field, message, type})
+        # is attached separately for downstream structured logging.
+        _field_names = [
+            err.get("field", "unknown")
+            for err in error_details.get("validation_errors", [])
+        ]
+        _err_count = len(error_details.get("validation_errors", []))
+        log_message = (
+            f"Validation error: {_err_count} field(s) failed validation: "
+            f"[{', '.join(_field_names)}]"
+        )
         if agent_name != "unknown":
-            log_message = f"Agent '{agent_name}' heartbeat validation error: {str(exc)}"
+            log_message = (
+                f"Agent '{agent_name}' heartbeat validation error: "
+                f"{_err_count} field(s) failed: [{', '.join(_field_names)}]"
+            )
         
         # Log validation error with enhanced details
         log_with_correlation(

backend/middleware/mfa_rate_limit_key.py

@@ -0,0 +1,147 @@
+"""MFA rate-limit key extraction (user-aware + ingress-aware).
+
+Problem with the default ``slowapi.util.get_remote_address``:
+
+  * Behind an ingress / reverse proxy / load balancer, every request looks
+    like it originates from the same upstream IP (the proxy itself). A
+    5/minute IP-bucket therefore becomes a 5/minute *whole-organization*
+    bucket. In a 500-user enterprise rolling out MFA, this stalls the
+    rollout to a trickle.
+
+Strategy (combined B + C from the design review):
+
+  1. **User-aware key (preferred):** if the request carries a valid Bearer
+     JWT, derive the bucket from ``user:<id>``. Each authenticated user
+     gets an isolated bucket, regardless of source IP. A malicious user
+     burning their own quota cannot starve the other 499.
+
+  2. **Trusted-proxy XFF fallback:** if the request is unauthenticated
+     (e.g. ``/login`` flow, future endpoints) and the TCP peer is in
+     ``MFA_TRUSTED_PROXY_CIDRS``, peel the first hop off ``X-Forwarded-For``.
+     This preserves real-IP buckets behind a known ingress without
+     accepting spoofed headers from the public internet.
+
+  3. **Default fallback:** plain ``request.client.host`` (slowapi default).
+
+JWT decode is intentionally signature-verified (replay/spoof protection)
+and *sync* — slowapi's decorator hook is sync, and our JWT library
+(python-jose) is sync as well. Failed verification silently downgrades
+to IP-based bucketing — never crashes the decorator.
+
+Env vars:
+  * ``MFA_TRUSTED_PROXY_CIDRS`` — comma-separated CIDR list of trusted
+    upstream proxies. Empty (default) disables XFF parsing entirely,
+    which is the safe choice when the deployment topology is unknown.
+    Examples:
+        ``MFA_TRUSTED_PROXY_CIDRS=10.0.0.0/8,172.16.0.0/12``
+        ``MFA_TRUSTED_PROXY_CIDRS=192.168.0.0/16``
+"""
+from __future__ import annotations
+
+import logging
+import os
+from ipaddress import ip_address, ip_network
+from typing import List
+
+from fastapi import Request
+from jose import jwt
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_trusted_cidrs() -> List:
+    """Parse the trusted-proxy CIDR list at import time.
+
+    A malformed entry is logged and skipped — we never crash the process
+    over a typo in operational config.
+    """
+    raw = os.getenv("MFA_TRUSTED_PROXY_CIDRS", "").strip()
+    if not raw:
+        return []
+    nets = []
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        try:
+            nets.append(ip_network(entry, strict=False))
+        except ValueError:
+            logger.warning(
+                "MFA_TRUSTED_PROXY_CIDRS: ignoring invalid CIDR %r", entry
+            )
+    return nets
+
+
+_TRUSTED_NETS = _parse_trusted_cidrs()
+
+
+def _peer_ip(request: Request) -> str:
+    """The TCP peer IP — never raises; falls back to ``0.0.0.0``."""
+    return request.client.host if request.client else "0.0.0.0"
+
+
+def _is_trusted_peer(peer: str) -> bool:
+    if not _TRUSTED_NETS:
+        return False
+    try:
+        peer_ip = ip_address(peer)
+    except ValueError:
+        return False
+    return any(peer_ip in net for net in _TRUSTED_NETS)
+
+
+def _real_ip(request: Request) -> str:
+    """If TCP peer is in a trusted proxy CIDR, peel off the first
+    ``X-Forwarded-For`` IP; otherwise return the peer.
+
+    ``X-Forwarded-For`` from an *untrusted* peer is intentionally ignored —
+    accepting it would let any client spoof their bucket.
+    """
+    peer = _peer_ip(request)
+    if not _is_trusted_peer(peer):
+        return peer
+    xff = request.headers.get("X-Forwarded-For")
+    if not xff:
+        return peer
+    first = xff.split(",")[0].strip()
+    return first or peer
+
+
+def _user_id_from_jwt(request: Request) -> str | None:
+    """Sync JWT decode → ``user_id`` (or ``sub``) claim. None on failure.
+
+    Uses the same secret + algorithm as ``auth_middleware`` so a token that
+    is valid for the API surface is also valid for the rate-limit key.
+    Bad / missing / expired tokens silently return None — slowapi falls
+    back to IP bucketing.
+    """
+    auth = request.headers.get("Authorization", "")
+    if not auth.startswith("Bearer "):
+        return None
+    token = auth[7:].strip()
+    if not token or token in {"null", "undefined"}:
+        return None
+    # Late import to avoid pulling jose into module-load when not needed.
+    try:
+        from config import JWT_ALGORITHM, JWT_SECRET_KEY
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None
+    uid = payload.get("user_id") or payload.get("sub")
+    if uid is None:
+        return None
+    return str(uid)
+
+
+def mfa_rate_limit_key(request: Request) -> str:
+    """slowapi ``key_func`` for MFA endpoints.
+
+    Order:
+      1. Authenticated → ``user:<id>``
+      2. Trusted-proxy XFF → ``ip:<first hop>``
+      3. TCP peer → ``ip:<peer>``
+    """
+    uid = _user_id_from_jwt(request)
+    if uid is not None:
+        return f"user:{uid}"
+    return f"ip:{_real_ip(request)}"

backend/middleware/mfa_rate_limits.py

@@ -0,0 +1,79 @@
+"""MFA rate-limit configuration (env-overridable).
+
+Best-practice pattern:
+  - Secure-by-default values live in code (kept in sync with the threat model).
+  - Operations can override per-environment via env vars (ConfigMap on K8s)
+    WITHOUT a code change / re-release.
+  - All limits funnel through a single named constant so the decorator stays
+    declarative (``@limiter.limit(MFA_LIMITS.enroll_start)``).
+
+Env-var precedence::
+
+    MFA_RATE_LIMIT_<NAME>  >  default in code
+
+slowapi limit string syntax: ``<count>/<period>`` where period is
+``second|minute|hour|day``. Example: ``"5/minute"``.
+
+NOTE: slowapi binds limits at import time. A change to an env var requires a
+backend restart (rolling restart on K8s, ``docker compose restart backend``
+locally). This is consistent with how ``SECRET_KEY`` / ``MFA_ENCRYPTION_KEY``
+behave.
+"""
+from __future__ import annotations
+
+import logging
+import os
+import re
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+# slowapi limit-string format guard. Keeps a typo from silently disabling
+# rate-limiting at process start.
+_LIMIT_RE = re.compile(r"^\d+/(second|minute|hour|day)$")
+
+
+def _env(name: str, default: str) -> str:
+    """Read ``MFA_RATE_LIMIT_<NAME>``; fall back to ``default``.
+
+    Validates the limit string. On bad input, logs a warning and returns the
+    secure default instead of crashing the process.
+    """
+    value = os.getenv(f"MFA_RATE_LIMIT_{name}", default).strip()
+    if not _LIMIT_RE.match(value):
+        logger.warning(
+            "MFA_RATE_LIMIT_%s='%s' is not a valid slowapi limit string "
+            "(expected '<n>/<second|minute|hour|day>'); using default '%s'.",
+            name, value, default,
+        )
+        return default
+    return value
+
+
+@dataclass(frozen=True)
+class MfaRateLimits:
+    """Aggregate of MFA endpoint rate-limit strings (slowapi format).
+
+    Defaults assume the rate-limit ``key_func`` is ``mfa_rate_limit_key``
+    (user-aware + ingress-aware), NOT raw IP. Per-user buckets are safe to
+    keep generous because a misbehaving user only burns their own quota and
+    cannot starve the rest of the org. If you re-key on raw IP, retighten
+    these values (see README + ``MFA_RATE_LIMIT_<NAME>`` env overrides).
+    """
+
+    # Enrollment lifecycle — per-user buckets, large enough for org-wide rollout
+    enroll_start: str = _env("ENROLL_START", "10/minute")
+    enroll_confirm: str = _env("ENROLL_CONFIRM", "10/minute")
+
+    # Self-service maintenance
+    disable: str = _env("DISABLE", "10/minute")
+    regenerate_backup_codes: str = _env("REGENERATE_BACKUP_CODES", "5/hour")
+
+    # Admin operations (per-admin bucket; bulk reset stays tight because
+    # it is an emergency-only flow).
+    admin_reset: str = _env("ADMIN_RESET", "60/hour")
+    admin_reset_all: str = _env("ADMIN_RESET_ALL", "1/day")
+
+
+# Module-level singleton — import this from routers/mfa.py.
+MFA_LIMITS = MfaRateLimits()

backend/middleware/rate_limiter.py

@@ -2,6 +2,7 @@
 Rate Limiting Middleware for Production Security
 Protects API endpoints from abuse and DDoS attacks
 """
+import os
 import time
 import logging
 from typing import Callable
@@ -16,10 +17,15 @@ from database.connection import redis_client
 
 logger = logging.getLogger(__name__)
 
-# Rate limiter instance using Redis backend  
+# Resolve the Redis storage URI from REDIS_URL (set by docker-compose / k8s
+# ConfigMap). Falls back to the compose service hostname for backward
+# compatibility when REDIS_URL is unset.
+_REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379").rstrip("/")
+_LIMITER_STORAGE_URI = f"{_REDIS_URL}/0" if "/" not in _REDIS_URL.split("//", 1)[-1] else _REDIS_URL
+
 limiter = Limiter(
     key_func=get_remote_address,
-    storage_uri="redis://redis:6379/0",
+    storage_uri=_LIMITER_STORAGE_URI,
     default_limits=["1000/hour"],  # Default global limit
     retry_after=lambda name, t: int(t) + 10
 )

backend/middleware/request_logger.py

@@ -0,0 +1,321 @@
+"""v1.11.0 — inbound half of the unified request/response log.
+
+Pure ASGI on purpose, NOT BaseHTTPMiddleware:
+
+  * `BaseHTTPMiddleware` hands the response back as a
+    `starlette.middleware.base._StreamingResponse`, which has no `.body` to
+    read, and
+  * `await request.body()` inside a `dispatch()` DRAINS the receive channel.
+    `POST /api/agents/heartbeat` (routers/agent.py) reads the raw stream
+    itself, as does the validation-error body preview in
+    middleware/error_handler.py — draining it here would break both.
+
+So we never consume anything: we TEE. `receive` and `send` are wrapped, every
+message is forwarded verbatim, and a size-capped copy is kept for the log row.
+Cost per in-flight request is therefore bounded at ~2 × max_body_bytes (8 KB
+by default), not the size of the upload.
+
+Registration: this MUST be the LAST `app.add_middleware(...)` call, because
+Starlette inserts at index 0 — the last registration is the OUTERMOST
+middleware. Outermost is what we want: we then see the exact status and body
+the client receives (including the JSONResponse that RequestLoggingMiddleware
+fabricates out of a swallowed exception), and we can seed
+`correlation_id_context` before anything downstream reads it.
+"""
+import logging
+import time
+import uuid
+from typing import Any, Dict, List, Optional, Tuple
+
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from utils.logging_config import correlation_id_context
+from utils.request_log_redaction import is_capturable_content_type, scrub_query_string
+from utils.request_log_settings import get_config
+from utils.request_log_sink import (
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    request_id_context,
+    request_log_sink,
+)
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Hard floor, NOT settable away through `requestlog.exclude_paths`. Without it
+# an operator who clears the exclude list turns the log viewer into a machine
+# that logs itself reading its own logs.
+_ALWAYS_EXCLUDED: Tuple[str, ...] = ("/api/request-logs",)
+
+
+def _header(scope: Scope, name: bytes) -> Optional[str]:
+    for key, value in scope.get("headers") or ():
+        if key == name:
+            try:
+                return value.decode("latin-1")
+            except Exception:
+                return None
+    return None
+
+
+def _identify(scope: Scope) -> Tuple[Optional[int], Optional[str]]:
+    """Resolve the caller from the JWT locally — NO database round-trip.
+
+    `log_activity_middleware` already pays a `SELECT ... FROM users` per
+    non-GET request; this middleware runs on every request including GETs, so a
+    second lookup per call is not acceptable. The token issued at
+    routers/auth.py carries both `user_id` and `username`, which is everything
+    the log row needs.
+
+    A token that fails to decode simply yields (None, None): this is a logging
+    path, not an authorization path — the real auth check still runs
+    downstream.
+    """
+    raw = _header(scope, b"authorization")
+    if not raw:
+        return None, None
+    token = raw[7:].strip() if raw.lower().startswith("bearer ") else raw.strip()
+    if not token or token in ("null", "undefined") or token.count(".") != 2:
+        return None, None
+    try:
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None, None
+
+    raw_uid = payload.get("user_id") or payload.get("sub")
+    try:
+        user_id = int(raw_uid) if raw_uid is not None else None
+    except (TypeError, ValueError):
+        user_id = None
+    username = payload.get("username")
+    return user_id, (str(username) if username else None)
+
+
+def _is_agent_call(scope: Scope) -> bool:
+    """True when the caller authenticated as an AGENT rather than as a user.
+
+    Every call the installed agent makes carries `X-API-Key` and never an
+    `Authorization` header (linux_install.sh / macos_install.sh: heartbeat,
+    config, pending-requests, upgrade-status, keepalived-*, config-response are
+    all `-H "X-API-Key: $AGENT_TOKEN"`), while the UI carries a JWT and never an
+    agent key. The one endpoint that accepts either -
+    `POST /api/agents/generate-install-script`, used by agent self-upgrade -
+    is correctly classified by the same rule: an operator generating a script
+    sends Authorization, the self-upgrading agent sends only the key.
+
+    Header-only, so it costs two scope reads and no database round-trip.
+    """
+    if _header(scope, b"authorization"):
+        return False
+    return bool(_header(scope, b"x-api-key"))
+
+
+def _client_ip(scope: Scope) -> Optional[str]:
+    """The peer address only.
+
+    `request_logs.client_ip` is an INET column, so a comma-joined
+    X-Forwarded-For string would raise on INSERT (the same trap as
+    `user_activity_logs.ip_address`). The XFF header is still captured — it is
+    on the header allowlist — so the original client is not lost behind a proxy.
+    """
+    client = scope.get("client")
+    if not client:
+        return None
+    try:
+        return str(client[0])
+    except Exception:
+        return None
+
+
+class RequestResponseLogMiddleware:
+    def __init__(self, app: ASGIApp):
+        self.app = app
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope.get("type") != "http":
+            await self.app(scope, receive, send)
+            return
+
+        cfg = get_config()
+        path = scope.get("path", "") or ""
+        method = scope.get("method", "") or ""
+
+        if (
+            not cfg.enabled
+            or not cfg.capture_inbound
+            # OPTIONS never reaches a handler — CORSMiddleware short-circuits
+            # it below us — and a preflight carries no information worth a row.
+            or method == "OPTIONS"
+            or (method == "GET" and not cfg.capture_get)
+            or any(path.startswith(prefix) for prefix in _ALWAYS_EXCLUDED)
+            or any(path.startswith(prefix) for prefix in cfg.exclude_paths)
+        ):
+            await self.app(scope, receive, send)
+            return
+
+        request_id = uuid.uuid4().hex
+        # Seed the id BEFORE the downstream app runs so error_handler's
+        # get_correlation_id() adopts ours instead of minting a second one; the
+        # X-Correlation-ID header then matches request_logs.request_id.
+        cid_token = correlation_id_context.set(request_id[:8])
+        rid_token = request_id_context.set(request_id)
+
+        cap = cfg.max_body_bytes if cfg.capture_bodies else 0
+        req_ctype = _header(scope, b"content-type")
+        req_capturable = is_capturable_content_type(req_ctype)
+
+        req_buf = bytearray()
+        res_buf = bytearray()
+        state = {
+            "req_bytes": 0,
+            "res_bytes": 0,
+            "status": None,
+            "res_headers": {},
+            "res_ctype": None,
+            "res_capturable": True,
+        }
+
+        async def tee_receive() -> Dict[str, Any]:
+            message = await receive()
+            try:
+                if message.get("type") == "http.request":
+                    chunk = message.get("body", b"") or b""
+                    state["req_bytes"] += len(chunk)
+                    if cap and req_capturable and len(req_buf) < cap:
+                        req_buf.extend(chunk[: cap - len(req_buf)])
+            except Exception:
+                pass
+            return message  # forwarded verbatim, always
+
+        async def tee_send(message: Dict[str, Any]) -> None:
+            try:
+                mtype = message.get("type")
+                if mtype == "http.response.start":
+                    state["status"] = message.get("status")
+                    raw_headers: List[Tuple[bytes, bytes]] = message.get("headers") or []
+                    headers = {}
+                    for key, value in raw_headers:
+                        try:
+                            headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+                        except Exception:
+                            continue
+                    state["res_headers"] = headers
+                    state["res_ctype"] = headers.get("content-type")
+                    state["res_capturable"] = is_capturable_content_type(state["res_ctype"])
+                    # Hand the id to the client so a user reporting a problem can
+                    # quote it and an operator can find the exact row.
+                    if isinstance(raw_headers, list):
+                        raw_headers.append((b"x-request-id", request_id.encode("latin-1")))
+                elif mtype == "http.response.body":
+                    chunk = message.get("body", b"") or b""
+                    state["res_bytes"] += len(chunk)
+                    if cap and state["res_capturable"] and len(res_buf) < cap:
+                        res_buf.extend(chunk[: cap - len(res_buf)])
+            except Exception:
+                pass
+            await send(message)  # forwarded verbatim, always
+
+        started = time.perf_counter()
+        error_text: Optional[str] = None
+        try:
+            await self.app(scope, tee_receive, tee_send)
+        except Exception as exc:
+            # Almost never taken: RequestLoggingMiddleware sits below us and
+            # converts exceptions into a JSONResponse first. It IS taken for
+            # paths on that middleware's own exclude list, so the row still has
+            # to be recorded before the exception continues upward.
+            error_text = f"{type(exc).__name__}: {exc}"[:2000]
+            raise
+        finally:
+            duration_ms = int((time.perf_counter() - started) * 1000)
+            try:
+                self._record(
+                    scope=scope,
+                    request_id=request_id,
+                    method=method,
+                    path=path,
+                    duration_ms=duration_ms,
+                    status=state["status"],
+                    req_buf=bytes(req_buf),
+                    req_bytes=state["req_bytes"],
+                    req_ctype=req_ctype,
+                    res_buf=bytes(res_buf),
+                    res_bytes=state["res_bytes"],
+                    res_ctype=state["res_ctype"],
+                    res_headers=state["res_headers"],
+                    error_text=error_text,
+                )
+            except Exception as exc:  # pragma: no cover - defensive
+                logger.debug(f"request_log: failed to record inbound row: {exc}")
+            try:
+                correlation_id_context.reset(cid_token)
+                request_id_context.reset(rid_token)
+            except Exception:
+                pass
+
+    @staticmethod
+    def _record(
+        *,
+        scope: Scope,
+        request_id: str,
+        method: str,
+        path: str,
+        duration_ms: int,
+        status: Optional[int],
+        req_buf: bytes,
+        req_bytes: int,
+        req_ctype: Optional[str],
+        res_buf: bytes,
+        res_bytes: int,
+        res_ctype: Optional[str],
+        res_headers: Dict[str, str],
+        error_text: Optional[str],
+    ) -> None:
+        raw_query = scope.get("query_string") or b""
+        try:
+            query = raw_query.decode("latin-1")
+        except Exception:
+            query = ""
+        scrubbed_query, query_params = scrub_query_string(query)
+
+        user_id, username = _identify(scope)
+
+        req_headers = {}
+        for key, value in scope.get("headers") or ():
+            try:
+                req_headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+            except Exception:
+                continue
+
+        request_log_sink.offer(
+            RequestLogRow(
+                request_id=request_id,
+                direction="inbound",
+                # Who was on the other end. `offer()` uses this to drop
+                # SUCCESSFUL agent polls, which are ~9 800 rows/day per node and
+                # would otherwise make the table's size a function of fleet size.
+                target=TARGET_INBOUND_AGENT if _is_agent_call(scope) else None,
+                method=method,
+                url=path + (("?" + scrubbed_query) if scrubbed_query else ""),
+                path=path,
+                query_string=scrubbed_query or None,
+                query_params=query_params,
+                status_code=status,
+                duration_ms=duration_ms,
+                user_id=user_id,
+                username=username,
+                client_ip=_client_ip(scope),
+                user_agent=req_headers.get("user-agent"),
+                request_headers=req_headers or None,
+                response_headers=res_headers or None,
+                request_body_raw=req_buf or None,
+                request_body_bytes=req_bytes,
+                request_content_type=req_ctype,
+                response_body_raw=res_buf or None,
+                response_body_bytes=res_bytes,
+                response_content_type=res_ctype,
+                error=error_text,
+            )
+        )

backend/models/cluster.py

@@ -1,6 +1,22 @@
-from pydantic import BaseModel
+from pydantic import BaseModel, field_validator
 from typing import Optional, List
 
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
+
+
+def _validated_acme_backend_url(value: Optional[str]) -> Optional[str]:
+    """Shared field validator body for `acme_backend_url`.
+
+    Pydantic turns the raised ValueError into a 422 with this message attached, so
+    the operator sees why the value was refused instead of discovering months later
+    that HTTP-01 never worked. Returns the normalised value — callers persist THIS,
+    not the raw input, so surrounding whitespace never reaches haproxy.cfg.
+    """
+    try:
+        return validate_acme_backend_url(value)
+    except AcmeBackendUrlError as exc:
+        raise ValueError(str(exc)) from None
+
 class HAProxyClusterCreate(BaseModel):
     name: str
     description: Optional[str] = None
@@ -8,7 +24,18 @@ class HAProxyClusterCreate(BaseModel):
     stats_socket_path: str = "/run/haproxy/admin.sock"
     haproxy_config_path: str = "/etc/haproxy/haproxy.cfg"
     haproxy_bin_path: str = "/usr/sbin/haproxy"  # HAProxy binary path
+    keepalived_config_path: str = "/etc/keepalived/keepalived.conf"  # HA/VIP: keepalived.conf path (Issue #27)
     pool_id: Optional[int] = None  # Which pool this cluster belongs to
+    # The create form submits both of these. Until they were declared here pydantic
+    # dropped them and the INSERT never carried them, so a cluster created with ACME
+    # switched on came back switched off with no error shown — the same silent-success
+    # failure this work exists to remove.
+    acme_enabled: Optional[bool] = None
+    acme_backend_url: Optional[str] = None
+
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
 
 class HAProxyClusterUpdate(BaseModel):
     name: Optional[str] = None
@@ -17,11 +44,16 @@ class HAProxyClusterUpdate(BaseModel):
     stats_socket_path: Optional[str] = None
     haproxy_config_path: Optional[str] = None
     haproxy_bin_path: Optional[str] = None
+    keepalived_config_path: Optional[str] = None
     pool_id: Optional[int] = None
     is_active: Optional[bool] = None
     acme_enabled: Optional[bool] = None
     acme_backend_url: Optional[str] = None
 
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
+
 class HAProxyClusterResponse(BaseModel):
     id: int
     name: str

backend/models/csr.py

@@ -0,0 +1,251 @@
+"""
+Pydantic models for the CSR (Certificate Signing Request) feature (v1.9.0).
+
+A CSR row is the precursor of an ssl_certificates row: the backend generates
+the private key + CSR locally, the operator has the CSR signed by an external
+CA and then imports the signed certificate. The CSR `name` therefore obeys the
+exact same path-traversal contract as the SSL certificate name (Bulgu #21) —
+at import time it becomes /etc/ssl/haproxy/{name}.pem on every agent and is
+shell-processed by the agent script as root.
+
+The import model deliberately has NO private key field: the key never leaves
+the server. It is stored on the ssl_csrs row at generation time and paired
+with the signed certificate server-side.
+"""
+
+import re
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+KEY_ALGORITHMS = ('rsa-2048', 'rsa-4096', 'ecdsa-p256', 'ecdsa-p384')
+
+# RFC 1035 LDH hostname, lowercase, optional single leftmost wildcard label.
+# Single-label names are allowed (internal CAs routinely sign bare hostnames).
+_DNS_NAME_PATTERN = re.compile(
+    r'^(\*\.)?[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?'
+    r'(\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)*$'
+)
+
+# Reject control characters in free-text subject fields: they would be
+# persisted, echoed into the UI / issuer column, and printed into agent logs
+# via `openssl -subject` output.
+_CONTROL_CHARS_PATTERN = re.compile(r'[\x00-\x1f\x7f]')
+
+_MAX_SANS = 100
+_MAX_CERT_PEM_BYTES = 64 * 1024       # a leaf certificate is ~2 KB; 64 KB is generous
+_MAX_CHAIN_PEM_BYTES = 256 * 1024     # agents re-download all cert content every poll
+
+
+def _validate_dns_name(value: str, field_label: str) -> str:
+    v = (value or '').strip().lower()
+    if not v:
+        raise ValueError(f'{field_label} must not be empty')
+    if len(v) > 253:
+        raise ValueError(f'{field_label} must be 253 characters or fewer')
+    if not _DNS_NAME_PATTERN.match(v):
+        raise ValueError(
+            f'{field_label} {value!r} is not a valid DNS name — lowercase '
+            'letters, digits, hyphens and dots only; a wildcard is allowed '
+            'only as the leftmost label (e.g. *.example.com).'
+        )
+    return v
+
+
+def _validate_subject_text(value: Optional[str], field_label: str, max_len: int = 64) -> Optional[str]:
+    if value is None:
+        return None
+    v = value.strip()
+    if not v:
+        return None
+    if len(v) > max_len:
+        raise ValueError(f'{field_label} must be {max_len} characters or fewer')
+    if _CONTROL_CHARS_PATTERN.search(v):
+        raise ValueError(f'{field_label} must not contain control characters')
+    return v
+
+
+def _validate_csr_name(v: str) -> str:
+    """Mirror of SSLCertificateCreate.validate_name_no_path_traversal (Bulgu #21)
+    with one deliberate tightening: max length 100, matching the
+    ssl_certificates.name VARCHAR(100) column (the historical 200-char limit
+    overflows the column and 500s — not replicated here)."""
+    if v is None:
+        raise ValueError('CSR name is required')
+    stripped = v.strip()
+    if not stripped:
+        raise ValueError('CSR name must not be empty')
+    if stripped != v:
+        raise ValueError('CSR name must not contain leading/trailing whitespace')
+    if len(stripped) > 100:
+        raise ValueError('CSR name must be 100 characters or fewer')
+    if not re.match(r'^[A-Za-z0-9_.-]+$', stripped):
+        raise ValueError(
+            f'CSR name={v!r} contains forbidden characters — only letters, '
+            'digits, underscore, hyphen, and dot are allowed (the name becomes '
+            'a filename component under /etc/ssl/haproxy/ at import).'
+        )
+    if '..' in stripped:
+        raise ValueError(f'CSR name={v!r} must not contain ".." (path traversal)')
+    if stripped.startswith('.'):
+        raise ValueError(f'CSR name={v!r} must not start with "." (hidden filename)')
+    if stripped.startswith('-'):
+        raise ValueError(f'CSR name={v!r} must not start with "-" (CLI flag confusion)')
+    return stripped
+
+
+class SSLCSRCreate(BaseModel):
+    name: str                                   # becomes the certificate name at import
+    common_name: str
+    organization: Optional[str] = None          # O
+    organizational_unit: Optional[str] = None   # OU
+    locality: Optional[str] = None              # L
+    state: Optional[str] = None                 # ST
+    country: Optional[str] = None               # C — exactly 2 letters
+    email: Optional[str] = None                 # emailAddress
+    sans: List[str] = []                        # DNS names; CN is auto-added server-side
+    key_algorithm: str = 'rsa-2048'
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        return _validate_csr_name(v)
+
+    @field_validator('common_name')
+    @classmethod
+    def validate_common_name(cls, v):
+        v = _validate_dns_name(v, 'Common Name')
+        # RFC 5280 ub-common-name — many CAs reject CNs longer than 64 chars.
+        if len(v) > 64:
+            raise ValueError(
+                'Common Name must be 64 characters or fewer (RFC 5280 upper '
+                'bound) — put longer names in the SAN list instead.'
+            )
+        return v
+
+    @field_validator('sans')
+    @classmethod
+    def validate_sans(cls, v):
+        if not v:
+            return []
+        if len(v) > _MAX_SANS:
+            raise ValueError(f'At most {_MAX_SANS} SAN entries are allowed')
+        seen = set()
+        result = []
+        for entry in v:
+            normalised = _validate_dns_name(entry, 'SAN entry')
+            if normalised not in seen:
+                seen.add(normalised)
+                result.append(normalised)
+        return result
+
+    @field_validator('organization')
+    @classmethod
+    def validate_organization(cls, v):
+        return _validate_subject_text(v, 'Organization (O)')
+
+    @field_validator('organizational_unit')
+    @classmethod
+    def validate_organizational_unit(cls, v):
+        return _validate_subject_text(v, 'Organizational Unit (OU)')
+
+    @field_validator('locality')
+    @classmethod
+    def validate_locality(cls, v):
+        return _validate_subject_text(v, 'Locality (L)')
+
+    @field_validator('state')
+    @classmethod
+    def validate_state(cls, v):
+        return _validate_subject_text(v, 'State/Province (ST)')
+
+    @field_validator('country')
+    @classmethod
+    def validate_country(cls, v):
+        # cryptography raises a bare ValueError for a non-2-char COUNTRY_NAME;
+        # pre-validate so the operator gets a friendly 422 instead of a 500.
+        if v is None:
+            return None
+        v = v.strip()
+        if not v:
+            return None
+        if not re.match(r'^[A-Za-z]{2}$', v):
+            raise ValueError('Country (C) must be exactly 2 letters (ISO 3166-1 alpha-2, e.g. TR, US)')
+        return v.upper()
+
+    @field_validator('email')
+    @classmethod
+    def validate_email(cls, v):
+        v = _validate_subject_text(v, 'Email', max_len=254)
+        if v is not None and ('@' not in v or v.startswith('@') or v.endswith('@')):
+            raise ValueError('Email must be a valid address (missing or misplaced "@")')
+        return v
+
+    @field_validator('key_algorithm')
+    @classmethod
+    def validate_key_algorithm(cls, v):
+        if v not in KEY_ALGORITHMS:
+            raise ValueError(
+                f'key_algorithm must be one of: {", ".join(KEY_ALGORITHMS)}'
+            )
+        return v
+
+
+class SSLCSRImport(BaseModel):
+    """Import the CA-signed certificate for a pending CSR. The private key is
+    NOT part of the request — it is already stored on the CSR row."""
+    certificate_content: str                    # PEM
+    chain_content: Optional[str] = None         # PEM, optional
+    usage_type: str = 'frontend'                # "frontend" or "server"
+    is_global: bool = False
+    cluster_ids: Optional[List[int]] = None
+    # Escape hatch for name collisions that appeared AFTER the CSR was
+    # created: overrides the CSR's reserved name for the certificate row.
+    name: Optional[str] = None
+
+    @field_validator('certificate_content')
+    @classmethod
+    def validate_certificate(cls, v):
+        if not v or not v.strip():
+            raise ValueError('Certificate content is required')
+        v = v.strip()
+        if len(v.encode('utf-8', errors='ignore')) > _MAX_CERT_PEM_BYTES:
+            raise ValueError('Certificate content exceeds the 64 KB limit')
+        if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+            raise ValueError('Certificate must be in PEM format')
+        return v
+
+    @field_validator('chain_content')
+    @classmethod
+    def validate_chain(cls, v):
+        if v and v.strip():
+            v = v.strip()
+            if len(v.encode('utf-8', errors='ignore')) > _MAX_CHAIN_PEM_BYTES:
+                raise ValueError('Certificate chain exceeds the 256 KB limit')
+            if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+                raise ValueError('Certificate chain must be in PEM format')
+            return v
+        return None
+
+    @field_validator('usage_type')
+    @classmethod
+    def validate_usage_type(cls, v):
+        if v not in ['frontend', 'server']:
+            raise ValueError('usage_type must be either "frontend" or "server"')
+        return v
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        if v is None or not str(v).strip():
+            return None
+        return _validate_csr_name(v)
+
+    @model_validator(mode='after')
+    def validate_cluster_selection(self):
+        if not self.is_global and not self.cluster_ids:
+            raise ValueError(
+                'cluster_ids is required when is_global is false — pick at '
+                'least one cluster or import the certificate as global.'
+            )
+        return self

backend/models/frontend.py

@@ -85,6 +85,11 @@ class FrontendConfig(BaseModel):
     response_headers: Optional[str] = None
     options: Optional[str] = None
     tcp_request_rules: Optional[str] = None
+    # Issue #38: SPOE filter directives (Coraza WAF etc.) + frontend log-format.
+    # Passthrough TEXT (no validator) — SPOE `filter ... config <path>` legitimately
+    # references an operator-managed file, so the ACL `-f` guard must NOT apply here.
+    log_format: Optional[str] = None
+    filters: Optional[str] = None
     timeout_client: Optional[int] = None
     timeout_http_request: Optional[int] = None
     rate_limit: Optional[int] = None
@@ -451,26 +456,17 @@ class FrontendConfig(BaseModel):
             if any(dangerous in rule.lower() for dangerous in ['$(', '`']):
                 raise ValueError(f'ACL rule contains potentially dangerous content: "{rule}"')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) — reject
-            # the HAProxy `-f <file>` pattern-file flag here too so the
-            # manual Frontend API mirrors the wizard's parity rule.
-            # HAProxy OpenManager does not provision pattern files
-            # onto the HAProxy node filesystem, so any `-f /path/...`
-            # reference will fail HAProxy's `-c` parse at apply time
-            # with "failed to open pattern file". Reject up-front so
-            # operators get the same actionable error from both the
-            # manual page and the wizard.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'ACL rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not "
-                    "provision pattern files onto the HAProxy node "
-                    "filesystem, so the reference would fail at "
-                    "reload time. Use inline values instead "
-                    "(e.g. `src 10.0.0.0/24` rather than "
-                    "`src -f /etc/haproxy/admins.lst`)."
-                )
+            # Issue #38 follow-up — the `-f <file>` pattern-file flag
+            # is ACCEPTED here (the Bulgu #12 hard reject was removed).
+            # Pattern files are operator-managed host files, exactly
+            # like the SPOE `filter ... config <path>` reference this
+            # release started preserving: bulk import always accepted
+            # `-f`, the free-form fields (request_headers,
+            # tcp_request_rules) always accepted it, and the agent
+            # runs `haproxy -c` before every reload so a missing file
+            # fails safely (previous config keeps running). The route
+            # handlers surface a non-blocking warning listing the
+            # referenced pattern files instead.
 
             validated_rules.append(rule)
 
@@ -510,20 +506,12 @@ class FrontendConfig(BaseModel):
             if not any(rule.startswith(redirect_type) for redirect_type in valid_redirects):
                 raise ValueError(f'Invalid redirect rule: "{rule}". Must start with: location, prefix, or scheme.')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) —
-            # mirror the wizard's `-f <file>` guard here. The
-            # `X !X` contradiction check used to live alongside
-            # this guard, but Bulgu #62 (round-22 audit) moved
-            # it into the route handler so updates can grandfather
-            # legacy rules created before the contradiction guard
-            # landed. See `routers/frontend.py::_collect_routing_rule_contradictions`.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'Redirect rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not provision "
-                    "pattern files onto the HAProxy node filesystem."
-                )
+            # Issue #38 follow-up — `-f <file>` pattern-file references
+            # are ACCEPTED (Bulgu #12 hard reject removed; see
+            # validate_acl_rules for the full rationale). The `X !X`
+            # contradiction check lives in the route handler
+            # (`routers/frontend.py::_collect_routing_rule_contradictions`,
+            # Bulgu #62) and is unchanged.
 
             validated_rules.append(rule)
 
@@ -531,9 +519,12 @@ class FrontendConfig(BaseModel):
 
     @validator('use_backend_rules')
     def validate_use_backend_rules_syntax(cls, v):
-        """Phase K Phase D follow-up (Bulgu #12 round 3) — manual
-        Frontend API parity guard: reject `-f <file>` references
-        and dangerous shell patterns.
+        """Manual Frontend API guard for dangerous shell patterns.
+
+        Issue #38 follow-up — the Bulgu #12 `-f <file>` hard reject
+        was removed (see validate_acl_rules for the rationale);
+        pattern-file references are operator-managed host files and
+        are surfaced as non-blocking warnings by the route handlers.
 
         Bulgu #62 (round-22 audit) — the `X !X` contradiction check
         previously lived here but moved into the route handler so
@@ -563,13 +554,5 @@ class FrontendConfig(BaseModel):
                     f'use_backend rule contains potentially dangerous '
                     f'content: "{rule}"'
                 )
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'use_backend rule "{rule}" uses the HAProxy '
-                    "`-f <file>` pattern-file flag, which is not "
-                    "supported in HAProxy OpenManager: the product "
-                    "does not provision pattern files onto the HAProxy "
-                    "node filesystem."
-                )
             validated_rules.append(rule)
         return validated_rules

backend/models/mfa.py

@@ -0,0 +1,73 @@
+"""MFA-specific Pydantic models — Issue #18, v1.6.0.
+
+Kept in a separate module so the existing User / UserUpdate contracts in
+``backend/models/user.py`` stay byte-identical for backwards compatibility.
+"""
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+class MfaVerifyRequest(BaseModel):
+    """Body of POST /api/auth/login/mfa-verify (pre-auth — no JWT)."""
+
+    mfa_token: str = Field(..., min_length=64, max_length=64)
+    # 6 digits for TOTP or 8 alphanumerics (with optional dash) for backup codes
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaEnrollStartResponse(BaseModel):
+    """Returned by POST /api/mfa/enroll/start."""
+
+    secret: str
+    otpauth_uri: str
+    expires_in: int  # pending enrollment TTL in seconds
+
+
+class MfaEnrollConfirmRequest(BaseModel):
+    """Body of POST /api/mfa/enroll/confirm (TOTP only — backup codes not yet issued)."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaEnrollConfirmResponse(BaseModel):
+    enabled: bool
+    backup_codes: List[str]
+    method: Literal["totp"] = "totp"
+
+
+class MfaDisableRequest(BaseModel):
+    """Body of POST /api/mfa/disable — TOTP or backup."""
+
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaRegenerateBackupRequest(BaseModel):
+    """Body of POST /api/mfa/backup-codes/regenerate — TOTP only."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaRegenerateBackupResponse(BaseModel):
+    backup_codes: List[str]
+
+
+class MfaAdminResetRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset/{user_id}."""
+
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaAdminResetAllRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset-all (emergency)."""
+
+    confirm: Literal["RESET ALL MFA"]
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaStatusResponse(BaseModel):
+    enabled: bool
+    method: Optional[str] = None
+    enrolled_at: Optional[str] = None
+    last_used_at: Optional[str] = None
+    backup_codes_remaining: int = 0

backend/models/site_wizard.py

@@ -179,35 +179,17 @@ _MAX_RULE_STRING_LEN = 4096
 # attempts.
 _DANGEROUS_RULE_PATTERNS = ("$(", "`")
 
-# Phase K Phase D follow-up (Bulgu #12 round 3) — the HAProxy `-f
-# <file>` ACL/condition flag instructs HAProxy to load match patterns
-# from a server-side file at parse time. HAProxy OpenManager is a
-# fully-managed product: we do NOT provision pattern files onto the
-# HAProxy node's filesystem, and operators have no UI to upload one.
-# A `-f /some/path` reference therefore ALWAYS resolves to
-# "file not found" when HAProxy's real `-c` parse runs at apply
-# time, producing exactly the operator-reported failure mode:
-#   [ALERT] parsing ACL 'acl1' : failed to open pattern file </path>.
-#   [ALERT] parsing switching rule : no such ACL : 'acl1'.
-#
-# Surface this BEFORE persist by rejecting `-f` in any rule string
-# that comes through the wizard / manual frontend API. Reject ALL
-# variants (` -f `, leading `-f `, trailing `... -f`) defensively so
-# operators cannot slip the flag through with creative spacing.
-# The check is anchored to ACL/condition rule strings only; raw
-# HAProxy snippet fields (tcp_request_rules, request_headers, ...)
-# are NOT touched because those are inherently free-form and
-# advanced operators may legitimately reference pre-provisioned
-# pattern files there.
-_ACL_FILE_FLAG_PATTERN = re.compile(r"(^|\s)-f(\s|$)")
-_ACL_FILE_FLAG_MESSAGE = (
-    "pattern-file references with '-f <file>' are not supported in ACL / "
-    "use_backend / redirect rules: HAProxy OpenManager does not provision "
-    "pattern files onto the HAProxy node's filesystem, so the reference "
-    "would always fail at HAProxy reload time. Use inline values "
-    "instead (e.g. `acl is_admin src 10.0.0.0/24` rather than "
-    "`acl is_admin src -f /etc/haproxy/admins.lst`)."
-)
+# Issue #38 follow-up — the HAProxy `-f <file>` ACL/condition flag
+# loads match patterns from a file on the HAProxy host. The Bulgu #12
+# hard reject (`_ACL_FILE_FLAG_PATTERN`/`_ACL_FILE_FLAG_MESSAGE`) was
+# removed: pattern files are operator-managed host files (exactly like
+# the SPOE `filter ... config <path>` reference preserved since
+# v1.8.8), bulk import and the free-form fields (tcp_request_rules,
+# request_headers) always accepted them, and the agent runs
+# `haproxy -c` before every reload so a missing file fails safely
+# (the previous config keeps running). The manual frontend route
+# handlers emit a non-blocking warning listing referenced pattern
+# files (`routers/frontend.py::_pattern_file_warnings`).
 
 # Phase K Phase D follow-up (Bulgu #13) — detect a routing /
 # redirect rule whose condition references the SAME ACL in both
@@ -305,13 +287,10 @@ def _validate_haproxy_directive_string(
                 f"{field_label} entry contains potentially dangerous content: "
                 f"{pattern!r}"
             )
-    # Phase K Phase D follow-up (Bulgu #12 round 3) — reject the
-    # HAProxy `-f <file>` pattern-file flag because OpenManager does
-    # not manage the HAProxy node filesystem. See the module-level
-    # `_ACL_FILE_FLAG_PATTERN` docstring for the full operator-
-    # reported failure mode this guards against.
-    if _ACL_FILE_FLAG_PATTERN.search(stripped):
-        raise ValueError(f"{field_label}: {_ACL_FILE_FLAG_MESSAGE}")
+    # Issue #38 follow-up — `-f <file>` pattern-file references are
+    # ACCEPTED (Bulgu #12 hard reject removed; see the module-level
+    # `_ACL_FILE_FLAG_PATTERN` comment). The route handlers surface
+    # a non-blocking pattern-file warning instead.
     # Phase K Phase D follow-up (Bulgu #13) — for routing /
     # redirect rules (not ACL definitions themselves), reject a
     # condition that contains the same ACL in both positive and
@@ -1083,21 +1062,12 @@ class FrontendStep(BaseModel):
         normalised: List[Union[str, dict]] = []
         for el in v:
             if isinstance(el, dict):
-                # Phase K Phase D follow-up (Bulgu #12 round 3
-                # extension) — dict-shaped redirect rules emit their
-                # `condition` / `target` fields VERBATIM into the
-                # rendered HAProxy directive. A dict with
-                # `condition: "if { src -f /etc/haproxy/x.lst }"`
-                # would slip past the string-only validator above
-                # and trigger the same operator-reported "failed to
-                # open pattern file" rejection at apply time. Reject
-                # `-f` in any string-shaped value the dict carries.
-                for field_name in ("condition", "target", "type"):
-                    val = el.get(field_name)
-                    if isinstance(val, str) and _ACL_FILE_FLAG_PATTERN.search(val):
-                        raise ValueError(
-                            f"redirect_rules.{field_name}: {_ACL_FILE_FLAG_MESSAGE}"
-                        )
+                # Issue #38 follow-up — dict-shaped redirect rules may
+                # carry `-f <file>` pattern-file references in their
+                # `condition`/`target` values; these are ACCEPTED now
+                # (Bulgu #12 hard reject removed — operator-managed
+                # host files, fail-safe apply; see module-level
+                # `_ACL_FILE_FLAG_PATTERN` comment).
                 # Bulgu #13 extension — same contradiction guard
                 # for dict-shaped redirect conditions.
                 cond_val = el.get("condition")

backend/models/vip.py

@@ -0,0 +1,245 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Pydantic request/response models for the VIP management API. Field validation
+is strict because several values flow into a generated keepalived.conf and into
+root-run agent commands — we reuse the same FORBIDDEN-metacharacter discipline as
+models/agent.py (Bulgu #81) and validate the VIP as a real IPv4 address.
+
+Pydantic idiom: the project runs pydantic>=2.5; this module uses the v2-native
+@field_validator/@model_validator style (matching models/ssl.py).
+"""
+import ipaddress
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+# Shell/keepalived.conf metacharacters that must never appear in a value that
+# reaches the generated config or a root-run agent command (mirrors
+# models/agent.py:202, the Bulgu #81 convention).
+_FORBIDDEN = set('$`;&|<>"\'\\\n\r\x00*?')
+
+
+def _validate_iface(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('network_interface must be a non-empty string')
+    s = v.strip()
+    if len(s) > 64:
+        raise ValueError('network_interface too long (max 64 chars)')
+    if any(c in _FORBIDDEN for c in s):
+        raise ValueError('network_interface contains a forbidden character')
+    # Linux iface names: letters, digits, and . _ - : @ (vlans/aliases/altnames)
+    import re as _re
+    if not _re.match(r'^[A-Za-z0-9][A-Za-z0-9._:@-]{0,63}$', s):
+        raise ValueError(
+            'network_interface must start alphanumeric and contain only '
+            'letters, digits, and . _ - : @'
+        )
+    return s
+
+
+def _validate_ipv4(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('virtual_ip must be a non-empty string')
+    s = v.strip()
+    try:
+        addr = ipaddress.ip_address(s)
+    except ValueError:
+        raise ValueError(f'virtual_ip={v!r} is not a valid IP address')
+    if addr.version != 4:
+        raise ValueError('virtual_ip must be IPv4 — IPv6 VIPs are not supported yet')
+    return s
+
+
+class VIPMemberIn(BaseModel):
+    agent_id: int
+    network_interface: str
+    role: str = 'BACKUP'
+    priority: int = 100
+
+    @field_validator('network_interface')
+    @classmethod
+    def _iface(cls, v):
+        return _validate_iface(v)
+
+    @field_validator('role')
+    @classmethod
+    def _role(cls, v):
+        u = (v or '').strip().upper()
+        if u not in ('MASTER', 'BACKUP'):
+            raise ValueError("role must be 'MASTER' or 'BACKUP'")
+        return u
+
+    @field_validator('priority')
+    @classmethod
+    def _priority(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 254):
+            raise ValueError('priority must be an integer between 1 and 254')
+        return v
+
+
+def _validate_members(members: List['VIPMemberIn']) -> List['VIPMemberIn']:
+    # >=1 node: a single-node VIP is a keepalived-managed floating IP without failover
+    # (valid, e.g. a one-box HAProxy that wants a stable VIP, or before a 2nd node is added).
+    # Two or more nodes give actual VRRP failover. The UI flags the single-node case.
+    if not members:
+        raise ValueError('a VIP needs at least 1 member node')
+    agent_ids = [m.agent_id for m in members]
+    if len(set(agent_ids)) != len(agent_ids):
+        raise ValueError('each node may appear at most once in a VIP')
+    masters = [m for m in members if m.role == 'MASTER']
+    if len(masters) != 1:
+        raise ValueError('exactly one member must be MASTER')
+    master_prio = masters[0].priority
+    if any(m.role == 'BACKUP' and m.priority >= master_prio for m in members):
+        raise ValueError('the MASTER must have a strictly higher priority than every BACKUP')
+    return members
+
+
+def _validate_auth_pass(v: Optional[str]) -> Optional[str]:
+    if v is None or v == '':
+        return None
+    # keepalived PASS auth_pass is silently truncated to 8 chars (B-3) — reject longer
+    # so MASTER/BACKUP never silently disagree.
+    if not (1 <= len(v) <= 8):
+        raise ValueError('auth_pass must be 1-8 characters (keepalived PASS limit)')
+    if any(c in _FORBIDDEN for c in v):
+        raise ValueError('auth_pass contains a forbidden character')
+    # No whitespace: keepalived PASS auth_pass is a single token, and a whitespace-containing
+    # secret would only partially redact in the masked config diff (review HIGH-2).
+    if any(c.isspace() for c in v):
+        raise ValueError('auth_pass must not contain whitespace')
+    return v
+
+
+class VIPCreate(BaseModel):
+    name: str
+    description: Optional[str] = None
+    pool_id: int
+    virtual_ip: str
+    prefix_length: int = 24
+    virtual_router_id: Optional[int] = None   # auto-allocated within the pool when omitted
+    advert_int: int = 1
+    auth_pass: Optional[str] = None           # plaintext on the wire; stored Fernet-encrypted
+    use_unicast: bool = True
+    track_haproxy: bool = True
+    members: List[VIPMemberIn]
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if not isinstance(v, str) or not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255:
+            raise ValueError('name too long (max 255 chars)')
+        if any(c in _FORBIDDEN for c in s):
+            raise ValueError('name contains a forbidden character')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v)
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 32):
+            raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be an integer between 1 and 255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('advert_int must be an integer between 1 and 255 (seconds)')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        _validate_members(self.members)
+        return self
+
+
+class VIPUpdate(BaseModel):
+    """All fields optional — only provided fields are changed. Any change sets the
+    VIP back to last_config_status='PENDING' (router-side)."""
+    name: Optional[str] = None
+    description: Optional[str] = None
+    virtual_ip: Optional[str] = None
+    prefix_length: Optional[int] = None
+    virtual_router_id: Optional[int] = None
+    advert_int: Optional[int] = None
+    auth_pass: Optional[str] = None           # provide only to rotate; omit to keep existing
+    use_unicast: Optional[bool] = None
+    track_haproxy: Optional[bool] = None
+    members: Optional[List[VIPMemberIn]] = None
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if v is None:
+            return v
+        if not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255 or any(c in _FORBIDDEN for c in s):
+            raise ValueError('name invalid (too long or forbidden character)')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v) if v is not None else v
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 32):
+            raise ValueError('prefix_length must be 1-32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be 1-255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('advert_int must be 1-255 seconds')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        if self.members is not None:
+            _validate_members(self.members)
+        return self

backend/requirements.txt

@@ -15,4 +15,5 @@ bcrypt>=4.0.1
 slowapi>=0.1.9
 psutil>=5.9.8
 pytz>=2023.3
-josepy>=1.14.0
+josepy>=1.14.0
+pyotp>=2.9.0

backend/routers/acme_diagnostics.py

@@ -16,9 +16,11 @@ Per-user 5/min rate-limit via user_activity_logs SQL count (M18 / R50).
 
 import json
 import logging
+import uuid
 from datetime import datetime
 from typing import List, Optional
 
+import asyncpg
 from fastapi import APIRouter, Header, HTTPException
 
 from auth_middleware import check_user_permission, get_current_user_from_token
@@ -58,16 +60,39 @@ async def _enforce_rate_limit(conn, user_id: int, action: str) -> None:
 
 
 async def _load_order(conn, order_id: int) -> dict:
-    row = await conn.fetchrow(
-        """
-        SELECT id, account_id, status, domains, cluster_ids, error_detail,
-               post_completion_actions, pending_apply_version_name,
-               wizard_staged_until, created_by
-        FROM letsencrypt_orders
-        WHERE id = $1
-        """,
-        order_id,
-    )
+    """Fetch the order row, or raise a clean 404.
+
+    Bulgu #96 (prod-canary audit): `letsencrypt_orders.id` is a Postgres
+    int4 column. A path-param `order_id` outside the int4 range
+    (e.g. > 2_147_483_647) used to bubble up as
+    `asyncpg.exceptions.DataError: invalid input for query argument $1:
+    ... (value out of int32 range)` — which the diagnostics endpoint
+    then surfaced in a `diagnostics_unavailable` envelope, leaking the
+    raw Postgres / asyncpg error string ("query argument $1",
+    "int32 range") into the operator-visible response body.
+    Semantically an out-of-range ID can never reference a real order,
+    so we treat it identically to "row not found" and return a clean
+    404 — same shape as the not-found path, no SQL detail leakage.
+    """
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT id, account_id, status, domains, cluster_ids, error_detail,
+                   post_completion_actions, pending_apply_version_name,
+                   wizard_staged_until, created_by, challenge_type
+            FROM letsencrypt_orders
+            WHERE id = $1
+            """,
+            order_id,
+        )
+    except asyncpg.exceptions.DataError as exc:
+        logger.info(
+            "ACME order lookup rejected by Postgres (out-of-range / "
+            "uncastable id): order_id=%s exc=%s",
+            order_id,
+            exc,
+        )
+        raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     if not row:
         raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     return dict(row)
@@ -86,35 +111,154 @@ def _parse_jsonb_list(raw, default):
     return default
 
 
+def _diagnostic_failure_envelope(
+    order_id: int,
+    correlation_id: str,
+    exc: Exception,
+    *,
+    stage: str,
+) -> dict:
+    """Build a structured response when the diagnostic suite itself
+    cannot run. Bulgu #94 (Round-25): we return HTTP 200 with this
+    envelope rather than 500 so the UI can still SHOW the operator
+    what happened — the panel's whole purpose is to surface failure
+    causes, and the panel itself silently 500-ing is the worst-case
+    UX. The server-side log carries the full traceback keyed by
+    correlation_id for operator follow-up.
+    """
+    return {
+        "order_id": order_id,
+        "status": "diagnostics_unavailable",
+        "checks": [
+            {
+                "id": "diagnostics_runner",
+                "label": "Diagnostic runner",
+                "status": "fail",
+                "severity": "error",
+                "message": (
+                    f"Diagnostics could not run ({stage}): "
+                    f"{exc.__class__.__name__}: {exc}"
+                ),
+                "details": {
+                    "stage": stage,
+                    "exception_type": exc.__class__.__name__,
+                    "exception_message": str(exc),
+                    "correlation_id": correlation_id,
+                    "hint": (
+                        "Check the backend log for correlation_id "
+                        f"{correlation_id} for the full traceback."
+                    ),
+                },
+                "duration_ms": None,
+            }
+        ],
+        "humanized_error": {
+            "title": "Diagnostic panel could not run",
+            "message": (
+                "The diagnostic runner itself crashed before any check "
+                "could complete. This is independent of whether the ACME "
+                "provider is reachable from this cluster."
+            ),
+            "hint": (
+                "Share the correlation_id below with the platform team; "
+                "they can grep the API log for the full traceback."
+            ),
+            "correlation_id": correlation_id,
+        },
+        "meta": {
+            "correlation_id": correlation_id,
+            "error_stage": stage,
+            "error_type": exc.__class__.__name__,
+            "error_message": str(exc),
+        },
+        "generated_at": datetime.utcnow().isoformat() + "Z",
+    }
+
+
 @router.post("/orders/{order_id}/diagnostics")
 async def run_diagnostics(order_id: int, authorization: str = Header(None)):
-    """Run the full pre-flight + post-failure diagnostic suite."""
+    """Run the full pre-flight + post-failure diagnostic suite.
+
+    Bulgu #94 (Round-25 audit): this endpoint must NEVER return HTTP 500
+    for an in-suite failure. The diagnostic panel exists precisely to
+    explain what is broken; producing an opaque 500 defeats the entire
+    feature. Authentication / authorisation / rate-limit / not-found
+    errors still raise the appropriate 4xx, but any unexpected
+    exception from check execution is converted to a 200 response with
+    a structured failure envelope so the UI can display the cause.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
         await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostics_run")
         order = await _load_order(conn, order_id)
+    except HTTPException:
+        await close_database_connection(conn)
+        raise
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        logger.exception(
+            "ACME diagnostics setup failed for order=%s correlation_id=%s",
+            order_id,
+            correlation_id,
+        )
+        try:
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="load_order")
+        finally:
+            await close_database_connection(conn)
 
+    try:
         domains = _parse_jsonb_list(order["domains"], [])
         cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
 
-        results = await run_checks(
-            conn,
-            domains=domains,
-            cluster_ids=cluster_ids,
-            account_id=order["account_id"],
-        )
+        try:
+            results = await run_checks(
+                conn,
+                domains=domains,
+                cluster_ids=cluster_ids,
+                account_id=order["account_id"],
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            # run_checks now wraps individual checks, but a top-level
+            # crash (e.g. lost DB connection) still needs to be visible.
+            logger.exception(
+                "ACME diagnostics top-level failure for order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
+            )
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="run_checks")
 
-        humanized_error = humanize_error_detail(order["error_detail"])
+        try:
+            humanized_error = humanize_error_detail(order["error_detail"])
+        except Exception as exc:  # noqa: BLE001 — defensive
+            logger.warning(
+                "humanize_error_detail failed for order=%s correlation_id=%s: %s",
+                order_id,
+                correlation_id,
+                exc,
+            )
+            humanized_error = {
+                "title": "ACME error (raw)",
+                "message": str(order["error_detail"]) if order["error_detail"] else "",
+                "hint": "",
+                "parse_error": exc.__class__.__name__,
+            }
 
         return {
             "order_id": order_id,
             "status": order["status"],
             "checks": results,
             "humanized_error": humanized_error,
+            "meta": {
+                "correlation_id": correlation_id,
+                "checks_total": len(results),
+                "checks_failed": sum(1 for r in results if r.get("status") == "fail"),
+                "checks_warn": sum(1 for r in results if r.get("status") == "warn"),
+            },
             "generated_at": datetime.utcnow().isoformat() + "Z",
         }
     finally:
@@ -127,7 +271,15 @@ async def rerun_diagnostic_check(
     check_id: str,
     authorization: str = Header(None),
 ):
-    """Re-run a single check (DNS / port80 / routing / account / agents)."""
+    """Re-run a single check (DNS / port80 / routing / account / agents).
+
+    Bulgu #94 follow-up (Round-25 audit): the rerun path is just as
+    sensitive to opaque 500s as the full-suite POST. If
+    `_enforce_rate_limit` / `_load_order` / `run_checks` raises an
+    unexpected exception, we surface a structured `fail` row in the
+    same shape the table already renders — so the operator clicking
+    "Re-run" never sees an opaque toast and the row updates in place.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
@@ -138,30 +290,120 @@ async def rerun_diagnostic_check(
             detail=f"Unknown check_id '{check_id}'. Valid: {', '.join(CHECK_IDS)}",
         )
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
-        await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
-        order = await _load_order(conn, order_id)
+        try:
+            await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
+            order = await _load_order(conn, order_id)
+        except HTTPException:
+            raise
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun setup failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run setup failed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "setup",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "setup"},
+            }
 
-        domains = _parse_jsonb_list(order["domains"], [])
-        cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
-
-        results = await run_checks(
-            conn,
-            domains=domains,
-            cluster_ids=cluster_ids,
-            account_id=order["account_id"],
-            only=[check_id],
-        )
+        try:
+            domains = _parse_jsonb_list(order["domains"], [])
+            cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
+            results = await run_checks(
+                conn,
+                domains=domains,
+                cluster_ids=cluster_ids,
+                account_id=order["account_id"],
+                only=[check_id],
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun run_checks failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run crashed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "run_checks",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "run_checks"},
+            }
 
         return {
             "order_id": order_id,
             "check": results[0] if results else None,
+            "meta": {"correlation_id": correlation_id},
         }
     finally:
         await close_database_connection(conn)
 
 
+async def _user_activity_columns(conn) -> set:
+    """Return the set of column names actually present on user_activity_logs.
+
+    Bulgu #95 (Round-25 audit) — the canonical migration for
+    `user_activity_logs` defines `id, user_id, action, resource_type,
+    resource_id, details, ip_address, user_agent, created_at, timestamp`.
+    There is NO `status` column. The original `/events` SELECT pulled
+    `status` directly, so every diagnostic-panel open against an order
+    that had any user-activity-log correlation raised
+    `UndefinedColumnError: column "status" does not exist` and the API
+    returned HTTP 500. We now introspect the schema and only project
+    the columns that exist, so deployments at any migration level keep
+    rendering the diagnostic panel.
+    """
+    try:
+        rows = await conn.fetch(
+            """
+            SELECT column_name
+            FROM information_schema.columns
+            WHERE table_name = 'user_activity_logs'
+            """
+        )
+        return {r["column_name"] for r in rows}
+    except Exception as exc:  # noqa: BLE001 — schema introspection is best-effort
+        logger.warning("user_activity_logs schema introspection failed: %s", exc)
+        return set()
+
+
 @router.get("/orders/{order_id}/events")
 async def get_order_events(
     order_id: int,
@@ -174,6 +416,12 @@ async def get_order_events(
       resource_id=order_id) for context.
 
     Sorted by created_at ASC (oldest first) so the timeline reads naturally.
+
+    Bulgu #94/#95 (Round-25 audit): every sub-query is wrapped so that a
+    partial failure (missing column, missing table, malformed JSONB) is
+    surfaced via `meta.errors[]` rather than collapsing the whole panel
+    to HTTP 500. The diagnostic UI is a debugging surface — it must not
+    itself become opaque when one of its data sources is degraded.
     """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
@@ -182,95 +430,168 @@ async def get_order_events(
     if limit <= 0 or limit > 500:
         limit = 100
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
+    section_errors: List[dict] = []
     try:
-        # Existence check
-        await _load_order(conn, order_id)
-
-        # Detect whether acme_order_events exists (zero-impact for envs that
-        # have not yet run the v1.5.0 migration). Returns empty event_log when
-        # not yet present rather than 500-ing.
-        events_table_exists = await conn.fetchval(
-            """
-            SELECT EXISTS (
-                SELECT 1 FROM information_schema.tables WHERE table_name = 'acme_order_events'
+        try:
+            await _load_order(conn, order_id)
+        except HTTPException:
+            raise
+        except Exception as exc:  # noqa: BLE001 — surface, don't 500
+            logger.exception(
+                "ACME events load_order failed order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
             )
-            """
-        )
+            return {
+                "order_id": order_id,
+                "events": [],
+                "count": 0,
+                "meta": {
+                    "correlation_id": correlation_id,
+                    "errors": [
+                        {
+                            "section": "load_order",
+                            "exception_type": exc.__class__.__name__,
+                            "message": str(exc),
+                        }
+                    ],
+                },
+            }
 
         events: List[dict] = []
-        if events_table_exists:
-            event_rows = await conn.fetch(
+
+        # --- Section 1: acme_order_events ---
+        try:
+            events_table_exists = await conn.fetchval(
                 """
-                SELECT id, event_type, severity, message, details, correlation_id, created_at
-                FROM acme_order_events
-                WHERE order_id = $1
-                ORDER BY created_at ASC, id ASC
-                LIMIT $2
-                """,
+                SELECT EXISTS (
+                    SELECT 1 FROM information_schema.tables
+                    WHERE table_name = 'acme_order_events'
+                )
+                """
+            )
+            if events_table_exists:
+                event_rows = await conn.fetch(
+                    """
+                    SELECT id, event_type, severity, message, details, correlation_id, created_at
+                    FROM acme_order_events
+                    WHERE order_id = $1
+                    ORDER BY created_at ASC, id ASC
+                    LIMIT $2
+                    """,
+                    order_id,
+                    limit,
+                )
+                for r in event_rows:
+                    # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
+                    # raw JSON strings (no codec on the pool). For the FE
+                    # contract the `details` field MUST be either a dict or
+                    # null — otherwise the React renderer ends up trying to
+                    # access `details.foo` on a plain string and silently
+                    # gets undefined.
+                    _det = r["details"]
+                    if isinstance(_det, str):
+                        try:
+                            _det = json.loads(_det)
+                        except Exception:
+                            _det = {}
+                    if not isinstance(_det, (dict, list)):
+                        _det = {} if _det is None else {"raw": str(_det)}
+                    events.append({
+                        "source": "acme_order_event",
+                        "id": r["id"],
+                        "event_type": r["event_type"],
+                        "severity": r["severity"],
+                        "message": r["message"],
+                        "details": _det,
+                        "correlation_id": r["correlation_id"],
+                        "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
+                            if r["created_at"] else None,
+                    })
+        except Exception as exc:  # noqa: BLE001 — surface and continue
+            logger.exception(
+                "ACME events acme_order_events query failed order=%s correlation_id=%s",
                 order_id,
-                limit,
+                correlation_id,
             )
-            for r in event_rows:
-                # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
-                # raw JSON strings (no codec on the pool). For the FE
-                # contract the `details` field MUST be either a dict or
-                # null — otherwise the React renderer ends up trying to
-                # access `details.foo` on a plain string and silently
-                # gets undefined.
-                _det = r["details"]
-                if isinstance(_det, str):
-                    try:
-                        _det = json.loads(_det)
-                    except Exception:
-                        _det = {}
-                if not isinstance(_det, (dict, list)):
-                    _det = {} if _det is None else {"raw": str(_det)}
-                events.append({
-                    "source": "acme_order_event",
-                    "id": r["id"],
-                    "event_type": r["event_type"],
-                    "severity": r["severity"],
-                    "message": r["message"],
-                    "details": _det,
-                    "correlation_id": r["correlation_id"],
-                    "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
-                        if r["created_at"] else None,
+            section_errors.append({
+                "section": "acme_order_events",
+                "exception_type": exc.__class__.__name__,
+                "message": str(exc),
+            })
+
+        # --- Section 2: user_activity_logs (best-effort, schema-aware) ---
+        try:
+            ua_columns = await _user_activity_columns(conn)
+            if "resource_id" in ua_columns and "resource_type" in ua_columns:
+                # Project only columns we know exist. `status` is NOT in
+                # the canonical schema and was the original 500 cause.
+                projection_candidates = [
+                    "id", "action", "resource_type", "resource_id",
+                    "details", "created_at", "user_id", "status",
+                ]
+                projection = [c for c in projection_candidates if c in ua_columns]
+                if "id" not in projection or "created_at" not in projection:
+                    raise RuntimeError(
+                        "user_activity_logs is missing required columns "
+                        "(id / created_at) — skipping correlation"
+                    )
+                sql = (
+                    f"SELECT {', '.join(projection)} "
+                    "FROM user_activity_logs "
+                    "WHERE resource_type = 'letsencrypt_order' AND resource_id = $1 "
+                    "ORDER BY created_at ASC, id ASC LIMIT $2"
+                )
+                ua_rows = await conn.fetch(sql, str(order_id), limit)
+                for r in ua_rows:
+                    rd = dict(r)
+                    raw_details = rd.get("details")
+                    if isinstance(raw_details, str):
+                        msg = raw_details[:500]
+                        details_obj = {}
+                        try:
+                            parsed = json.loads(raw_details)
+                            if isinstance(parsed, (dict, list)):
+                                details_obj = parsed
+                        except Exception:
+                            details_obj = {}
+                    elif isinstance(raw_details, (dict, list)):
+                        msg = ""
+                        details_obj = raw_details
+                    else:
+                        msg = ""
+                        details_obj = {}
+                    raw_status = rd.get("status") or ""
+                    severity = "info" if str(raw_status).lower() in ("success", "ok", "") else "warn"
+                    events.append({
+                        "source": "user_activity_log",
+                        "id": rd.get("id"),
+                        "event_type": rd.get("action"),
+                        "severity": severity,
+                        "message": msg,
+                        "details": details_obj,
+                        "correlation_id": None,
+                        "created_at": rd["created_at"].isoformat().replace("+00:00", "Z")
+                            if rd.get("created_at") else None,
+                    })
+            else:
+                section_errors.append({
+                    "section": "user_activity_logs",
+                    "exception_type": "SchemaMissing",
+                    "message": "user_activity_logs lacks resource_type/resource_id columns",
                 })
-
-        # User activity rows correlated by resource — schema is permissive
-        # (`resource_type`/`resource_id` may not always be populated for older
-        # rows) so this query stays best-effort.
-        ua_rows = await conn.fetch(
-            """
-            SELECT id, action, resource_type, resource_id, status, details, created_at, user_id
-            FROM user_activity_logs
-            WHERE resource_type = 'letsencrypt_order' AND resource_id = $1
-            ORDER BY created_at ASC, id ASC
-            LIMIT $2
-            """,
-            str(order_id),
-            limit,
-        ) if await conn.fetchval(
-            """
-            SELECT EXISTS (
-                SELECT 1 FROM information_schema.columns
-                WHERE table_name = 'user_activity_logs' AND column_name = 'resource_id'
+        except Exception as exc:  # noqa: BLE001 — surface and continue
+            logger.exception(
+                "ACME events user_activity_logs query failed order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
             )
-            """
-        ) else []
-
-        for r in ua_rows:
-            events.append({
-                "source": "user_activity_log",
-                "id": r["id"],
-                "event_type": r["action"],
-                "severity": "info" if (r["status"] or "").lower() in ("success", "ok", "") else "warn",
-                "message": (r["details"] or "")[:500] if isinstance(r["details"], str) else "",
-                "details": r["details"] if not isinstance(r["details"], (str, type(None))) else {},
-                "correlation_id": None,
-                "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
-                    if r["created_at"] else None,
+            section_errors.append({
+                "section": "user_activity_logs",
+                "exception_type": exc.__class__.__name__,
+                "message": str(exc),
             })
 
         events.sort(key=lambda e: (e["created_at"] or "", e.get("id") or 0))
@@ -279,6 +600,10 @@ async def get_order_events(
             "order_id": order_id,
             "events": events,
             "count": len(events),
+            "meta": {
+                "correlation_id": correlation_id,
+                "errors": section_errors,
+            },
         }
     finally:
         await close_database_connection(conn)

backend/routers/agent.py

@@ -9,8 +9,15 @@ import os
 import json
 import ipaddress
 import hashlib
+import re
 # Pipeline trigger - force backend redeploy v2
 
+# v1.10.4 — a discovered keepalived.conf is stored and served to the UI, so the VRRP password is
+# masked out of the stored copy (the real value lives Fernet-encrypted in its own column). Mask
+# the WHOLE remainder of the line, mirroring vip.py's version-diff masking, so a password
+# containing whitespace cannot partially leak.
+_AUTH_PASS_MASK_RE = re.compile(r"(auth_pass\s+).*")
+
 from models import AgentCreate
 from models.agent import AgentToggle, AgentHeartbeat, AgentScriptRequest, AgentUpgradeRequest
 from database.connection import get_database_connection, close_database_connection
@@ -26,9 +33,43 @@ logger = logging.getLogger(__name__)
 # Global version storage (acts as in-memory database)
 AGENT_VERSIONS = {
     "macos": "2.1.0",  # Updated via endpoint
-    "linux": "2.0.0"
+    "linux": "2.1.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -217,7 +258,7 @@ def calculate_agent_health(status, last_seen):
         return "offline"
 
 @router.get("", summary="Get All Agents", response_description="List of all agents")
-async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None)):
+async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All Agents
     
@@ -269,9 +310,22 @@ async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(
     - **haproxy_status**: Status of HAProxy service on agent's server
     - **last_seen**: Last heartbeat timestamp
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): the agent inventory (names, hostnames, IPs,
+    # pools, OS) is operator data and was previously served unauthenticated — it is
+    # also the read-back channel used in the RCE exfil PoC. Require EITHER a valid
+    # operator JWT OR a valid agent X-API-Key: deployed agents poll this endpoint
+    # (with their key, not a JWT) to read their own applied_config_version and avoid
+    # re-applying config on restart, so a JWT-only gate would break them. Checked
+    # before the try so the 401 is not swallowed by the generic handler.
+    if authorization:
+        current_user = await get_current_user_from_token(authorization)  # raises 401 on invalid JWT
+    else:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         try:
             if pool_id:
                 agents = await conn.fetch("""
@@ -729,15 +783,24 @@ async def generate_uninstall_script(platform: str, authorization: str = Header(N
     sudo ./uninstall-agent.sh
     ```
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): require authentication (operator JWT or agent
+    # key), consistent with generate-install-script. The uninstall script itself is
+    # generic (no secrets/topology), but an agent-management endpoint should not be
+    # anonymously reachable. Checked before the try so the 401 is not swallowed.
+    if authorization:
+        await get_current_user_from_token(authorization)
+    elif not await validate_agent_api_key(x_api_key):
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
-        # Validate platform
-        platform_lower = platform.lower()
-        if platform_lower not in ['linux', 'macos']:
-            raise HTTPException(
-                status_code=400, 
-                detail=f"Invalid platform: {platform}. Must be 'linux' or 'macos'"
-            )
-        
+        # Normalize platform to a canonical key (always 'linux' or 'macos').
+        # macOS agents register with platform 'darwin' (from `uname -s`), so the
+        # strict ['linux','macos'] check used to 400 on the UI's uninstall flow
+        # (GET /generate-uninstall-script/darwin). Reuse the same get_platform_key()
+        # helper the install-script generator uses, so darwin/osx/mac and the
+        # linux distro variants all resolve correctly. Backward-compatible:
+        # 'linux'/'macos' still map to themselves.
+        platform_lower = get_platform_key(platform)
+
         # Read uninstall script from agent_scripts directory (same as install scripts)
         import os
         script_filename = f"uninstall-agent-{platform_lower}.sh"
@@ -861,7 +924,25 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
         # Validate cluster access if agent belongs to a cluster
         if agent['cluster_id']:
             await validate_user_cluster_access(current_user['id'], agent['cluster_id'], conn)
-        
+
+        # HA/VIP (Issue #27): block deleting a node that's still a member of an active VIP.
+        # Otherwise the CASCADE would silently drop it from the VIP (breaking the one-MASTER
+        # topology with no signal) and the still-running node would keep advertising the VIP
+        # with no way to be told to tear down (review MED-3). Make the operator remove it
+        # from the VIP first — that stages a clean PENDING change they can apply.
+        try:
+            vip_member = await conn.fetchrow(
+                "SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+                "WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
+        except Exception:  # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
+            vip_member = None
+        if vip_member:
+            await close_database_connection(conn)
+            raise HTTPException(
+                status_code=409,
+                detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
+                        f"VIP on the HA / VIP page (and apply) before deleting the agent."))
+
         await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
         
         await close_database_connection(conn)
@@ -905,14 +986,24 @@ def _extract_agent_ip(heartbeat_data: AgentHeartbeat) -> Optional[str]:
     return None
 
 @router.post("/{agent_id}/heartbeat")
-async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat):
+async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat, x_api_key: Optional[str] = Header(None)):
     """Receive agent heartbeat and update status."""
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). This legacy by-ID
+    # heartbeat previously had NO auth, allowing unauthenticated state spoofing of
+    # any agent row. Deployed agents use the by-name heartbeat; a valid global
+    # agent token is now required here too. NOTE: raised BEFORE the try below so
+    # the 401 is not swallowed by the generic `except Exception` handler.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on by-id heartbeat for agent ID {agent_id}")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         await conn.execute("""
-            UPDATE agents 
-            SET status = 'online', 
+            UPDATE agents
+            SET status = 'online',
                 last_seen = CURRENT_TIMESTAMP,
                 hostname = COALESCE($2, hostname),
                 haproxy_status = COALESCE($3, haproxy_status),
@@ -1035,6 +1126,8 @@ async def agent_config_applied_notification(agent_name: str, notification_data:
         await close_database_connection(conn)
         return {"status": "ok", "message": "Config applied notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config applied notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1130,6 +1223,8 @@ async def agent_config_validation_failed(agent_name: str, notification_data: dic
         
         return {"status": "ok", "message": "Validation error notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process validation error notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1419,6 +1514,8 @@ async def agent_config_sync(agent_name: str, sync_data: dict, x_api_key: Optiona
         logger.info(f"CONFIG SYNC: Agent '{agent_name}' synced {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers with database")
         return {"status": "ok", "message": f"Config synced - {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers processed"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config sync from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1436,47 +1533,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1493,21 +1576,25 @@ async def agent_heartbeat_by_name(
         logger.error(f"Unexpected error processing heartbeat: {e}")
         raise HTTPException(status_code=500, detail="Internal server error")
     
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). A valid global agent
+    # token is required to heartbeat OR auto-register. Deployed agents always send
+    # X-API-Key; an absent/invalid key is an unauthenticated caller. This is done
+    # OUTSIDE the processing try below (whose generic `except Exception` would
+    # otherwise convert the 401 into a 500), and before opening a DB connection
+    # (validate_agent_api_key(None) needs no DB). Closes keyless heartbeat spoofing
+    # and keyless rogue-agent auto-registration (the `elif not agent` keyless path
+    # below is now unreachable, since agent_auth is guaranteed truthy past here).
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on heartbeat for agent '{heartbeat_data.name}'")
+        raise HTTPException(status_code=401, detail="Authentication required")
+
     # Continue with normal heartbeat processing
     try:
-        # Validate agent API key for security
-        from auth_middleware import validate_agent_api_key
-        agent_auth = await validate_agent_api_key(x_api_key)
-        
         conn = await get_database_connection()
         agent_name = heartbeat_data.name
 
-        # If API key provided, validate it exists but allow placeholder agent updates
-        if x_api_key and not agent_auth:
-            await close_database_connection(conn)
-            logger.warning(f"Invalid API key provided by agent '{agent_name}'")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-
         agent = await conn.fetchrow("SELECT id, pool_id, api_key FROM agents WHERE name = $1", agent_name)
         
         # If agent exists and is using a different API key, update the token association
@@ -1656,9 +1743,13 @@ async def agent_heartbeat_by_name(
             """, x_api_key)
 
         # Check if agent was in upgrading status and version has changed
-        current_agent_status = await conn.fetchval("SELECT status FROM agents WHERE id = $1", agent_id)
-        current_agent_version = await conn.fetchval("SELECT version FROM agents WHERE id = $1", agent_id)
-        current_upgrade_status = await conn.fetchval("SELECT upgrade_status FROM agents WHERE id = $1", agent_id)
+        # (v1.8.6: one round-trip instead of three; row is None exactly when the
+        # per-column fetchvals would each have returned None)
+        current_agent_row = await conn.fetchrow(
+            "SELECT status, version, upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_status = current_agent_row['status'] if current_agent_row else None
+        current_agent_version = current_agent_row['version'] if current_agent_row else None
+        current_upgrade_status = current_agent_row['upgrade_status'] if current_agent_row else None
         
         # Determine new status - preserve upgrading status unless version actually changed
         new_status = current_agent_status or 'online'
@@ -1749,9 +1840,14 @@ async def agent_heartbeat_by_name(
             heartbeat_data.operating_system, heartbeat_data.kernel_version,
             heartbeat_data.uptime, heartbeat_data.cpu_count, heartbeat_data.memory_total,
             heartbeat_data.disk_space, 
-            # Convert lists to JSON for JSONB columns
-            heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str) else json.dumps(heartbeat_data.network_interfaces or []),
-            heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str) else json.dumps(heartbeat_data.capabilities or []),
+            # Convert lists to JSON for JSONB columns. Don't WIPE network_interfaces/capabilities
+            # when a heartbeat omits them: send NULL so COALESCE keeps the existing value (a bare
+            # `or []` would store "[]" and erase the reported NICs / keepalived_management on every
+            # daemon heartbeat that doesn't include them — issue #27 corporate test).
+            (heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str)
+             else (json.dumps(heartbeat_data.network_interfaces) if heartbeat_data.network_interfaces else None)),
+            (heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str)
+             else (json.dumps(heartbeat_data.capabilities) if heartbeat_data.capabilities else None)),
             agent_ip, heartbeat_data.haproxy_status, heartbeat_data.haproxy_version,
             heartbeat_data.applied_config_version, new_status, update_applied_version,
             heartbeat_data.keepalive_state, heartbeat_data.keepalive_ip)
@@ -1907,9 +2003,19 @@ async def agent_heartbeat_by_name(
 @router.get("/{agent_name}/config")
 async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get HAProxy configuration for specific agent"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked BEFORE any
+    # DB work and before the existence check, so an unauthenticated caller learns
+    # neither the full haproxy.cfg nor whether the agent exists. Raised before the
+    # try so it is not swallowed by the generic handler; validate_agent_api_key(None)
+    # returns None without touching the DB.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' config fetch")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent info first to check pool
         # CRITICAL: Include cluster's haproxy_bin_path, haproxy_config_path, stats_socket_path
         # These are needed for dynamic validation - cluster admin can change paths without reinstalling agent
@@ -1921,30 +2027,19 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' config fetch")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching config using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Config fetch authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Config fetch authorized for agent '{agent_name}'")
+
         if not agent_info['enabled']:
             await close_database_connection(conn)
             return {
@@ -2035,9 +2130,18 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
 @router.get("/{agent_name}/ssl-certificates")
 async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = None, x_api_key: Optional[str] = Header(None)):
     """Get SSL certificates for specific agent's cluster"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). This response
+    # returns SSL private_key_content, so authentication is checked BEFORE any DB
+    # work and before the existence check. Raised before the try so the 401 is not
+    # swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' SSL certificates")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent and cluster info first
         agent_info = await conn.fetchrow("""
             SELECT a.id, a.name, a.pool_id, hc.id as cluster_id, hc.name as cluster_name,
@@ -2046,29 +2150,18 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' SSL certificates")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
         
         if not agent_info['enabled']:
             await close_database_connection(conn)
@@ -2180,13 +2273,319 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             logger.info(f"SSL INCREMENTAL: No certificates updated since {since}")
             
         return response_data
-        
+
     except HTTPException:
         raise
     except Exception as e:
         logger.error(f"SSL certificates retrieval failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
+
+@router.get("/{agent_name}/keepalived-config")
+async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
+    teardown / no-op signal.
+
+    Snapshot-based (T-1): keys off vip_instances.is_active + the member's
+    applied_config_content, NOT the live last_config_status — so a PENDING edit never
+    flips a running member to not_configured (no mid-edit teardown). Auth is MANDATORY (a
+    valid agent key is required), but — like the /config and /ssl-certificates endpoints —
+    the agent API key is a SHARED/global install token, so the config is resolved by the
+    requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
+    (a hard 403 would break every VIP member whose name isn't the one row the shared token
+    resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
+    the agent stays inert; a node with no membership row always gets not_configured.
+    """
+    conn = None
+    try:
+        # Auth FIRST and MANDATORY: a valid agent key is REQUIRED (the response carries the
+        # VRRP secret). The token is shared/global, so resolve by agent_name and only LOG a
+        # name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
+        # unauthenticated caller can't probe which agent names exist.
+        from auth_middleware import validate_agent_api_key
+        if not x_api_key:
+            raise HTTPException(status_code=401, detail="Agent API key required")
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        # The agent API key is a SHARED/global install token (many agent rows per token),
+        # so validate_agent_api_key resolves it to one arbitrary agent for that token. Resolve
+        # the keepalived config strictly by the requested agent_name and treat a token/name
+        # mismatch as an advisory audit log — exactly like the /config and /ssl-certificates
+        # endpoints. (A hard 403 here would reject every VIP member whose name isn't the one
+        # row the shared token happens to return, so the VIP could never converge — review HIGH-1.)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        # Resolve the agent + its cluster's keepalived.conf path (cluster-driven, like the
+        # HAProxy paths). config_path is returned in EVERY response so the agent knows where
+        # to write/own-marker-check even on not_configured/teardown.
+        agent = await conn.fetchrow("""
+            SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
+                   hc.keepalived_config_path,
+                   -- v1.11.1: does the server already hold a discovery for this node? The agent
+                   -- caches the hash of its last discovery report next to the config and skips
+                   -- re-posting while it matches. That cache used to be written even when the
+                   -- POST was REJECTED, so a node could be hidden from the adoption panel for
+                   -- good: the file never changes, so the agent never speaks again. Telling it
+                   -- what we actually hold lets it recover on its own, with no extra request and
+                   -- no one having to touch the node.
+                   EXISTS (SELECT 1 FROM vip_discoveries vd WHERE vd.agent_id = a.id)
+                       AS discovery_known
+            FROM agents a
+            LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
+            WHERE a.name = $1
+            -- A pool may hold more than one cluster, and the join then multiplies this row. With
+            -- no ordering the fetch took an arbitrary one, so the keepalived.conf PATH handed to
+            -- the agent was non-deterministic whenever two clusters in a pool disagreed on it:
+            -- the agent would look at the wrong file, find nothing there, and the node would
+            -- never appear for adoption — intermittently, which is the worst way to fail.
+            --
+            -- A CUSTOMISED path wins over the shipped default, then the lowest cluster id. The
+            -- column defaults to '/etc/keepalived/keepalived.conf' rather than NULL, so ordering
+            -- by id alone could have picked a default-valued row over one the operator had
+            -- deliberately set — turning "undefined" into "reliably wrong" for that install.
+            -- When every cluster in the pool carries the default the string is identical, so the
+            -- ordering cannot change what any working deployment already receives.
+            ORDER BY (hc.keepalived_config_path IS NULL
+                      OR hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'),
+                     hc.id
+            LIMIT 1
+        """, agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+        config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
+        if not agent['enabled']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
+
+        row = await conn.fetchrow("""
+            SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
+                   v.purge_on_teardown,
+                   m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
+            FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
+            WHERE m.agent_id = $1
+            -- Active VIP first (an agent has at most one). With NO active VIP, pick the most
+            -- RECENTLY updated inactive membership so a teardown reflects the latest delete
+            -- (incl. its purge flag) — not a stale older VIP the node was once part of.
+            ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
+            LIMIT 1
+        """, agent['id'])
+
+        if not row:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
+        if not row['is_active']:
+            # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
+            # the agent still only purges on nodes where IT installed keepalived (install marker).
+            return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
+                    "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None,
+                    "purge": bool(row['purge_on_teardown'])}
+        if not row['applied_config_content']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
+
+        from services.keepalived_config import build_haproxy_check_script
+        check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
+        return {
+            "agent_name": agent_name,
+            "status": "available",
+            "config_path": config_path, "discovery_known": bool(agent["discovery_known"]),
+            "keepalived": {
+                "desired_state": "enabled",
+                "install_if_missing": True,
+                "vip_id": row['vip_id'],
+                "vip_name": row['vip_name'],
+                "config_content": row['applied_config_content'],
+                "config_hash": row['applied_config_hash'],
+                "check_script": check_script,
+                # v1.10.4 adoption handoff. The agent refuses to overwrite a keepalived.conf
+                # without our ownership marker — the guard that protects a hand-maintained
+                # setup. Adoption does not weaken it: it authorises exactly ONE takeover, of
+                # exactly the file we analysed, by pinning its hash. If the file changed since
+                # adoption the hashes differ and the agent keeps refusing, so an edit made
+                # between adoption and Apply can never be silently overwritten.
+                "allow_takeover": bool(row['takeover_expected_hash']),
+                "takeover_expected_hash": row['takeover_expected_hash'],
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
+        # Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
+        return {"agent_name": agent_name, "status": "not_configured",
+                "config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-status")
+async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — agent reports the outcome of a keepalived deploy/teardown.
+
+    Auth mirrors config-applied's post-Bulgu-#75 guard (reject a MISSING key — never the
+    `and` short-circuit that accepted no-key requests). The token is a shared/global install
+    token, so the status is recorded strictly for the requested agent_name and a token/name
+    mismatch is an advisory audit log (like /config-applied), not a 403 — review HIGH-1.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived status using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        vip_id = status_data.get("vip_id")
+        state = (status_data.get("state") or "").strip()[:24]
+        config_hash = (status_data.get("config_hash") or "")[:64]
+        message = status_data.get("message")
+        # v1.10.9 — retire the adoption takeover authorisation once the node CONFIRMS it is
+        # running our rendered config. `takeover_expected_hash` is the permission to overwrite a
+        # keepalived.conf that lacks our ownership marker; it was written at adoption and never
+        # cleared, so it stayed valid indefinitely and "one-shot" was only true in the sense of
+        # "for exactly that file content". Clearing it the moment the member acks OUR hash makes
+        # the claim real: if the file is replaced by hand afterwards the agent refuses and reports
+        # "externally managed", which is the visible behaviour an operator should get.
+        #
+        # Gated on the acked hash MATCHING applied_config_hash, so a partial or failed deploy
+        # never drops the authorisation and leaves the VIP unable to converge.
+        # The hash is passed TWICE on purpose. Reusing one placeholder for both the assignment
+        # (`last_deploy_hash=$n`, a VARCHAR column) and the comparison inside the CASE made
+        # PostgreSQL deduce two different types for it and asyncpg refused the whole statement
+        # with AmbiguousParameterError ("text versus character varying"). Because the failure is
+        # in the UPDATE itself, not in one column, EVERY status ack was lost and every VIP sat at
+        # SYNCING forever — including teardown acks. A separate placeholder is only ever compared
+        # against the column, so its type is unambiguous.
+        _retire_takeover = ("takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL "
+                            "AND applied_config_hash = {p} THEN NULL ELSE takeover_expected_hash END")
+        if vip_id is None:
+            # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
+            await conn.execute(f"""
+                UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
+                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$5")}
+                WHERE agent_id=$1
+            """, agent['id'], state, message, config_hash, config_hash)
+        else:
+            await conn.execute(f"""
+                UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
+                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$6")}
+                WHERE agent_id=$1 AND vip_id=$2
+            """, agent['id'], int(vip_id), state, message, config_hash, config_hash)
+        return {"status": "ok"}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-status update failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-status update failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-discovery")
+async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key: Optional[str] = Header(None)):
+    """v1.10.4 — the agent reports a keepalived.conf it found on the node but does NOT own.
+
+    This is what makes adopting a hand-maintained VIP possible: the heartbeat only carries the
+    VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+    fields, so the file itself has to be read. Read-only on the agent side — reporting never
+    changes anything on the node.
+
+    Auth mirrors /keepalived-status: a MISSING key is rejected outright, and because the token
+    is a shared install token a name mismatch is an advisory audit log rather than a 403.
+
+    SECRETS: the reported content may contain the VRRP `auth_pass`. It is split immediately —
+    the password is Fernet-encrypted into its own column and the stored copy of the file has it
+    masked, so nothing readable through the API or a DB dump carries it in cleartext. The
+    parse result is never logged.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived discovery using API key "
+                        f"from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        config_path = (payload.get("config_path") or "/etc/keepalived/keepalived.conf")[:500]
+        exists = bool(payload.get("exists"))
+        if not exists:
+            # The file is gone (keepalived removed, or we adopted and now own it) — drop the row
+            # so the UI stops offering a stale candidate.
+            await conn.execute("DELETE FROM vip_discoveries WHERE agent_id = $1", agent['id'])
+            return {"status": "cleared"}
+
+        content = payload.get("config_content") or ""
+        if len(content) > 256_000:
+            raise HTTPException(status_code=413, detail="keepalived.conf too large to analyse")
+        is_managed = bool(payload.get("is_managed"))
+        config_hash = hashlib.md5(content.encode("utf-8", "replace")).hexdigest()
+
+        from services.keepalived_parser import analyse_keepalived_conf, KeepalivedParseError
+        from services.keepalived_config import encrypt_vrrp_secret
+
+        parse_error = None
+        analysis = None
+        auth_enc = None
+        try:
+            analysis = analyse_keepalived_conf(content)
+            # Split the secret out of everything we persist or serve.
+            for cand in analysis.get("candidates", []):
+                secret = (cand.get("vip") or {}).pop("auth_pass", None)
+                cand["vip"]["has_auth_pass"] = bool(secret)
+                if secret and auth_enc is None:
+                    auth_enc = encrypt_vrrp_secret(secret)
+        except KeepalivedParseError as exc:
+            parse_error = str(exc)[:500]
+        except Exception as exc:  # noqa: BLE001 — a malformed file must not 500 the agent loop
+            parse_error = f"could not analyse the config ({type(exc).__name__})"
+
+        masked = _AUTH_PASS_MASK_RE.sub(r"\1********", content)
+        await conn.execute("""
+            INSERT INTO vip_discoveries
+                (agent_id, config_path, config_hash, is_managed, raw_config_masked,
+                 auth_pass_encrypted, analysis, parse_error, reported_at)
+            VALUES ($1,$2,$3,$4,$5,$6,$7,$8,CURRENT_TIMESTAMP)
+            ON CONFLICT (agent_id) DO UPDATE SET
+                config_path = EXCLUDED.config_path,
+                config_hash = EXCLUDED.config_hash,
+                is_managed = EXCLUDED.is_managed,
+                raw_config_masked = EXCLUDED.raw_config_masked,
+                auth_pass_encrypted = EXCLUDED.auth_pass_encrypted,
+                analysis = EXCLUDED.analysis,
+                parse_error = EXCLUDED.parse_error,
+                reported_at = CURRENT_TIMESTAMP
+        """, agent['id'], config_path, config_hash, is_managed, masked, auth_enc,
+            json.dumps(analysis) if analysis is not None else None, parse_error)
+        return {"status": "recorded", "config_hash": config_hash}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-discovery failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-discovery failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
 @router.get("/script-version")
 async def get_latest_script_version(platform: str = "macos"):
     """Get the latest available agent script version for specified platform"""
@@ -2234,16 +2633,24 @@ async def get_latest_script_version(platform: str = "macos"):
 @router.get("/{agent_name}/upgrade-status")
 async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get agent upgrade status - used by agents to check if they should upgrade"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked before any
+    # DB work; deployed agents always send X-API-Key. Raised before the try so the
+    # 401 is not swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' upgrade status")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Check if agent has upgrade pending (include platform and pool for validation)
         agent = await conn.fetchrow("""
             SELECT status, version as current_version, platform, pool_id
-            FROM agents 
+            FROM agents
             WHERE name = $1
         """, agent_name)
-        
+
         if not agent:
             await close_database_connection(conn)
             return {
@@ -2251,26 +2658,15 @@ async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = H
                 "target_version": "",
                 "message": "Agent not found"
             }
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' upgrade status")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
+
         await close_database_connection(conn)
         
         # Agent should upgrade if status is 'upgrading'
@@ -2704,7 +3100,17 @@ async def get_agent_script_template(platform: str, authorization: str = Header(N
     """Get the latest script template for specified platform from database"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): the raw install/upgrade script is a
+        # version-management surface. Gate reads with agents.version too, matching
+        # the write path above (operator/security_admin/super_admin retain access).
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         conn = await get_database_connection()
         
         # Get latest script template for platform
@@ -2755,7 +3161,19 @@ async def save_agent_script_template(platform: str, template_data: dict, authori
     """Save updated script template to database using shared helper function"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): agent script templates become the
+        # install/self-upgrade script executed as root on HAProxy nodes. A poisoned
+        # template is RCE. Authentication alone is NOT enough — require the same
+        # agents.version permission as POST /versions; otherwise any JWT holder
+        # (including viewer) could overwrite the active script.
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         script_content = template_data.get('script_content', '')
         version = template_data.get('version', '')
         

backend/routers/auth.py

@@ -9,14 +9,48 @@ from datetime import datetime, timedelta
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
 from models.user import LoginRequest, User, UserCreate, UserUpdate, UserPasswordUpdate
+from models.mfa import MfaVerifyRequest
 from utils.activity_log import log_user_activity
 from auth_middleware import get_current_user_from_token
+from services import mfa_service
 
 # Rate limiting temporarily disabled
 
 router = APIRouter(prefix="/api/auth", tags=["Authentication"])
 logger = logging.getLogger(__name__)
 
+MFA_PENDING_TTL_SECONDS = 300  # 5 minutes — pre-verification challenge lifetime
+MFA_PENDING_MAX_ATTEMPTS = 5  # invalidate token after this many wrong codes
+
+
+async def _fetch_mfa_state(conn, user_id: int):
+    """Return (mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step) or None
+    when the MFA columns aren't yet present (pre-migration deploys).
+    """
+    try:
+        return await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+    except Exception as exc:
+        logger.warning(f"MFA columns not available (assuming disabled): {exc}")
+        return None
+
+
+async def _cleanup_expired_pending_logins(conn, user_id: int) -> None:
+    """Lazy cleanup of expired pending MFA challenges for this user."""
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_logins WHERE user_id = $1 AND expires_at < NOW()",
+            user_id,
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-login cleanup skipped: {exc}")
+
 # Security scheme
 security = HTTPBearer()
 
@@ -96,7 +130,7 @@ async def login(login_request: LoginRequest, request: Request):
                 SELECT id, username, email, password_hash, is_active, role, 
                        created_at, updated_at, last_login_at
                 FROM users 
-                WHERE username = $1
+                WHERE username = $1 AND is_active = TRUE
             """, login_request.username)
         except Exception as schema_error:
             logger.warning(f"Schema error, trying fallback query: {schema_error}")
@@ -106,7 +140,7 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login_at
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
             except Exception as column_error:
                 logger.warning(f"last_login_at column error, trying last_login: {column_error}")
@@ -115,26 +149,67 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
         
         if not user:
+            # Covers both "no such user" and "soft-deleted (is_active=FALSE)".
+            # We deliberately return the same generic 401 in either case to
+            # avoid leaking whether an account exists (account enumeration
+            # prevention). Soft-deleted rows are filtered out by the
+            # `AND is_active = TRUE` predicate above.
             await close_database_connection(conn)
             logger.warning(f"Failed login attempt for username: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
-        if not user['is_active']:
-            await close_database_connection(conn)
-            logger.warning(f"Login attempt for inactive account: {login_request.username}")
-            raise HTTPException(status_code=401, detail="Account is deactivated")
-        
+
         # Verify password
         import bcrypt
         if not bcrypt.checkpw(login_request.password.encode('utf-8'), user['password_hash'].encode('utf-8')):
             await close_database_connection(conn)
             logger.warning(f"Wrong password for user: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
+
+        # Issue #18 — MFA branch (v1.6.0): if the user opted in, defer JWT mint and
+        # last_login update until /api/auth/login/mfa-verify completes.
+        mfa_state = await _fetch_mfa_state(conn, user['id'])
+        if mfa_state and mfa_state.get('mfa_enabled'):
+            await _cleanup_expired_pending_logins(conn, user['id'])
+            challenge_token = mfa_service.generate_challenge_token()
+            try:
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_logins (user_id, challenge_token, expires_at, ip_address)
+                    VALUES ($1, $2, NOW() + ($3 || ' seconds')::interval, $4)
+                    """,
+                    user['id'],
+                    challenge_token,
+                    str(MFA_PENDING_TTL_SECONDS),
+                    str(request.client.host) if request.client else None,
+                )
+            except Exception as exc:
+                await close_database_connection(conn)
+                logger.error(f"Failed to create MFA pending login: {exc}")
+                raise HTTPException(status_code=500, detail="MFA challenge creation failed")
+
+            await close_database_connection(conn)
+
+            await log_user_activity(
+                user_id=user['id'],
+                action='mfa.login.challenge_issued',
+                resource_type='mfa',
+                resource_id=str(user['id']),
+                details={'login_method': 'username_password'},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get('user-agent'),
+            )
+
+            return {
+                "mfa_required": True,
+                "mfa_token": challenge_token,
+                "methods": ["totp", "backup"],
+                "expires_in": MFA_PENDING_TTL_SECONDS,
+            }
+
         # Update last login (try different column names)
         try:
             await conn.execute("""
@@ -242,6 +317,348 @@ async def login(login_request: LoginRequest, request: Request):
         logger.error(f"Login error: {e}")
         raise HTTPException(status_code=500, detail="Login failed")
 
+@router.post(
+    "/login/mfa-verify",
+    summary="MFA Verification (Step 2 of Login)",
+    response_description="JWT access token after successful TOTP/backup verification",
+)
+async def login_mfa_verify(payload: MfaVerifyRequest, request: Request):
+    """
+    # MFA Verification — Step 2 of the two-step login flow
+
+    Submit a 6-digit TOTP code OR an 8-character backup code (with optional dash)
+    together with the ``mfa_token`` returned by ``POST /api/auth/login`` for an
+    MFA-enabled account. On success, returns the same response shape as a
+    non-MFA login (Branch A).
+
+    ## Request Body
+    - **mfa_token**: 64-char challenge token from /login response
+    - **code**: 6 digits (TOTP) or `XXXX-YYYY` (backup)
+
+    ## Error Responses
+    - **401**: Invalid code (attempts counter increments)
+    - **410**: Challenge expired or invalidated (too many wrong attempts)
+    """
+    ip_address = str(request.client.host) if request.client else None
+    user_agent = request.headers.get('user-agent')
+
+    # Outcome captured from the transactional block so we can do JWT mint /
+    # activity logging AFTER commit (no side effects on rollback).
+    success_payload: Optional[dict] = None
+    failure: Optional[dict] = None  # { user_id, attempts, invalidated, reason, http_status, detail }
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Round 1 audit fix — wrap the whole verify+update in a single
+        # transaction with row-level locks (FOR UPDATE) so two concurrent
+        # /mfa-verify calls cannot both consume the same TOTP step or the
+        # same pending challenge. We never raise inside the transaction once
+        # we've started mutating the pending row (would rollback the mark);
+        # instead we capture `failure` and raise after commit.
+        async with conn.transaction():
+            pending = await conn.fetchrow(
+                """
+                SELECT id, user_id, attempts, expires_at, used_at
+                  FROM mfa_pending_logins
+                 WHERE challenge_token = $1
+                 FOR UPDATE
+                """,
+                payload.mfa_token,
+            )
+
+            if not pending:
+                failure = {
+                    'user_id': None,
+                    'attempts': 0,
+                    'invalidated': True,
+                    'reason': 'challenge_not_found',
+                    'http_status': 410,
+                    'detail': 'MFA challenge not found or expired',
+                }
+            elif pending['used_at'] is not None:
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_already_used',
+                    'http_status': 410,
+                    'detail': 'MFA challenge already used',
+                }
+            elif pending['expires_at'] and pending['expires_at'] < datetime.utcnow():
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_expired',
+                    'http_status': 410,
+                    'detail': 'MFA challenge expired',
+                }
+            elif pending['attempts'] >= MFA_PENDING_MAX_ATTEMPTS:
+                await conn.execute(
+                    "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                    pending['id'],
+                )
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'too_many_attempts_pre_check',
+                    'http_status': 410,
+                    'detail': 'MFA challenge invalidated (too many attempts)',
+                }
+
+            if failure is None:
+                # Lock the user row so the atomic TOTP-step bump cannot race a
+                # parallel verify on a different pending challenge for the
+                # same account.
+                user_row = await conn.fetchrow(
+                    """
+                    SELECT id, username, email, role, is_active,
+                           created_at, updated_at, last_login_at,
+                           mfa_secret_encrypted, mfa_last_used_totp_step
+                      FROM users
+                     WHERE id = $1
+                     FOR UPDATE
+                    """,
+                    pending['user_id'],
+                )
+                if not user_row or not user_row['is_active']:
+                    failure = {
+                        'user_id': pending['user_id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': False,
+                        'reason': 'user_unavailable',
+                        'http_status': 401,
+                        'detail': 'User not available',
+                    }
+                elif not user_row['mfa_secret_encrypted']:
+                    failure = {
+                        'user_id': user_row['id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': True,
+                        'reason': 'mfa_not_configured',
+                        'http_status': 410,
+                        'detail': 'MFA not configured for this user',
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(user_row['mfa_secret_encrypted'])
+                    verified_method: Optional[str] = None
+                    codes_remaining: Optional[int] = None
+
+                    if secret_plain:
+                        ok, step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, user_row['mfa_last_used_totp_step']
+                        )
+                        if ok:
+                            # Atomic step bump — refuse if another request already
+                            # consumed this (or a newer) TOTP step.
+                            bumped = await conn.fetchval(
+                                """
+                                UPDATE users
+                                   SET mfa_last_used_totp_step = $1,
+                                       mfa_last_used_at = NOW()
+                                 WHERE id = $2
+                                   AND (mfa_last_used_totp_step IS NULL
+                                        OR mfa_last_used_totp_step < $1)
+                                 RETURNING id
+                                """,
+                                step,
+                                user_row['id'],
+                            )
+                            if bumped:
+                                verified_method = 'totp'
+
+                    if verified_method is None:
+                        # Backup codes — atomic single-use consumption.
+                        rows = await conn.fetch(
+                            """
+                            SELECT id, code_hash FROM mfa_backup_codes
+                             WHERE user_id = $1 AND used_at IS NULL
+                            """,
+                            user_row['id'],
+                        )
+                        for row in rows:
+                            if await mfa_service.check_backup_code(payload.code, row['code_hash']):
+                                consumed_id = await conn.fetchval(
+                                    """
+                                    UPDATE mfa_backup_codes
+                                       SET used_at = NOW()
+                                     WHERE id = $1 AND used_at IS NULL
+                                     RETURNING id
+                                    """,
+                                    row['id'],
+                                )
+                                if consumed_id:
+                                    verified_method = 'backup'
+                                    codes_remaining = await conn.fetchval(
+                                        "SELECT COUNT(*) FROM mfa_backup_codes WHERE user_id = $1 AND used_at IS NULL",
+                                        user_row['id'],
+                                    )
+                                break
+
+                    if verified_method is None:
+                        new_attempts = pending['attempts'] + 1
+                        invalidated = new_attempts >= MFA_PENDING_MAX_ATTEMPTS
+                        await conn.execute(
+                            """
+                            UPDATE mfa_pending_logins
+                               SET attempts = $1,
+                                   used_at = CASE WHEN $2 THEN NOW() ELSE used_at END
+                             WHERE id = $3
+                            """,
+                            new_attempts,
+                            invalidated,
+                            pending['id'],
+                        )
+                        failure = {
+                            'user_id': user_row['id'],
+                            'attempts': new_attempts,
+                            'invalidated': invalidated,
+                            'reason': 'invalid_code',
+                            'http_status': 410 if invalidated else 401,
+                            'detail': 'MFA challenge invalidated (too many attempts)'
+                            if invalidated else 'Invalid MFA code',
+                        }
+                    else:
+                        # Verified — finalize state inside the transaction so a
+                        # concurrent verify sees used_at on retry.
+                        await conn.execute(
+                            "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                            pending['id'],
+                        )
+                        if verified_method != 'totp':
+                            await conn.execute(
+                                "UPDATE users SET mfa_last_used_at = NOW() WHERE id = $1",
+                                user_row['id'],
+                            )
+                        try:
+                            await conn.execute(
+                                "UPDATE users SET last_login_at = CURRENT_TIMESTAMP WHERE id = $1",
+                                user_row['id'],
+                            )
+                        except Exception as exc:
+                            logger.warning(f"last_login_at update failed (continuing): {exc}")
+
+                        user_roles = await conn.fetch(
+                            """
+                            SELECT r.id, r.name, r.display_name, r.permissions
+                              FROM user_roles ur
+                              JOIN roles r ON ur.role_id = r.id
+                             WHERE ur.user_id = $1 AND ur.is_active = TRUE AND r.is_active = TRUE
+                            """,
+                            user_row['id'],
+                        )
+
+                        permissions: dict = {}
+                        roles_list: list = []
+                        for role_row in user_roles:
+                            roles_list.append({
+                                'id': role_row['id'],
+                                'name': role_row['name'],
+                                'display_name': role_row['display_name'],
+                            })
+                            role_permissions = role_row['permissions']
+                            if isinstance(role_permissions, str):
+                                import json
+                                role_permissions = json.loads(role_permissions)
+                            if role_permissions:
+                                for perm in role_permissions:
+                                    if '.' in perm:
+                                        resource, action = perm.split('.', 1)
+                                        if resource not in permissions:
+                                            permissions[resource] = {}
+                                        permissions[resource][action] = True
+
+                        success_payload = {
+                            'user': dict(user_row),
+                            'roles_list': roles_list,
+                            'permissions': permissions,
+                            'method': verified_method,
+                            'codes_remaining': codes_remaining,
+                        }
+
+        # ------------------------------------------------------------------
+        # Transaction has committed. Side-effects (JWT mint, audit log) below.
+        # ------------------------------------------------------------------
+        await close_database_connection(conn)
+        conn = None
+
+        if failure is not None:
+            if failure['user_id'] is not None:
+                await log_user_activity(
+                    user_id=failure['user_id'],
+                    action='mfa.login.failed',
+                    resource_type='mfa',
+                    resource_id=str(failure['user_id']),
+                    details={
+                        'reason': failure['reason'],
+                        'attempts': failure['attempts'],
+                        'invalidated': failure['invalidated'],
+                    },
+                    ip_address=ip_address,
+                    user_agent=user_agent,
+                )
+            raise HTTPException(status_code=failure['http_status'], detail=failure['detail'])
+
+        # Success path
+        assert success_payload is not None  # for type checkers; transaction guarantees this
+        user_row = success_payload['user']
+
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        token_payload = {
+            "user_id": user_row['id'],
+            "username": user_row['username'],
+            "email": user_row['email'],
+            "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+            "exp": datetime.utcnow() + timedelta(hours=24),
+        }
+        token = jwt.encode(token_payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+        await log_user_activity(
+            user_id=user_row['id'],
+            action='mfa.login.success',
+            resource_type='mfa',
+            resource_id=str(user_row['id']),
+            details={
+                'method': success_payload['method'],
+                'codes_remaining': success_payload['codes_remaining']
+                if success_payload['method'] == 'backup' else None,
+            },
+            ip_address=ip_address,
+            user_agent=user_agent,
+        )
+
+        return {
+            "access_token": token,
+            "token_type": "bearer",
+            "expires_in": 86400,
+            "user": {
+                "id": user_row['id'],
+                "username": user_row['username'],
+                "email": user_row['email'],
+                "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+                "is_active": user_row['is_active'],
+                "created_at": user_row['created_at'].isoformat() if user_row.get('created_at') else None,
+                "last_login_at": datetime.utcnow().isoformat(),
+            },
+            "roles": success_payload['roles_list'],
+            "permissions": success_payload['permissions'],
+        }
+
+    except HTTPException:
+        raise
+    except Exception as exc:
+        logger.error(f"MFA verify error: {exc}")
+        raise HTTPException(status_code=500, detail="MFA verification failed")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
 @router.post("/logout", summary="User Logout", response_description="Logout confirmation")
 async def logout(request: Request, authorization: str = Header(None)):
     """

backend/routers/backend.py

@@ -333,31 +333,38 @@ async def get_backends(
                 """)
         
         result = []
+        # Issue #24: servers must honor include_inactive exactly like the backend
+        # queries above. Pre-fix these sub-queries hardcoded `is_active = TRUE`
+        # (added in f34a6ee to hide soft-deleted entities), so a server toggled
+        # OFF (is_active=false) vanished from the UI with no way to reactivate it.
+        # Default callers (include_inactive=false) keep the is_active filter →
+        # byte-identical behavior; include_inactive=true now also returns inactive
+        # (disabled / soft-deleted) servers. last_config_status is selected so the
+        # frontend can tell DISABLED (re-enableable) from DELETION (pending delete).
+        server_active_filter = "" if include_inactive else "AND is_active = TRUE"
         for backend in backends:
-            # Get servers for this backend with cluster_id (ONLY show active servers)
-            # CRITICAL FIX: Add is_active = TRUE filter to prevent soft-deleted servers from appearing
             if cluster_id:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND cluster_id = $2 AND is_active = TRUE ORDER BY server_name
+                    FROM backend_servers
+                    WHERE backend_name = $1 AND cluster_id = $2 {server_active_filter} ORDER BY server_name
                 """, backend["name"], cluster_id)
             else:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND is_active = TRUE ORDER BY server_name
-                """, backend["name"]) 
+                    FROM backend_servers
+                    WHERE backend_name = $1 {server_active_filter} ORDER BY server_name
+                """, backend["name"])
             
             # Prepare server list with real-time HAProxy status from agents
             server_list = []
@@ -413,6 +420,7 @@ async def get_backends(
                     "fall": s.get("fall"),
                     "rise": s.get("rise"),
                     "is_active": s["is_active"],
+                    "last_config_status": s.get("last_config_status") or "APPLIED",  # Issue #24: lets UI distinguish DISABLED (re-enableable) from DELETION
                     "status": server_status,
                     "status_age_minutes": status_age_minutes,
                     "last_status_update": s.get("haproxy_status_updated_at").isoformat().replace('+00:00', 'Z') if s.get("haproxy_status_updated_at") else None,
@@ -1909,12 +1917,12 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
         conn = await get_database_connection()
         
-        # Get server info
+        # Get server info. Issue #24: fetch the FULL row (not just 5 columns) so we
+        # can snapshot the pre-toggle state for reject-rollback (see config version below).
         server = await conn.fetchrow("""
-            SELECT id, server_name, backend_name, is_active, cluster_id
-            FROM backend_servers WHERE id = $1
+            SELECT * FROM backend_servers WHERE id = $1
         """, server_id)
-        
+
         if not server:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail="Server not found")
@@ -1975,22 +1983,45 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
                 
                 # Generate new HAProxy config (after database commit)
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
-                
+
                 # Create new config version
                 config_hash = hashlib.sha256(config_content.encode()).hexdigest()
                 version_name = f"server-{server_id}-toggle-{int(time.time())}"
-                
+
                 # Get system admin user ID for created_by
                 conn2 = await get_database_connection()
                 admin_user_id = await conn2.fetchval("SELECT id FROM users WHERE username = 'admin' LIMIT 1") or 1
-                
+
+                # Issue #24: persist an entity snapshot so a Reject of this toggle
+                # rolls back is_active to its pre-toggle value. Pre-fix the toggle's
+                # config version carried NO metadata, so reject only reset
+                # last_config_status and the server stayed disabled (out of sync with
+                # the still-active live config). Mirrors the server-edit snapshot path;
+                # reject's rollback_entity_from_snapshot('server') restores is_active.
+                import json
+                from utils.entity_snapshot import save_entity_snapshot
+                entity_snapshot_metadata = await save_entity_snapshot(
+                    conn=conn2,
+                    entity_type="server",
+                    entity_id=server_id,
+                    old_values=dict(server),            # full pre-toggle row
+                    new_values={"is_active": new_status},
+                    operation="UPDATE",
+                )
+                old_config = await conn2.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                metadata = {"pre_apply_snapshot": old_config or "", **entity_snapshot_metadata}
+
                 # Create PENDING config version
                 config_version_id = await conn2.fetchval("""
-                    INSERT INTO config_versions 
-                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
-                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                    INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING', $6)
                     RETURNING id
-                """, cluster_id, version_name, config_content, config_hash, admin_user_id)
+                """, cluster_id, version_name, config_content, config_hash, admin_user_id, json.dumps(metadata))
                 
                 logger.error(f"SERVER TOGGLE DEBUG: Created PENDING config version {version_name} for cluster {cluster_id}")
                 

backend/routers/cluster.py

@@ -111,6 +111,14 @@ router = APIRouter(prefix="/api/clusters", tags=["clusters"])
 logger = logging.getLogger(__name__)
 
 
+class _AcmeNoConfigChange(Exception):
+    """Internal signal: an ACME edit renders the same config, so mint nothing.
+
+    Control flow, not an error — it unwinds out of the version-minting block without
+    tripping the generic `except Exception` handler that would log it as a failure.
+    """
+
+
 class _ConcurrentlyDrained(Exception):
     """Sentinel raised inside ``apply_pending_changes`` when the
     advisory-lock-protected re-fetch shows that another caller
@@ -292,12 +300,16 @@ async def create_cluster(cluster: HAProxyClusterCreate, authorization: str = Hea
         
         # Create cluster
         cluster_id = await conn.fetchval("""
-            INSERT INTO haproxy_clusters (name, description, connection_type, is_active, 
-                                        stats_socket_path, haproxy_config_path, haproxy_bin_path, pool_id)
-            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7)
+            INSERT INTO haproxy_clusters (name, description, connection_type, is_active,
+                                        stats_socket_path, haproxy_config_path, haproxy_bin_path,
+                                        keepalived_config_path, pool_id,
+                                        acme_enabled, acme_backend_url)
+            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8, COALESCE($9, FALSE), $10)
             RETURNING id
         """, cluster.name, cluster.description, cluster.connection_type,
-            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path, cluster.pool_id)
+            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path,
+            cluster.keepalived_config_path, cluster.pool_id,
+            cluster.acme_enabled, cluster.acme_backend_url)
         
         await close_database_connection(conn)
         
@@ -387,7 +399,8 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
         # Check if cluster exists and get current values
         existing_cluster = await conn.fetchrow("""
             SELECT name, description, connection_type, is_active, stats_socket_path,
-                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled
+                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled,
+                   acme_backend_url
             FROM haproxy_clusters WHERE id = $1
         """, cluster_id)
         if not existing_cluster:
@@ -436,7 +449,12 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"haproxy_bin_path = ${param_counter}")
             update_values.append(cluster.haproxy_bin_path)
             param_counter += 1
-            
+
+        if cluster.keepalived_config_path is not None:
+            update_fields.append(f"keepalived_config_path = ${param_counter}")
+            update_values.append(cluster.keepalived_config_path)
+            param_counter += 1
+
         if cluster.pool_id is not None:
             update_fields.append(f"pool_id = ${param_counter}")
             update_values.append(cluster.pool_id)
@@ -446,7 +464,13 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"acme_enabled = ${param_counter}")
             update_values.append(cluster.acme_enabled)
             param_counter += 1
-        if cluster.acme_backend_url is not None:
+        # Keyed on "was the field submitted?", not "is it non-None". With a plain
+        # `is not None` test there is no way to CLEAR the value: the validator maps an
+        # empty box to None, which is indistinguishable from "not supplied", so once an
+        # operator set a per-cluster URL they could never revert to the global setting —
+        # the field would accept the edit and silently keep the old value.
+        _acme_url_submitted = 'acme_backend_url' in cluster.model_fields_set
+        if _acme_url_submitted:
             update_fields.append(f"acme_backend_url = ${param_counter}")
             update_values.append(cluster.acme_backend_url)
             param_counter += 1
@@ -456,14 +480,75 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_query = f"UPDATE haproxy_clusters SET {', '.join(update_fields)} WHERE id = $1"
             await conn.execute(update_query, *update_values)
 
-        # If acme_enabled actually changed, create a PENDING config version with entity snapshot
-        if cluster.acme_enabled is not None and cluster.acme_enabled != existing_cluster.get('acme_enabled', False):
+        # Create a PENDING config version when an ACME edit would change what the
+        # HAProxy nodes actually run.
+        #
+        # This used to trigger only on `acme_enabled` flipping. `acme_backend_url` is
+        # written to the DB a few lines above but minted nothing, so correcting a wrong
+        # challenge backend from the panel was a silent no-op: the value changed, no
+        # pending version existed, Apply answered "No pending changes to apply", and the
+        # nodes kept the old address indefinitely. That made the one field an operator
+        # needs to fix HTTP-01 impossible to actually apply.
+        _acme_toggled = (
+            cluster.acme_enabled is not None
+            and cluster.acme_enabled != existing_cluster.get('acme_enabled', False)
+        )
+        if _acme_toggled or _acme_url_submitted:
             try:
-                from services.haproxy_config import generate_haproxy_config_for_cluster
+                from services.haproxy_config import (
+                    generate_haproxy_config_for_cluster,
+                    is_config_generation_error,
+                    extract_acme_backend_target,
+                )
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
+                if is_config_generation_error(config_content):
+                    # Same sentinel-instead-of-exception contract as the apply path. A
+                    # PENDING version holding the sentinel is a landmine: the operator
+                    # sees a pending change and applies it, replacing the whole config.
+                    logger.error(
+                        f"ACME TOGGLE: config generation for cluster {cluster_id} returned an "
+                        f"error sentinel; no PENDING version created: {config_content!r}"
+                    )
+                    raise HTTPException(
+                        status_code=422,
+                        detail=(
+                            "ACME setting was saved, but a configuration could not be generated "
+                            "for this cluster, so no pending change was created. "
+                            f"Generator reported: {config_content.strip()[:300]}"
+                        ),
+                    )
+                # A URL edit that renders the same `server _acme_mgmt` line changes
+                # nothing on the nodes, so minting a version would put a no-op pending
+                # change in front of the operator. Compare that one line rather than the
+                # whole config: the generator also renders unrelated PENDING entities,
+                # so a full-text diff reports a change on every edit anyone has queued.
+                _active_config = await conn.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                _new_target = extract_acme_backend_target(config_content)
+                _old_target = extract_acme_backend_target(_active_config)
+                if not _acme_toggled and _new_target == _old_target:
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} URL updated but the rendered "
+                        f"challenge backend is unchanged ({_new_target!r}); no config "
+                        f"version created."
+                    )
+                    raise _AcmeNoConfigChange()
+
                 import time as _time
                 import json as _json
-                version_name = f"cluster-{cluster_id}-acme-{'enable' if cluster.acme_enabled else 'disable'}-{int(_time.time())}"
+                if _acme_toggled:
+                    _acme_kind = 'enable' if cluster.acme_enabled else 'disable'
+                else:
+                    _acme_kind = 'backend'
+                version_name = f"cluster-{cluster_id}-acme-{_acme_kind}-{int(_time.time())}"
+                logger.info(
+                    f"ACME-BACKEND: cluster {cluster_id} pending config version "
+                    f"'{version_name}' created — challenge backend {_old_target!r} -> "
+                    f"{_new_target!r}. Apply the cluster for the nodes to pick it up."
+                )
 
                 from utils.entity_snapshot import save_entity_snapshot
                 snapshot_metadata = await save_entity_snapshot(
@@ -475,7 +560,14 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                         "acme_backend_url": existing_cluster.get('acme_backend_url'),
                     },
                     new_values={
-                        "acme_enabled": cluster.acme_enabled,
+                        # `acme_enabled` is None when only the URL was submitted; the
+                        # snapshot must record the value that is actually in force, or a
+                        # rollback would write NULL over a working flag.
+                        "acme_enabled": (
+                            cluster.acme_enabled
+                            if cluster.acme_enabled is not None
+                            else existing_cluster.get('acme_enabled', False)
+                        ),
                         "acme_backend_url": getattr(cluster, 'acme_backend_url', None) or existing_cluster.get('acme_backend_url'),
                     },
                     operation="UPDATE"
@@ -490,9 +582,20 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                     """, cluster_id, version_name, config_content, current_user.get('id', 1), metadata_json)
                 finally:
                     await close_database_connection(conn2)
+            except _AcmeNoConfigChange:
+                # Not an error: the edit was accepted and simply renders the same
+                # address, so there is nothing for the operator to apply.
+                pass
+            except HTTPException:
+                # The sentinel guard above deliberately fails the request. Without this
+                # clause the generic handler below would swallow it and report success
+                # while no PENDING version exists — the exact silent-success failure
+                # mode this change exists to remove.
+                await close_database_connection(conn)
+                raise
             except Exception as acme_err:
                 logger.error(f"Failed to create ACME config version for cluster {cluster_id}: {acme_err}")
-        
+
         await close_database_connection(conn)
         
         # Log activity
@@ -517,7 +620,7 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
 
 
 @router.get("/{cluster_id}", summary="Get Cluster by ID", response_description="Cluster details")
-async def get_cluster(cluster_id: int, authorization: str = Header(None)):
+async def get_cluster(cluster_id: int, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get Specific HAProxy Cluster
     
@@ -528,8 +631,12 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters/1" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters/1" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -554,20 +661,30 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     - **404**: Cluster not found
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 final convergence): authenticate
-        # the caller before fetching cluster topology by ID. Pre-fix
-        # this sibling of GET /api/clusters was anonymous, so an
-        # attacker could iterate cluster IDs to enumerate the same
-        # info (stats socket, paths, ACME flags, pool identity) the
-        # list endpoint just locked down. Closes the asymmetry.
+    # R18c audit fix (round 6 final convergence): authenticate the caller
+    # before fetching cluster topology by ID. Pre-fix this sibling of
+    # GET /api/clusters was anonymous, so an attacker could iterate cluster
+    # IDs to enumerate the same info (stats socket, paths, ACME flags, pool
+    # identity) the list endpoint just locked down.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         cluster = await conn.fetchrow("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name
             FROM haproxy_clusters c
@@ -591,6 +708,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
             "stats_socket_path": cluster["stats_socket_path"],
             "haproxy_config_path": cluster["haproxy_config_path"],
             "haproxy_bin_path": cluster["haproxy_bin_path"],
+            "keepalived_config_path": cluster.get("keepalived_config_path"),
             "pool_id": cluster["pool_id"],
             "pool_name": cluster["pool_name"],
             "acme_enabled": cluster.get("acme_enabled", False),
@@ -602,7 +720,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
         raise HTTPException(status_code=500, detail=str(e))
 
 @router.get("", summary="Get All Clusters", response_description="List of all clusters")
-async def get_clusters(authorization: str = Header(None)):
+async def get_clusters(authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All HAProxy Clusters
     
@@ -610,8 +728,12 @@ async def get_clusters(authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -662,23 +784,34 @@ async def get_clusters(authorization: str = Header(None)):
     ## Error Responses
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 #3 — KRITIK info leak): require an
-        # authenticated caller. Pre-fix the endpoint accepted
-        # anonymous GETs and returned cluster topology including
-        # internal HAProxy paths (stats socket, config path, bin
-        # path), pool ids, ACME flags, and agent counts. This is
-        # both reconnaissance for an attacker and the spine of the
-        # cluster-scoped RBAC the rest of the platform builds on,
-        # so guarding it at the read layer is essential after R18c
-        # round 5's roster + role guards.
+    # R18c audit fix (round 6 #3 — KRITIK info leak): require an authenticated
+    # caller. Pre-fix the endpoint accepted anonymous GETs and returned cluster
+    # topology including internal HAProxy paths (stats socket, config path, bin
+    # path), pool ids, ACME flags, and agent counts. This is both reconnaissance
+    # for an attacker and the spine of the cluster-scoped RBAC the rest of the
+    # platform builds on, so guarding it at the read layer is essential.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    # (Guard kept OUTSIDE the try below: get_clusters' broad `except Exception`
+    # re-wraps raised HTTPExceptions into 500, which produced the "500 - 401"
+    # in the issue log; raising here yields a clean 401.)
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         clusters = await conn.fetch("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name,
                    COALESCE(agent_counts.total_agents, 0) as total_agents,
@@ -737,6 +870,7 @@ async def get_clusters(authorization: str = Header(None)):
                 "stats_socket_path": cluster["stats_socket_path"],
                 "haproxy_config_path": cluster["haproxy_config_path"],
                 "haproxy_bin_path": cluster["haproxy_bin_path"],
+                "keepalived_config_path": cluster.get("keepalived_config_path"),
                 "pool_id": cluster["pool_id"],
                 "pool_name": cluster["pool_name"],
                 "acme_enabled": cluster.get("acme_enabled", False),
@@ -1313,6 +1447,8 @@ async def list_cluster_config_versions(cluster_id: int, authorization: str = Hea
                 version_type = "WAF Rule"
             elif "ssl-" in version['version_name']:
                 version_type = "SSL Certificate"
+            elif "vip-" in version['version_name']:
+                version_type = "HA / VIP"
             
             # Parse validation error if present
             validation_error = version.get("validation_error")
@@ -1417,10 +1553,14 @@ async def apply_pending_changes(
         # Users will handle configuration completeness through the centralized Apply Management page
         
         # Get all pending config versions for this cluster
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP apply/reject endpoints
+        # (keepalived is not part of haproxy.cfg). Exclude them so a generic cluster apply
+        # from any entity page never marks a VIP version APPLIED without enacting it. This
+        # is a no-op for every non-VIP cluster (no vip-* rows exist).
         pending_versions = await conn.fetch("""
             SELECT id, version_name, created_at, config_content, checksum, metadata
-            FROM config_versions 
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
             ORDER BY created_at ASC
         """, cluster_id)
         
@@ -1870,6 +2010,28 @@ defaults
                 logger.info(f"🧩 APPLY: Generating fresh configuration from database for cluster {cluster_id}")
                 fresh_config_content = await generate_haproxy_config_for_cluster(cluster_id, conn)
             
+            # `generate_haproxy_config_for_cluster` reports failure by RETURNING a
+            # one-line comment instead of raising (haproxy_config.py outer `except`).
+            # Without this guard that sentinel is hashed, stored as an APPLIED version
+            # and shipped to every agent — silently replacing the cluster's entire
+            # configuration with a comment. Any generator exception (an out-of-range
+            # port in acme_backend_url is enough) triggers it. Refuse the apply instead;
+            # the previous APPLIED version stays in force.
+            from services.haproxy_config import is_config_generation_error
+            if is_config_generation_error(fresh_config_content):
+                logger.error(
+                    f"APPLY ABORTED: config generation for cluster {cluster_id} returned an "
+                    f"error sentinel instead of a configuration: {fresh_config_content!r}"
+                )
+                raise HTTPException(
+                    status_code=422,
+                    detail=(
+                        "Configuration could not be generated for this cluster, so nothing "
+                        "was applied and the running configuration is unchanged. "
+                        f"Generator reported: {fresh_config_content.strip()[:300]}"
+                    ),
+                )
+
             # Create a new consolidated config version with fresh content
             import hashlib
             import time
@@ -2539,6 +2701,96 @@ async def get_config_version_diff(cluster_id: int, version_id: int, authorizatio
                     }
                 }
         
+        # HA/VIP (Issue #27): vip-{id}-{action} versions show the generated keepalived.conf
+        # each member node will deploy as the change content (VRRP secret masked). Mirrors
+        # the ssl-* special case above so VIP uses the STANDARD View Change diff modal.
+        # `adopt` MUST stay in this alternation. A vip-* action missing here does not degrade
+        # gracefully: the version falls through to the generic HAProxy diff, which compares this
+        # row's keepalived.conf against the cluster's previous haproxy.cfg and shows the whole
+        # HAProxy config as removed. v1.10.4 added `adopt` without it (fixed in v1.10.8).
+        vip_match = re.search(r'vip-(\d+)-(create|update|delete|adopt)', current_version['version_name'])
+        if vip_match:
+            vip_id = int(vip_match.group(1))
+            vip_action = vip_match.group(2)
+            rendered, vip_meta = None, None
+            old_content = ""
+            try:
+                from routers.vip import render_vip_config_masked
+                rendered, vip_meta = await render_vip_config_masked(conn, vip_id)
+                # For a create/update diff, fetch the PREVIOUS applied vip-* config for this VIP
+                # (the last-deployed keepalived.conf) so an EDIT shows ONLY the changed lines
+                # instead of the whole config as "added". Both sides are already secret-masked.
+                if vip_action != "delete":
+                    prev_applied = await conn.fetchrow(
+                        "SELECT config_content FROM config_versions WHERE version_name LIKE $1 "
+                        "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1",
+                        f"vip-{vip_id}-%", cluster_id, current_version['id'])
+                    if prev_applied and prev_applied['config_content']:
+                        old_content = prev_applied['config_content']
+            except Exception as vip_err:
+                logger.warning(f"VIP DIFF: render failed for vip {vip_id}: {vip_err}")
+            await close_database_connection(conn)
+
+            changes = []
+            line_number = 1
+            if vip_action == "delete" or rendered is None:
+                title = (vip_meta or {}).get("name") or f"VIP {vip_id}"
+                for line in [f"# HA/VIP change: {title}",
+                             "# keepalived will be stopped and the virtual IP released on each member node."
+                             if vip_action == "delete" else
+                             "# (configuration is not available to render yet)"]:
+                    changes.append({"type": "context", "line": line, "line_number": line_number})
+                    line_number += 1
+                summary = {"added": 0, "removed": 1 if vip_action == "delete" else 0,
+                           "total_changes": 1 if vip_action == "delete" else 0}
+            else:
+                # Real line diff. Match the STANDARD haproxy diff format: the line is stored
+                # WITHOUT a +/- prefix (the UI adds it from `type` — the old `+ {line}` here
+                # caused the doubled "+ +"). A CREATE (no previous applied config) shows
+                # everything as added; an UPDATE shows ONLY the lines that actually changed.
+                import difflib
+                new_content = current_version['config_content'] or rendered or ""
+                added_count = 0
+                removed_count = 0
+                line_number = 0
+                if not old_content:
+                    for i, l in enumerate(new_content.split('\n')):
+                        changes.append({"type": "added", "line": l, "line_number": i + 1})
+                        added_count += 1
+                else:
+                    for dl in difflib.unified_diff(old_content.split('\n'), new_content.split('\n'),
+                                                   lineterm='', n=3):
+                        if dl.startswith('@@'):
+                            mm = re.search(r'@@ -(\d+),?\d* \+(\d+),?\d* @@', dl)
+                            if mm:
+                                line_number = int(mm.group(2))
+                            continue
+                        if dl.startswith('---') or dl.startswith('+++'):
+                            continue
+                        if dl.startswith('+'):
+                            changes.append({"type": "added", "line": dl[1:], "line_number": line_number})
+                            added_count += 1
+                            line_number += 1
+                        elif dl.startswith('-'):
+                            changes.append({"type": "removed", "line": dl[1:], "line_number": line_number})
+                            removed_count += 1
+                        elif dl.startswith(' '):
+                            changes.append({"type": "context", "line": dl[1:], "line_number": line_number})
+                            line_number += 1
+                summary = {"added": added_count, "removed": removed_count,
+                           "total_changes": added_count + removed_count}
+
+            return {
+                "current_version": {
+                    "id": current_version['id'],
+                    "version_name": current_version['version_name'],
+                    "created_at": current_version['created_at'].isoformat().replace('+00:00', 'Z')
+                },
+                "previous_version": None,
+                "changes": changes,
+                "summary": summary,
+            }
+
         # Check if current version has config content
         if not current_version['config_content']:
             # Special handling for restore versions - they might not have content yet
@@ -3566,17 +3818,19 @@ async def confirm_restore_config_version(
                     # UPDATE existing frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        UPDATE frontends 
-                        SET bind_address = $1, bind_port = $2, default_backend = $3, 
+                        UPDATE frontends
+                        SET bind_address = $1, bind_port = $2, default_backend = $3,
                             mode = $4, ssl_enabled = $5, ssl_port = $6,
                             maxconn = $7, timeout_client = $8,
+                            log_format = $11, filters = $12,
                             updated_at = CURRENT_TIMESTAMP, last_config_status = 'PENDING'
                         WHERE id = $9 AND cluster_id = $10
-                    """, 
+                    """,
                         parsed_fe.bind_address, parsed_fe.bind_port, parsed_fe.default_backend,
                         parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        fe_id, cluster_id
+                        fe_id, cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_updated"] += 1
                     logger.info(f"RESTORE: Updated frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")
@@ -3584,16 +3838,17 @@ async def confirm_restore_config_version(
                     # CREATE new frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        INSERT INTO frontends 
+                        INSERT INTO frontends
                         (name, bind_address, bind_port, default_backend, mode, ssl_enabled, ssl_port,
                          maxconn, timeout_client,
-                         cluster_id, is_active, last_config_status, created_at, updated_at)
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
-                    """, 
+                         cluster_id, log_format, filters, is_active, last_config_status, created_at, updated_at)
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
+                    """,
                         parsed_fe.name, parsed_fe.bind_address, parsed_fe.bind_port,
                         parsed_fe.default_backend, parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        cluster_id
+                        cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_created"] += 1
                     logger.info(f"RESTORE: Created frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")
@@ -4226,6 +4481,20 @@ async def undo_reject_config_version(
                 ),
             )
 
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP entity, so undo is handled
+        # by the VIP router — it re-stages the rejected change as PENDING from the version's
+        # captured pending_state (reactivating the VIP if a rejected create soft-deleted it,
+        # or re-applying a rejected edit). Returns an error string if it can't (e.g. the
+        # name/address/VRID was reused since reject) -> surface a clean 409.
+        if version_name.startswith('vip-'):
+            from routers.vip import restore_vip_from_rejected_version
+            err = await restore_vip_from_rejected_version(conn, version_id)
+            await close_database_connection(conn)
+            if err:
+                raise HTTPException(status_code=409, detail=f"Cannot undo this VIP change — {err}.")
+            return {"message": "VIP change restored to PENDING — review and Apply it from Apply Management",
+                    "version_name": version_name}
+
         async with conn.transaction():
             # Mark the version as PENDING again
             await conn.execute("""
@@ -4902,9 +5171,17 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         await validate_user_cluster_access(current_user['id'], cluster_id, conn)
 
         # Get all pending config versions for this cluster (CRITICAL: Include metadata for rollback!)
+        # HA/VIP (Issue #27): exclude vip-* versions — they are rejected/reverted by the
+        # VIP reject endpoint (which restores keepalived state), not the generic rollback.
+        # ORDER BY created_at ASC: the rollback loop dedups per entity and keeps the FIRST-processed
+        # snapshot, so the OLDEST snapshot must win — its old_values hold the true pre-change state.
+        # Critical when one entity has multiple pending versions (e.g. cluster ACME enable->disable->enable):
+        # rolling back to the oldest restores the original acme_enabled. (Matches the apply SELECT, which
+        # already orders created_at ASC.)
         pending_versions = await conn.fetch("""
             SELECT id, version_name, metadata FROM config_versions
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
+            ORDER BY created_at ASC
         """, cluster_id)
         
         # CRITICAL FIX: Detect and clean orphan config versions
@@ -5116,11 +5393,12 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         )
         
         # Mark all pending config versions as REJECTED (don't delete them)
+        # HA/VIP (Issue #27): leave vip-* versions to the VIP reject endpoint.
         rejected_count = len(pending_versions)
         await conn.execute("""
-            UPDATE config_versions 
+            UPDATE config_versions
             SET status = 'REJECTED'
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
         """, cluster_id)
 
         # Update WAF rules status to APPLIED (rolled back)

backend/routers/config.py

@@ -3,7 +3,7 @@ Configuration Management and Validation API
 Provides endpoints for HAProxy configuration validation, templates, and optimization
 """
 
-from fastapi import APIRouter, HTTPException, Header, Request
+from fastapi import APIRouter, HTTPException, Header, Request, Depends
 from pydantic import BaseModel
 from typing import Dict, List, Any, Optional
 import logging
@@ -18,7 +18,7 @@ from utils.config_templates import (
 )
 from utils.haproxy_config_parser import parse_haproxy_config
 from utils.logging_config import log_with_correlation, PerformanceLogger
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from database.connection import get_database_connection, close_database_connection
 
 router = APIRouter(prefix="/api/config", tags=["Configuration Management"])
@@ -62,7 +62,7 @@ class ConfigOptimizationRequest(BaseModel):
     optimization_level: str = "balanced"  # conservative, balanced, aggressive
     target_environment: str = "production"  # development, staging, production
 
-@router.post("/validate")
+@router.post("/validate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth; runs HAProxy validator on caller input
 async def validate_configuration(
     request: ConfigValidationRequest,
     current_user: dict = None,
@@ -203,7 +203,7 @@ async def get_template_details(template_id: str):
         )
         raise HTTPException(status_code=500, detail=f"Failed to get template: {str(e)}")
 
-@router.post("/templates/{template_id}/generate")
+@router.post("/templates/{template_id}/generate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def generate_configuration(
     template_id: str,
     request: TemplateGenerationRequest,
@@ -271,7 +271,7 @@ async def generate_configuration(
                 detail=f"Configuration generation failed: {str(e)}"
             )
 
-@router.post("/optimize")
+@router.post("/optimize", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def optimize_configuration(
     request: ConfigOptimizationRequest,
     current_user: dict = None,
@@ -855,6 +855,9 @@ async def parse_bulk_config(
                 "response_headers": frontend.response_headers,
                 "options": frontend.options,
                 "tcp_request_rules": frontend.tcp_request_rules,
+                # Issue #38: SPOE filters + frontend log-format
+                "log_format": frontend.log_format,
+                "filters": frontend.filters,
                 # CRITICAL: SSL Advanced Options (parsed from bind directive)
                 "ssl_alpn": frontend.ssl_alpn,
                 "ssl_npn": frontend.ssl_npn,
@@ -1089,7 +1092,69 @@ async def parse_bulk_config(
         
         # Add auto-assignment info at the beginning
         enhanced_warnings = ssl_auto_assign_info + enhanced_warnings
-        
+
+        # ─────────────────────────────────────────────────────────────────
+        # Issue #38: SPOE pre-flight advisories. Surface, at preview time, the
+        # SPOE configurations that would FAIL HAProxy's `haproxy -c` at apply so
+        # the operator sees them BEFORE importing. Cluster-aware: the referenced
+        # SPOE engine config (e.g. coraza.cfg) is a sibling of the cluster's
+        # haproxy_config_path, which HAProxy OpenManager does not provision.
+        # ─────────────────────────────────────────────────────────────────
+        try:
+            _cfg_path = await conn.fetchval(
+                "SELECT haproxy_config_path FROM haproxy_clusters WHERE id = $1",
+                request.cluster_id,
+            ) or "/etc/haproxy/haproxy.cfg"
+            _cfg_dir = _cfg_path.rsplit("/", 1)[0] or "/etc/haproxy"
+            for _fe in frontends_data:
+                _rh = _fe.get("request_headers") or ""
+                _filters = _fe.get("filters") or ""
+                # engines declared by `filter spoe engine <name> config <path>`
+                _declared_engines = set(re.findall(
+                    r"filter\s+spoe\s+engine\s+(\S+)", _filters, re.IGNORECASE))
+                # engines referenced by `... send-spoe-group <name> <group>`
+                _used_engines = set(re.findall(
+                    r"send-spoe-group\s+(\S+)", _rh, re.IGNORECASE))
+                _missing = _used_engines - _declared_engines
+                if _missing:
+                    enhanced_warnings.append(
+                        f"⚠️ Frontend '{_fe['name']}': 'send-spoe-group' references SPOE "
+                        f"engine(s) {', '.join(sorted(_missing))} but no matching "
+                        f"'filter spoe engine <name> ...' line was found. HAProxy will "
+                        f"reject this at apply with \"unable to find SPOE engine\". Add the "
+                        f"filter line to this frontend."
+                    )
+                for _path in re.findall(
+                        r"filter\s+spoe\s+engine\s+\S+\s+config\s+(\S+)",
+                        _filters, re.IGNORECASE):
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': SPOE engine config '{_path}' and its "
+                        f"SPOA backend must exist on the HAProxy host (cluster config dir: "
+                        f"{_cfg_dir}). HAProxy OpenManager preserves the filter directive but "
+                        f"does not provision these files; otherwise 'haproxy -c' fails at apply."
+                    )
+                # Issue #38 follow-up: ACL `-f <file>` pattern-file advisory.
+                # Scan only the structured rule fields (acl/use_backend) —
+                # request_headers/tcp_request_rules were always free-form and
+                # warning on them now would add new noise for existing users.
+                _pattern_paths = []
+                for _rule in (_fe.get("acl_rules") or []) + (_fe.get("use_backend_rules") or []):
+                    if isinstance(_rule, str):
+                        _pattern_paths.extend(
+                            re.findall(r"(?:^|\s)-f\s+(\S+)", _rule))
+                if _pattern_paths:
+                    _uniq = sorted(set(_pattern_paths))
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': ACL/routing rules reference pattern "
+                        f"file(s) {', '.join(_uniq)}. Each file must exist at that exact path "
+                        f"on every HAProxy host in the cluster (cluster config dir: {_cfg_dir}) "
+                        f"— HAProxy OpenManager does not create or distribute pattern files. "
+                        f"A missing file fails safely at 'haproxy -c' (previous config keeps "
+                        f"running)."
+                    )
+        except Exception as _spoe_adv_err:
+            logger.warning(f"SPOE advisory generation skipped: {_spoe_adv_err}")
+
         # BULK IMPORT MVP: Check existing entities for UPSERT detection
         # Mark each entity as new or update for UI display
         # CRITICAL: Only mark as UPDATE if there are actual field changes
@@ -1150,7 +1215,17 @@ async def parse_bulk_config(
                 if frontend.get("tcp_request_rules") and frontend["tcp_request_rules"] != existing["tcp_request_rules"]:
                     has_changes = True
                     changes["tcp_request_rules"] = {"old": existing["tcp_request_rules"], "new": frontend["tcp_request_rules"]}
-                
+                # Issue #38: SPOE filters + log-format change detection. REQUIRED for
+                # persistence (not just display): without it, an import that only adds
+                # a `filter`/`log-format` to an existing frontend would be flagged
+                # "no change" and the directive would never be written to the DB.
+                if frontend.get("log_format") and frontend["log_format"] != existing.get("log_format"):
+                    has_changes = True
+                    changes["log_format"] = {"old": existing.get("log_format"), "new": frontend["log_format"]}
+                if frontend.get("filters") and frontend["filters"] != existing.get("filters"):
+                    has_changes = True
+                    changes["filters"] = {"old": existing.get("filters"), "new": frontend["filters"]}
+
                 # CRITICAL: SSL Advanced Options change detection
                 if frontend.get("ssl_alpn") is not None and frontend.get("ssl_alpn") != existing.get("ssl_alpn"):
                     has_changes = True
@@ -2094,7 +2169,18 @@ async def bulk_create_entities(
                         update_fields.append(f"options = ${param_index}")
                         update_values.append(frontend_data["options"])
                         param_index += 1
-                    
+
+                    # Issue #38: SPOE filters + frontend log-format (merge strategy)
+                    if frontend_data.get("log_format") and frontend_data["log_format"] != existing_full.get("log_format"):
+                        update_fields.append(f"log_format = ${param_index}")
+                        update_values.append(frontend_data["log_format"])
+                        param_index += 1
+
+                    if frontend_data.get("filters") and frontend_data["filters"] != existing_full.get("filters"):
+                        update_fields.append(f"filters = ${param_index}")
+                        update_values.append(frontend_data["filters"])
+                        param_index += 1
+
                     # CRITICAL FIX: Update SSL advanced options (alpn, npn, ciphers, etc.)
                     # These are parsed from bind directive and should be preserved in database
                     if "ssl_alpn" in frontend_data and frontend_data.get("ssl_alpn") != existing_full.get("ssl_alpn"):
@@ -2214,9 +2300,10 @@ async def bulk_create_entities(
                             timeout_client, timeout_http_request, maxconn,
                             request_headers, response_headers, tcp_request_rules, options,
                             rate_limit, compression, log_separate, monitor_uri,
-                            cluster_id, acl_rules, use_backend_rules, redirect_rules, updated_at
-                        ) 
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                            cluster_id, acl_rules, use_backend_rules, redirect_rules,
+                            log_format, filters, updated_at
+                        )
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
                         RETURNING id
                     """, 
                         frontend_data["name"],
@@ -2257,7 +2344,9 @@ async def bulk_create_entities(
                         request.cluster_id,
                         json.dumps(frontend_data.get("acl_rules", [])),  # acl_rules
                         json.dumps(frontend_data.get("use_backend_rules", [])),  # use_backend_rules
-                        json.dumps([])   # redirect_rules
+                        json.dumps([]),   # redirect_rules
+                        frontend_data.get("log_format"),  # Issue #38
+                        frontend_data.get("filters")      # Issue #38
                     )
                     
                     created_entities["frontends"].append({

backend/routers/configuration.py

@@ -201,13 +201,15 @@ async def get_pending_config_requests(agent_name: str, x_api_key: Optional[str]
     Called during heartbeat.
     """
     try:
-        # Validate agent API key
+        # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Deployed agents
+        # always send X-API-Key; an absent/invalid key is unauthenticated. This
+        # endpoint also mutates state (marks requests 'processing'), so a keyless
+        # caller could otherwise starve the real agent.
         agent_auth = await validate_agent_api_key(x_api_key)
-        
-        if x_api_key and not agent_auth:
-            logger.warning(f"Invalid API key provided by agent '{agent_name}' for pending requests")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-        
+        if not agent_auth:
+            logger.warning(f"Missing/invalid API key from '{agent_name}' for pending requests")
+            raise HTTPException(status_code=401, detail="Authentication required")
+
         conn = await get_database_connection()
         
         # Get pending requests

backend/routers/csr.py

@@ -0,0 +1,375 @@
+"""
+CSR (Certificate Signing Request) endpoints (v1.9.0).
+
+Generate a private key + CSR in-app, download the CSR PEM, have it signed by
+an external CA, then import the signed certificate — which creates a normal
+ssl_certificates row that flows through the existing pipeline
+(config version → Apply Management → agent pull).
+
+Security posture:
+- All endpoints enforce ssl.* permissions explicitly (including the read
+  endpoints — deliberately stricter than the legacy cert detail route).
+- The private key is NEVER returned by any endpoint here; after import it is
+  reachable only via the existing certificate detail route.
+- Key generation is offloaded to a thread (RSA-4096 takes seconds; the
+  backend runs a single-worker event loop by default) and rate-limited
+  per user via the user_activity_logs COUNT pattern (acme_diagnostics
+  precedent — slowapi is not registered on the app).
+"""
+
+import asyncio
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, HTTPException, Request, Header
+
+from database.connection import get_database_connection, close_database_connection
+from auth_middleware import get_current_user_from_token, check_user_permission
+from models.csr import SSLCSRCreate, SSLCSRImport
+from services import csr_service, ssl_service
+from routers.ssl import _assert_safe_cert_name, validate_user_cluster_access
+from utils.activity_log import log_user_activity
+
+router = APIRouter(prefix="/api/ssl/csrs", tags=["SSL CSRs"])
+logger = logging.getLogger(__name__)
+
+_RATE_LIMIT_CREATE_PER_MIN = 10
+
+# Columns exposed to the API — private_key_pem is deliberately absent so a
+# future `SELECT *` refactor cannot silently start leaking it.
+_CSR_LIST_COLUMNS = """
+    c.id, c.name, c.common_name, c.subject, c.sans, c.key_algorithm,
+    c.status, c.ssl_certificate_id, c.completed_at, c.created_at, c.updated_at,
+    s.name AS certificate_name, u.username AS created_by_username
+"""
+
+_INT32_MAX = 2_147_483_647
+
+
+def _client_ip(request: Optional[Request]) -> Optional[str]:
+    try:
+        return str(request.client.host) if request and request.client else None
+    except Exception:
+        return None
+
+
+def _user_agent(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.headers.get("user-agent") if request else None
+    except Exception:
+        return None
+
+
+async def _require(authorization: Optional[str], action: str):
+    """Authenticate + enforce ssl.<action>; returns current_user or raises 401/403."""
+    current_user = await get_current_user_from_token(authorization)
+    ok = await check_user_permission(current_user["id"], "ssl", action, current_user=current_user)
+    if not ok:
+        raise HTTPException(status_code=403, detail=f"Insufficient permissions: ssl.{action} required")
+    return current_user
+
+
+def _assert_int32_id(csr_id: int) -> None:
+    """ssl_csrs.id is int4 — an out-of-range path param would surface as an
+    asyncpg DataError 500 (Bulgu #96 precedent); return a clean 404 instead."""
+    if csr_id < 1 or csr_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="CSR not found")
+
+
+def _assert_valid_cluster_id(cluster_id: int) -> None:
+    """Same int4 guard for body-supplied cluster ids: haproxy_clusters.id is
+    SERIAL/int4, so an out-of-range value would raise asyncpg DataError inside
+    validate_user_cluster_access and surface as a 500 with the raw driver
+    error. Fail with the same clean 404 the cluster lookup itself produces."""
+    if not isinstance(cluster_id, int) or cluster_id < 1 or cluster_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="Cluster not found")
+
+
+async def _enforce_create_rate_limit(conn, user_id: int) -> None:
+    """Per-user per-minute limit on key generation, counted against the
+    csr_create audit-log action (acme_diagnostics _enforce_rate_limit pattern,
+    backed by the (user_id, action, created_at DESC) composite index)."""
+    cnt = await conn.fetchval(
+        """
+        SELECT COUNT(*)
+        FROM user_activity_logs
+        WHERE user_id = $1
+          AND action = 'csr_create'
+          AND created_at >= NOW() - INTERVAL '60 seconds'
+        """,
+        user_id,
+    )
+    if cnt is not None and cnt >= _RATE_LIMIT_CREATE_PER_MIN:
+        raise HTTPException(
+            status_code=429,
+            detail=(
+                f"Rate limit exceeded: at most {_RATE_LIMIT_CREATE_PER_MIN} "
+                "CSRs may be created per minute"
+            ),
+        )
+
+
+@router.post("")
+async def create_csr(payload: SSLCSRCreate, request: Request, authorization: Optional[str] = Header(None)):
+    """Generate a private key + CSR. Returns the CSR PEM immediately (so the
+    UI can show copy/download in one round trip) — never the private key."""
+    current_user = await _require(authorization, "create")
+    conn = None
+    try:
+        # Belt and braces on top of the model validator — same duplication
+        # convention as the certificate create route.
+        _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+        await _enforce_create_rate_limit(conn, current_user["id"])
+
+        # Fail fast on a taken name BEFORE burning CPU on key generation;
+        # insert_csr_row re-checks and the partial unique index closes the race.
+        await csr_service.assert_csr_name_available(conn, payload.name)
+
+        bundle = await asyncio.to_thread(csr_service.generate_csr_bundle, payload)
+        csr_id = await csr_service.insert_csr_row(conn, payload, bundle, current_user["id"])
+
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_create',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'csr_name': payload.name,
+                'common_name': payload.common_name,
+                'sans': bundle['sans'],
+                'key_algorithm': payload.key_algorithm,
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": f"CSR '{payload.name}' created successfully",
+            "csr": csr_service.csr_row_to_dict(row, include_pem=True),
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error creating CSR: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("")
+async def list_csrs(authorization: Optional[str] = Header(None)):
+    """List CSRs (no PEM payloads — fetch the detail route for the CSR PEM).
+    Cluster-agnostic: a CSR binds to clusters only at import time."""
+    await _require(authorization, "read")
+    conn = None
+    try:
+        conn = await get_database_connection()
+        rows = await conn.fetch(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            ORDER BY c.created_at DESC
+            """
+        )
+        return [csr_service.csr_row_to_dict(r) for r in rows]
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing CSRs: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("/{csr_id}")
+async def get_csr(csr_id: int, authorization: Optional[str] = Header(None)):
+    """CSR detail including the CSR PEM. The private key is never included."""
+    await _require(authorization, "read")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        return csr_service.csr_row_to_dict(row, include_pem=True)
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{csr_id}/import")
+async def import_csr_certificate(
+    csr_id: int,
+    payload: SSLCSRImport,
+    request: Request,
+    authorization: Optional[str] = Header(None),
+):
+    """Import the CA-signed certificate for a pending CSR. Creates an
+    ssl_certificates row (source='csr', PENDING) and stages one config
+    version per affected cluster — the operator applies manually."""
+    current_user = await _require(authorization, "create")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        if payload.name:
+            _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+
+        if not payload.is_global:
+            for cluster_id in payload.cluster_ids or []:
+                _assert_valid_cluster_id(cluster_id)
+                await validate_user_cluster_access(current_user["id"], cluster_id, conn)
+
+        result = await csr_service.import_signed_certificate(
+            conn, csr_id, payload, current_user["id"]
+        )
+        cert_id = result["certificate_id"]
+
+        if payload.is_global:
+            cluster_rows = await conn.fetch(
+                "SELECT id FROM haproxy_clusters WHERE is_active = TRUE"
+            )
+            affected_clusters = [r['id'] for r in cluster_rows]
+        else:
+            affected_clusters = payload.cluster_ids or []
+
+        # Post-commit staging — a config-generation failure never rolls back
+        # the certificate (same semantics as the manual create flow).
+        sync_results = await ssl_service.stage_ssl_config_versions(
+            conn, cert_id, affected_clusters, action='create',
+            created_by=current_user["id"],
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='create',
+            resource_type='ssl_certificate',
+            resource_id=str(cert_id),
+            details={
+                'certificate_name': result['certificate_name'],
+                'domain': result.get('primary_domain', 'unknown'),
+                'via': 'csr',
+                'csr_id': csr_id,
+                'usage_type': payload.usage_type,
+                'is_global': payload.is_global,
+                'cluster_ids': payload.cluster_ids,
+                'warnings': result['warnings'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_import',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'certificate_id': cert_id,
+                'certificate_name': result['certificate_name'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": (
+                f"Certificate '{result['certificate_name']}' imported "
+                "successfully. Go to Apply Management to deploy."
+            ),
+            "certificate_id": cert_id,
+            "warnings": result["warnings"],
+            "sync_results": sync_results,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error importing signed certificate for CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.delete("/{csr_id}")
+async def delete_csr(csr_id: int, request: Request, authorization: Optional[str] = Header(None)):
+    """Hard delete. For a pending CSR this permanently destroys the private
+    key (any certificate later signed from that CSR becomes unusable); for a
+    completed CSR it only removes history — the imported certificate is not
+    affected (the FK points csr → cert)."""
+    current_user = await _require(authorization, "delete")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        async with conn.transaction():
+            # FOR UPDATE serialises against an in-flight import of the same CSR.
+            row = await conn.fetchrow(
+                "SELECT id, name, status FROM ssl_csrs WHERE id = $1 FOR UPDATE",
+                csr_id,
+            )
+            if not row:
+                raise HTTPException(status_code=404, detail="CSR not found")
+            await conn.execute("DELETE FROM ssl_csrs WHERE id = $1", csr_id)
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='delete',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={'csr_name': row['name'], 'status': row['status']},
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        if row['status'] == 'pending':
+            message = (
+                f"CSR '{row['name']}' deleted — its private key has been "
+                "permanently destroyed."
+            )
+        else:
+            message = (
+                f"CSR '{row['name']}' deleted (history only) — the imported "
+                "certificate is not affected."
+            )
+        return {"message": message}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error deleting CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)

backend/routers/dashboard.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Header
+from fastapi import APIRouter, HTTPException, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 from datetime import datetime
 import logging
@@ -11,7 +12,7 @@ from agent_notifications import get_cluster_agents_status
 router = APIRouter(prefix="/api", tags=["dashboard", "pools"])
 logger = logging.getLogger(__name__)
 
-@router.get("/dashboard/overview")
+@router.get("/dashboard/overview", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaked cluster/pool/agent stats, names, health & alerts anonymously (auth was optional)
 async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization: str = Header(None)):
     """Get dashboard overview with comprehensive statistics, optionally filtered by cluster"""
     try:
@@ -238,7 +239,7 @@ async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization
         logger.error(f"Error fetching dashboard overview: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/dashboard/stats")
+@router.get("/dashboard/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): aggregate cluster/agent counts
 async def get_dashboard_stats():
     """Get dashboard statistics"""
     try:
@@ -279,7 +280,7 @@ async def get_dashboard_stats():
     except Exception as e:
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/pools")
+@router.get("/pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): pool names/env/counts
 async def get_pools():
     """Get all HAProxy cluster pools"""
     try:
@@ -350,7 +351,7 @@ async def get_pools():
         logger.error(f"Error fetching pools: {e}")
         return {"pools": []}
 
-@router.get("/haproxy-cluster-pools")
+@router.get("/haproxy-cluster-pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_cluster_pools():
     """Get all HAProxy cluster pools (legacy endpoint)"""
     # Just call the main pools endpoint
@@ -474,7 +475,7 @@ async def update_pool(pool_id: int, pool: PoolUpdate, authorization: str = Heade
         logger.error(f"Failed to update pool: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to update pool: {str(e)}")
 
-@router.get("/haproxy-cluster-pools/{pool_id}/agents")
+@router.get("/haproxy-cluster-pools/{pool_id}/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): full agent inventory — same class as GET /api/agents
 async def get_pool_agents(pool_id: int):
     """Get all agents for a specific pool"""
     try:
@@ -561,7 +562,7 @@ async def get_pool_agents(pool_id: int):
         logger.error(f"Error fetching pool agents: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to fetch pool agents: {str(e)}")
 
-@router.get("/haproxy/stats")
+@router.get("/haproxy/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_stats(cluster_id: Optional[int] = None):
     """Get HAProxy statistics"""
     try:

backend/routers/dashboard_stats.py

@@ -3,13 +3,22 @@ Dashboard Stats Router
 API endpoints for HAProxy statistics dashboard
 """
 
-from fastapi import APIRouter, HTTPException, Query
+from fastapi import APIRouter, HTTPException, Query, Depends
 from typing import Optional, List
 import logging
 
 from services.dashboard_stats_service import dashboard_stats_service
+from auth_middleware import require_authenticated_user
 
-router = APIRouter(prefix="/api/dashboard-stats", tags=["dashboard-stats"])
+# SECURITY (GHSA-3p5c-m5m4-mjpx): this entire router (traffic metrics, backend
+# health, cluster topology, agent status) was mounted without authentication.
+# Require a valid JWT on every route. The frontend Dashboard already sends the
+# operator JWT on these calls, so this is transparent to the UI.
+router = APIRouter(
+    prefix="/api/dashboard-stats",
+    tags=["dashboard-stats"],
+    dependencies=[Depends(require_authenticated_user)],
+)
 logger = logging.getLogger(__name__)
 
 

backend/routers/frontend.py

@@ -117,6 +117,41 @@ def _rule_contradiction_text(rule: Any) -> Optional[str]:
     return None
 
 
+def _pattern_file_warnings(
+    acl_rules: Optional[List[Any]] = None,
+    use_backend_rules: Optional[List[Any]] = None,
+    redirect_rules: Optional[List[Any]] = None,
+) -> List[str]:
+    """Issue #38 follow-up — non-blocking `-f <file>` pattern-file
+    advisory for the manual frontend API.
+
+    The Bulgu #12 hard reject was removed from the Pydantic models:
+    pattern files are operator-managed host files (same policy as the
+    SPOE `filter ... config <path>` reference preserved since v1.8.8)
+    and the agent's pre-reload `haproxy -c` makes a missing file fail
+    safely. This helper returns one warning listing the unique file
+    paths referenced across the rule fields, or [] when no rule uses
+    `-f` — operators who don't use pattern files see no change.
+    """
+    paths: List[str] = []
+    for rules in (acl_rules, use_backend_rules, redirect_rules):
+        for rule in rules or []:
+            text = rule if isinstance(rule, str) else (
+                rule.get("condition") if isinstance(rule, dict) else None)
+            if isinstance(text, str):
+                paths.extend(re.findall(r"(?:^|\s)-f\s+(\S+)", text))
+    if not paths:
+        return []
+    uniq = sorted(set(paths))
+    return [
+        f"ACL/routing rules reference pattern file(s) {', '.join(uniq)}. "
+        f"Each file must exist at that exact path on every HAProxy host "
+        f"in the cluster — HAProxy OpenManager does not create or "
+        f"distribute pattern files. A missing file fails safely at "
+        f"'haproxy -c' (the previous config keeps running)."
+    ]
+
+
 def _collect_routing_rule_contradictions(
     rules: List[Any], origin_label: str,
 ) -> List[Tuple[str, Any]]:
@@ -408,6 +443,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -424,6 +460,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -453,6 +490,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -465,6 +503,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -480,6 +519,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -492,6 +532,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -601,6 +642,8 @@ async def get_frontends(
                     "response_headers": f.get("response_headers"),
                     "options": f.get("options"),
                     "tcp_request_rules": f.get("tcp_request_rules"),
+                    "log_format": f.get("log_format"),  # Issue #38
+                    "filters": f.get("filters"),        # Issue #38
                     "timeout_client": f.get("timeout_client"),
                     "timeout_http_request": f.get("timeout_http_request"),
                     "rate_limit": f.get("rate_limit"),
@@ -735,18 +778,18 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 acl_rules, redirect_rules, use_backend_rules,
                 request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
                 rate_limit, compression, log_separate, monitor_uri,
-                cluster_id, maxconn, updated_at
-            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                cluster_id, maxconn, log_format, filters, updated_at
+            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
             RETURNING id
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, frontend.ssl_enabled,
             frontend.ssl_certificate_id, ssl_cert_ids_json, frontend.ssl_port, frontend.ssl_cert_path, frontend.ssl_cert, frontend.ssl_verify,
-            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites, 
+            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
             frontend.ssl_min_ver, frontend.ssl_max_ver, frontend.ssl_strict_sni,
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters)
         
         # If cluster_id provided, create new config version for agents
         sync_results = []
@@ -825,12 +868,19 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 user_agent=request.headers.get('user-agent')
             )
         
-        return {
+        response: dict = {
             "message": f"Frontend '{frontend.name}' created successfully",
             "id": frontend_id,
             "frontend": frontend.dict(),
             "sync_results": sync_results
         }
+        # Issue #38 follow-up — non-blocking pattern-file advisory
+        # (additive field; absent when no rule references `-f`).
+        pattern_warnings = _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if pattern_warnings:
+            response["warnings"] = pattern_warnings
+        return response
     except HTTPException:
         raise
     except Exception as e:
@@ -1060,9 +1110,10 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                 acl_rules = $20, redirect_rules = $21, use_backend_rules = $22,
                 request_headers = $23, response_headers = $24, options = $25, tcp_request_rules = $26, timeout_client = $27, timeout_http_request = $28,
                 rate_limit = $29, compression = $30, log_separate = $31, monitor_uri = $32,
-                cluster_id = $33, maxconn = $34, updated_at = CURRENT_TIMESTAMP
-            WHERE id = $35
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+                cluster_id = $33, maxconn = $34, log_format = $35, filters = $36,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $37
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, ssl_enabled,
             ssl_certificate_id, ssl_cert_ids_json, ssl_port, ssl_cert_path, ssl_cert, ssl_verify,
             frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
@@ -1070,7 +1121,7 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn, frontend_id)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters, frontend_id)
             
         # Debug: Check what was actually saved
         updated_frontend = await conn.fetchrow("""
@@ -1124,6 +1175,8 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                     "response_headers": frontend.response_headers,
                     "options": filtered_options,
                     "tcp_request_rules": frontend.tcp_request_rules,
+                    "log_format": frontend.log_format,  # Issue #38
+                    "filters": frontend.filters,        # Issue #38
                     "timeout_client": frontend.timeout_client,
                     "timeout_http_request": frontend.timeout_http_request,
                     "rate_limit": frontend.rate_limit,
@@ -1235,8 +1288,12 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
         # yellow toast on the next refresh. The save SUCCEEDED; the
         # warnings only flag latent legacy data the operator may
         # want to clean up at their convenience.
-        if contradiction_warnings:
-            response["warnings"] = contradiction_warnings
+        # Issue #38 follow-up — append the pattern-file advisory to
+        # the same list (additive; empty when no rule uses `-f`).
+        all_warnings = list(contradiction_warnings or []) + _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if all_warnings:
+            response["warnings"] = all_warnings
         return response
     except HTTPException:
         raise

backend/routers/health.py

@@ -3,8 +3,9 @@ Production-Ready Health Check and Monitoring Endpoints
 Provides comprehensive system health monitoring for Kubernetes and production environments
 """
 
-from fastapi import APIRouter, HTTPException
+from fastapi import APIRouter, HTTPException, Depends
 from fastapi.responses import JSONResponse
+from auth_middleware import require_authenticated_user
 import logging
 import asyncio
 import time
@@ -74,7 +75,7 @@ async def readiness_probe():
         logger.error(f"Readiness probe failed: {e}")
         raise HTTPException(status_code=503, detail=f"Not ready: {str(e)}")
 
-@router.get("/deep")
+@router.get("/deep", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks host CPU/mem/disk, DB/redis/python versions, PID
 async def deep_health_check():
     """Comprehensive health check with detailed system information"""
     global _health_cache
@@ -197,7 +198,7 @@ async def deep_health_check():
         
         raise HTTPException(status_code=503, detail=error_response)
 
-@router.get("/agents")
+@router.get("/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks agent names/hostnames
 async def agents_health():
     """Monitor agent connectivity and health status"""
     try:
@@ -261,7 +262,7 @@ async def agents_health():
         logger.error(f"Agent health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Agent health check failed: {str(e)}")
 
-@router.get("/clusters")
+@router.get("/clusters", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks cluster names, HAProxy versions, pending counts
 async def clusters_health():
     """Monitor HAProxy cluster health and configuration status"""
     try:
@@ -329,7 +330,7 @@ async def clusters_health():
         logger.error(f"Cluster health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Cluster health check failed: {str(e)}")
 
-@router.get("/errors")
+@router.get("/errors", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): app error metrics, sibling of /deep,/agents,/clusters
 async def error_statistics():
     """Get application error statistics and metrics"""
     try:

backend/routers/letsencrypt.py

@@ -1,6 +1,7 @@
 from fastapi import APIRouter, HTTPException, Header
-from pydantic import BaseModel, Field, field_validator
-from typing import Optional, List
+from pydantic import BaseModel, Field, field_validator, model_validator
+from typing import Optional, List, Dict
+import base64
 import json
 import logging
 import re
@@ -10,6 +11,40 @@ from datetime import datetime
 from database.connection import get_database_connection, close_database_connection
 from services.acme_service import acme_service
 from services.haproxy_config import generate_haproxy_config_for_cluster
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import encrypt_dns_credentials, decrypt_dns_credentials
+
+# Issue #35: DNS-01 challenge methods.
+_CHALLENGE_TYPES = ("http-01", "dns-01")
+
+
+async def _dns01_enabled() -> bool:
+    """Global kill-switch (system_settings acme.dns01_enabled, default False). Read via the ACME
+    settings dict so non-admins never need the admin-only /api/settings/acme endpoint."""
+    try:
+        settings = await acme_service._get_settings()
+        val = settings.get('dns01_enabled')
+        if isinstance(val, str):
+            return val.strip().lower() in ('1', 'true', 'yes', 'on')
+        return bool(val)
+    except Exception:
+        return False
+
+
+# Per-user sliding-window rate limit for the manual dns-confirm action (soft anti-abuse so a user
+# can't spam the CA via the confirm button). Per-process; sufficient for a manual UI action.
+_DNS_CONFIRM_RL: Dict[int, list] = {}
+_DNS_CONFIRM_LIMIT = 5
+_DNS_CONFIRM_WINDOW = 60.0
+
+
+async def _enforce_dns_confirm_rate_limit(user_id: int) -> None:
+    now = time.time()
+    bucket = [t for t in _DNS_CONFIRM_RL.get(user_id, []) if now - t < _DNS_CONFIRM_WINDOW]
+    if len(bucket) >= _DNS_CONFIRM_LIMIT:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded: dns-confirm allowed 5 requests per minute")
+    bucket.append(now)
+    _DNS_CONFIRM_RL[user_id] = bucket
 
 logger = logging.getLogger(__name__)
 
@@ -25,8 +60,81 @@ class AccountCreate(BaseModel):
     email: str
     directory_url: Optional[str] = None
     tos_agreed: bool = True
-    eab_kid: Optional[str] = None
-    eab_hmac_key: Optional[str] = None
+    # EAB (External Account Binding) for CAs that require it (ZeroSSL, Google). The KID is opaque
+    # (bound only); the HMAC key must be base64url so newAccount's _b64url_decode won't raise a
+    # cryptic binascii error (a common copy mistake is standard-base64 '+'/'/' vs urlsafe '-'/'_').
+    eab_kid: Optional[str] = Field(default=None, max_length=256)
+    eab_hmac_key: Optional[str] = Field(default=None, max_length=512)
+    # Issue #35: per-account default challenge method + DNS provider (for dns-01).
+    challenge_type: str = "http-01"
+    dns_provider: Optional[str] = None
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @field_validator('eab_hmac_key')
+    @classmethod
+    def _validate_eab_hmac_key(cls, v):
+        if not v:
+            return v
+        try:
+            base64.urlsafe_b64decode(v + '=' * (-len(v) % 4))
+        except Exception:
+            raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
+        return v
+
+    @field_validator('directory_url')
+    @classmethod
+    def _validate_directory_url(cls, v):
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): reject non-https URLs and literal
+        # non-public IP hosts at the API boundary. The full DNS-based SSRF check
+        # runs at fetch time (acme_service.get_directory -> ssrf_guard).
+        if not v:
+            return v
+        from urllib.parse import urlparse
+        import ipaddress
+        from utils.ssrf_guard import is_public_ip
+        parsed = urlparse(v.strip())
+        if parsed.scheme.lower() != 'https':
+            raise ValueError("directory_url must be an https URL")
+        host = parsed.hostname
+        if not host:
+            raise ValueError("directory_url has no host")
+        try:
+            ipaddress.ip_address(host)
+            is_ip_literal = True
+        except ValueError:
+            is_ip_literal = False
+        if is_ip_literal and not is_public_ip(host):
+            raise ValueError("directory_url must not point to a private/loopback IP address")
+        return v
+
+    @model_validator(mode='after')
+    def _require_provider_for_dns01(self):
+        if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
+            raise ValueError("dns_provider is required when challenge_type is 'dns-01'")
+        return self
+
+
+class DnsCredentialsUpsert(BaseModel):
+    dns_provider: str = Field(..., min_length=1, max_length=50)
+    credentials: Dict[str, str] = Field(default_factory=dict)
+
+    @field_validator('credentials')
+    @classmethod
+    def _validate_credentials(cls, v):
+        if len(v) > 20:
+            raise ValueError("Too many credential fields")
+        for key, val in v.items():
+            if not isinstance(key, str) or not re.match(r'^[a-zA-Z0-9_]{1,50}$', key):
+                raise ValueError(f"Invalid credential field name: {key!r}")
+            if not isinstance(val, str) or len(val) > 4000:
+                raise ValueError(f"Credential value for {key!r} is missing or too long")
+        return v
 
 
 class CertificateRequest(BaseModel):
@@ -38,6 +146,8 @@ class CertificateRequest(BaseModel):
     account_id: Optional[int] = None
     cluster_ids: List[int] = Field(default_factory=list)
     auto_renew: bool = True
+    # Issue #35: optional override; when None the account's default method is used.
+    challenge_type: Optional[str] = None
 
     @field_validator('domains')
     @classmethod
@@ -54,7 +164,30 @@ class CertificateRequest(BaseModel):
             if not _DOMAIN_REGEX.match(d_norm):
                 raise ValueError(f"Invalid domain format: '{d}'")
             normalized.append(d_norm)
-        return normalized
+        # De-duplicate (case/whitespace variants normalize to the same value) while preserving order,
+        # so we don't submit a redundant SAN to the CA or render duplicate-keyed tags in the UI.
+        return list(dict.fromkeys(normalized))
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v is not None and v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @model_validator(mode='after')
+    def _wildcard_requires_dns01(self):
+        # Static cross-field guard: a wildcard SAN can ONLY be issued via dns-01 (the CA rejects
+        # wildcard over http-01). The runtime dns01_enabled gate + provider resolution happen in the
+        # endpoint (validators can't do async/DB). When challenge_type is None here, the effective
+        # method is resolved from the account in the endpoint, which re-checks this.
+        if any((d or '').startswith('*.') for d in (self.domains or [])):
+            # Only reject when the caller EXPLICITLY chose a non-dns-01 method. When challenge_type is
+            # None, the effective method is resolved from the account in the endpoint, which re-checks
+            # wildcard-requires-dns-01 — so account-default dns-01 inheritance still works for wildcards.
+            if self.challenge_type is not None and self.challenge_type != 'dns-01':
+                raise ValueError("Wildcard certificates require challenge_type 'dns-01'")
+        return self
 
 
 # --- Account management ---
@@ -77,7 +210,7 @@ async def list_accounts(authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         rows = await conn.fetch(
-            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at FROM letsencrypt_accounts ORDER BY id"
+            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at, challenge_type, dns_provider FROM letsencrypt_accounts ORDER BY id"
         )
         return [dict(r) for r in rows]
     finally:
@@ -107,16 +240,36 @@ async def create_account(body: AccountCreate, authorization: str = Header(None))
         eab_kid = body.eab_kid or settings.get('eab_kid', '') or None
         eab_hmac_key = body.eab_hmac_key or settings.get('eab_hmac_key', '') or None
 
+        # Issue #35: a dns-01 account must name a supported DNS provider.
+        if body.challenge_type == 'dns-01':
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((body.dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {body.dns_provider}")
+
         result = await acme_service.register_account(
             email=body.email,
             directory_url=directory_url,
             tos_agreed=body.tos_agreed,
             eab_kid=eab_kid,
             eab_hmac_key=eab_hmac_key,
+            challenge_type=body.challenge_type,
+            dns_provider=(body.dns_provider or None),
         )
         return result
+    except HTTPException:
+        # Preserve deliberate status codes (e.g. 409 DNS-01 disabled, 422 unsupported provider) —
+        # the broad except below would otherwise downgrade them all to 400.
+        raise
     except Exception as e:
         logger.error(f"ACME account registration failed: {e}")
+        # Humanize the common EAB-required failure (ZeroSSL/Google). The ACME error propagates as a
+        # string ("Account registration failed: {<dict>}"), so match the URN substring in str(e).
+        if 'externalaccountrequired' in str(e).lower():
+            raise HTTPException(status_code=400, detail=(
+                "This CA requires External Account Binding (EAB). Enter the EAB Key ID and HMAC Key "
+                "from your ZeroSSL/Google account and retry."
+            ))
         raise HTTPException(status_code=400, detail=str(e))
 
 
@@ -186,6 +339,147 @@ async def remove_account(account_id: int, authorization: str = Header(None)):
         await close_database_connection(conn)
 
 
+# --- Issue #35: DNS-01 providers + per-account DNS credentials ---
+
+@router.get("/dns-providers")
+async def get_dns_providers(authorization: str = Header(None)):
+    """List supported DNS providers + their credential-field schema (for the UI). Also returns the
+    global dns01_enabled gate so a non-admin cert UI can read it without the admin-only settings API.
+    Authenticated (any user); not admin-only."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    return {"dns01_enabled": await _dns01_enabled(), "providers": list_providers()}
+
+
+@router.get("/accounts/{account_id}/dns-credentials")
+async def get_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Masked metadata only — provider + which credential fields are set + updated_at. NEVER returns
+    the ciphertext or any plaintext token. Read-only for any authenticated user (matches list_accounts)."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            "SELECT dns_provider, credentials_encrypted, updated_at FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+            account_id,
+        )
+        if not row:
+            return {"configured": False, "dns_provider": None, "credential_fields_present": [], "updated_at": None}
+        present = []
+        decrypted = decrypt_dns_credentials(row["credentials_encrypted"])
+        if isinstance(decrypted, dict):
+            present = sorted(decrypted.keys())
+        return {
+            "configured": True,
+            "dns_provider": row["dns_provider"],
+            "credential_fields_present": present,
+            "updated_at": row["updated_at"],
+        }
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/accounts/{account_id}/dns-credentials")
+async def upsert_dns_credentials(account_id: int, body: DnsCredentialsUpsert, authorization: str = Header(None)):
+    """Store (encrypted) DNS provider credentials for an account. Admin-only. Verifies the
+    credentials against the provider BEFORE persisting; returns a sanitized result (never the token)."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    provider_name = body.dns_provider.strip()
+    if not is_supported(provider_name):
+        raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {provider_name}")
+
+    conn = await get_database_connection()
+    try:
+        exists = await conn.fetchval("SELECT 1 FROM letsencrypt_accounts WHERE id = $1", account_id)
+        if not exists:
+            raise HTTPException(status_code=404, detail="ACME account not found")
+
+        # Verify credentials synchronously; only persist on success. The detail is user-safe.
+        try:
+            provider = get_provider(provider_name, dict(body.credentials))
+            verify = await provider.verify_credentials()
+        except DnsProviderError as exc:
+            raise HTTPException(status_code=422, detail=str(exc))
+        except HTTPException:
+            raise
+        except Exception:
+            # Defensive: never let a provider-internal exception string (which could echo creds in a
+            # future provider) reach the client. Always a sanitized 422.
+            raise HTTPException(status_code=422, detail="DNS provider credential verification failed")
+        if not verify.get("ok"):
+            raise HTTPException(status_code=422, detail=verify.get("detail") or "DNS provider credential verification failed")
+
+        token = encrypt_dns_credentials(dict(body.credentials))
+        await conn.execute(
+            """INSERT INTO letsencrypt_account_dns_credentials (account_id, dns_provider, credentials_encrypted, updated_at)
+               VALUES ($1, $2, $3, NOW())
+               ON CONFLICT (account_id) DO UPDATE SET
+                   dns_provider = EXCLUDED.dns_provider,
+                   credentials_encrypted = EXCLUDED.credentials_encrypted,
+                   updated_at = NOW()""",
+            account_id, provider_name, token,
+        )
+        # Keep the account's provider selection in sync.
+        await conn.execute(
+            "UPDATE letsencrypt_accounts SET dns_provider = $1, updated_at = NOW() WHERE id = $2",
+            provider_name, account_id,
+        )
+        return {"ok": True, "dns_provider": provider_name, "detail": verify.get("detail", "Credentials stored.")}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/accounts/{account_id}/dns-credentials")
+async def delete_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Remove an account's stored DNS credentials. Admin-only."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    conn = await get_database_connection()
+    try:
+        await conn.execute("DELETE FROM letsencrypt_account_dns_credentials WHERE account_id = $1", account_id)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/orders/{order_id}/dns-confirm")
+async def confirm_dns_order(order_id: int, authorization: str = Header(None)):
+    """Manual DNS-01 only: the user asserts the TXT record is published; tell the CA to validate.
+    Requires ssl.create + a per-user rate limit; acts only on a dns-01 + manual + pending order."""
+    from auth_middleware import get_current_user_from_token, check_user_permission
+    current_user = await get_current_user_from_token(authorization)
+    has_perm = await check_user_permission(current_user['id'], 'ssl', 'create')
+    if not has_perm:
+        raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.create required")
+    await _enforce_dns_confirm_rate_limit(current_user['id'])
+
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+    finally:
+        await close_database_connection(conn)
+    if not order:
+        raise HTTPException(status_code=404, detail="Order not found")
+    if order['challenge_type'] != 'dns-01' or (order['dns_provider'] or 'manual') != 'manual':
+        raise HTTPException(status_code=409, detail="This order is not a manual DNS-01 order")
+    if order['status'] not in ('pending', 'processing'):
+        raise HTTPException(status_code=409, detail=f"Order is '{order['status']}' and cannot be confirmed")
+
+    from services.dns01_orchestrator import confirm_manual_dns01
+    await confirm_manual_dns01(order_id)
+    return {"ok": True, "message": "DNS-01 confirmation submitted; the CA will validate shortly."}
+
+
 # --- Certificate operations ---
 
 @router.post("/certificates")
@@ -222,32 +516,61 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
         logger.info(f"ACME: Using account_id={account_id} for certificate request")
 
         warnings = []
-        # Audit Tur 5 / Commit 8c: empty cluster_ids in UI means "global certificate".
-        # Resolve to all ACME-enabled active clusters; only fail if NONE exist.
+
+        # Issue #35: resolve the effective challenge method (request override, else account default).
+        conn_acct = await get_database_connection()
+        try:
+            acct = await conn_acct.fetchrow(
+                "SELECT challenge_type, dns_provider FROM letsencrypt_accounts WHERE id = $1", account_id
+            )
+        finally:
+            await close_database_connection(conn_acct)
+        effective_challenge = (body.challenge_type or (acct['challenge_type'] if acct else None) or 'http-01')
+        dns_provider = (acct['dns_provider'] if acct else None)
+        is_dns01 = (effective_challenge == 'dns-01')
+        has_wildcard = any(d.startswith('*.') for d in body.domains)
+
+        if is_dns01:
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail="The selected ACME account has no DNS provider configured for DNS-01.")
+            if (dns_provider or 'manual') == 'manual':
+                # Manual DNS-01 cannot be renewed unattended; auto-renew is forced off on the issued
+                # certificate (see _complete_certificate). Tell the requester so it isn't a surprise.
+                warnings.append(
+                    "Manual DNS-01 certificates cannot auto-renew unattended. Auto-renew will be disabled; "
+                    "re-publish the TXT record and request renewal before expiry."
+                )
+        elif has_wildcard:
+            raise HTTPException(status_code=422, detail="Wildcard certificates require a DNS-01 account.")
+
+        # Empty cluster_ids = "global certificate". For DNS-01 no ACME Challenge Routing / port 80 is
+        # needed, so resolve to ALL active clusters; http-01 still requires acme_enabled clusters.
         if not body.cluster_ids:
             conn_resolve = await get_database_connection()
             try:
-                acme_clusters_resolved = await conn_resolve.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
-                )
+                if is_dns01:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE")
             finally:
                 await close_database_connection(conn_resolve)
-            if not acme_clusters_resolved:
+            if not resolved:
+                if is_dns01:
+                    raise HTTPException(status_code=422, detail="Cannot issue certificate: no active clusters configured.")
                 raise HTTPException(
                     status_code=422,
                     detail="Cannot issue certificate: no ACME-enabled clusters configured. "
                            "Enable ACME Challenge Routing on at least one cluster in Cluster Management, "
                            "Apply the configuration change, then retry."
                 )
-            body.cluster_ids = [c['id'] for c in acme_clusters_resolved]
+            body.cluster_ids = [c['id'] for c in resolved]
             warnings.append(
-                f"No clusters specified — applied to all ACME-enabled cluster(s) ({len(body.cluster_ids)})"
+                f"No clusters specified — applied to all {'active' if is_dns01 else 'ACME-enabled'} cluster(s) ({len(body.cluster_ids)})"
             )
-            logger.warning(
-                f"ACME: Empty cluster_ids → global cert fallback to {len(body.cluster_ids)} ACME-enabled cluster(s)"
-            )
-        else:
-            # Validate that referenced clusters exist + are ACME-enabled (warn-only).
+        elif not is_dns01:
+            # http-01 only: warn if no cluster has ACME Challenge Routing enabled.
             try:
                 conn_warn = await get_database_connection()
                 try:
@@ -255,7 +578,6 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
                         "SELECT COUNT(*) FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
                     )
                     if acme_clusters == 0:
-                        logger.warning("ACME: No clusters with ACME Challenge Routing enabled - certificate validation will likely fail")
                         warnings.append(
                             "No clusters have ACME Challenge Routing enabled. "
                             "Certificate validation will fail. Enable it in Cluster Management and Apply Changes first."
@@ -269,14 +591,30 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             account_id=account_id,
             domains=body.domains,
             cluster_ids=body.cluster_ids,
+            challenge_type=effective_challenge,
+            created_by=current_user['id'],
         )
 
-        challenges = await acme_service.respond_to_challenges(order['order_id'])
+        # Audit trail: record who requested the certificate + the method (esp. for DNS-01/wildcard,
+        # which has a wider blast radius than http-01). Never raises into the request path.
+        try:
+            from utils.activity_log import record_event
+            await record_event(
+                order['order_id'], "acme.order.requested",
+                message=f"Certificate requested ({effective_challenge}) by user {current_user['id']}",
+                details={"user_id": current_user['id'], "challenge_type": effective_challenge,
+                         "dns_provider": dns_provider, "domains": body.domains},
+            )
+        except Exception:
+            pass
+
+        # http-01 posts the challenge response immediately (token served continuously). dns-01 is
+        # driven by the orchestrator AFTER the TXT is published (manual waits for dns-confirm), so we
+        # must NOT respond here.
+        challenges = []
+        if not is_dns01:
+            challenges = await acme_service.respond_to_challenges(order['order_id'])
 
-        # Commit 5b: re-fetch the order's *current* status from DB. The status
-        # returned by create_order() reflects the moment of creation; after
-        # respond_to_challenges() the CA may have already advanced it (e.g. to
-        # 'processing'). Surfacing stale status leads UI to under-poll.
         conn_status = await get_database_connection()
         try:
             fresh_status = await conn_status.fetchval(
@@ -287,14 +625,23 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             await close_database_connection(conn_status)
         effective_status = fresh_status or order['status']
 
-        logger.info(f"ACME: Order {order['order_id']} created, {len(challenges)} challenge(s) posted, status={effective_status}")
+        if is_dns01:
+            msg = ("Order created. Publish the DNS TXT record shown for each domain, then confirm."
+                   if dns_provider == 'manual'
+                   else "Order created. The DNS TXT record(s) will be published automatically; waiting for CA validation.")
+        else:
+            msg = "Order created. ACME challenges have been posted. Waiting for CA validation."
+
+        logger.info(f"ACME: Order {order['order_id']} created ({effective_challenge}), status={effective_status}")
 
         return {
             "order_id": order['order_id'],
             "status": effective_status,
             "domains": body.domains,
+            "challenge_type": effective_challenge,
+            "dns_provider": dns_provider,
             "challenges": challenges,
-            "message": "Order created. ACME challenges have been posted. Waiting for CA validation.",
+            "message": msg,
             "warnings": warnings,
         }
     except HTTPException:
@@ -316,7 +663,8 @@ async def list_orders(authorization: str = Header(None)):
         rows = await conn.fetch("""
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.ssl_certificate_id, o.cluster_ids, o.error_detail,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             ORDER BY o.created_at DESC
@@ -345,7 +693,8 @@ async def get_order(order_id: int, authorization: str = Header(None)):
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.certificate_url, o.finalize_url, o.expires_at,
                    o.error_detail, o.ssl_certificate_id, o.cluster_ids,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             WHERE o.id = $1
@@ -353,13 +702,23 @@ async def get_order(order_id: int, authorization: str = Header(None)):
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
 
+        # Issue #35: include challenge_type + dns_txt_value (PUBLIC DNS data — NOT key_authorization,
+        # NOT the API token) so the UI can render manual DNS-01 instructions. Explicit column list.
         challenges = await conn.fetch(
-            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
+            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at, "
+            "challenge_type, dns_txt_value FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
         )
         result = dict(order)
         result['domains'] = json.loads(result['domains']) if isinstance(result['domains'], str) else result['domains']
         result['cluster_ids'] = json.loads(result['cluster_ids']) if isinstance(result['cluster_ids'], str) else result['cluster_ids']
-        result['challenges'] = [dict(c) for c in challenges]
+        ch_list = []
+        for c in challenges:
+            cd = dict(c)
+            if cd.get('challenge_type') == 'dns-01':
+                # Server computes the record name (wildcard *.-stripping lives server-side).
+                cd['dns_record_name'] = acme_service._challenge_dns_name(cd['domain'])
+            ch_list.append(cd)
+        result['challenges'] = ch_list
         return result
     finally:
         await close_database_connection(conn)
@@ -470,18 +829,23 @@ async def renew_order(order_id: int, authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         order = await conn.fetchrow(
-            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1", order_id
+            "SELECT account_id, domains, cluster_ids, challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
         )
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
         domains = json.loads(order['domains']) if isinstance(order['domains'], str) else order['domains']
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
+        challenge_type = order['challenge_type'] or 'http-01'
 
         new_order = await acme_service.create_order(
-            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids
+            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids,
+            challenge_type=challenge_type, created_by=current_user['id'],
         )
-        challenges = await acme_service.respond_to_challenges(new_order['order_id'])
-        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenges": challenges}
+        # dns-01 is driven by the orchestrator after the TXT is published; only http-01 responds here.
+        challenges = []
+        if challenge_type != 'dns-01':
+            challenges = await acme_service.respond_to_challenges(new_order['order_id'])
+        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenge_type": challenge_type, "challenges": challenges}
     finally:
         await close_database_connection(conn)
 
@@ -562,12 +926,22 @@ async def import_le_ca_chain(authorization: str = Header(None)):
         ("https://letsencrypt.org/certs/r11.pem", "R11 Intermediate"),
     ]
     chain_parts = []
+    # v1.11.0: each download is recorded as an outbound row. The bodies are
+    # public CA certificates, not secrets, and the 8 KB body cap truncates them —
+    # what matters here is which URL failed, with what status.
+    from utils.http_instrumentation import outbound_span, TARGET_LETSENCRYPT_CA
+
     async with aiohttp.ClientSession() as session:
         for url, name in ca_urls:
             try:
-                async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
-                    if resp.status == 200:
-                        chain_parts.append(await resp.text())
+                async with outbound_span(
+                    target=TARGET_LETSENCRYPT_CA, method="GET", url=url
+                ) as span:
+                    async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
+                        text = await resp.text() if resp.status == 200 else None
+                        span.set_response(resp.status, getattr(resp, "headers", None), text)
+                        if resp.status == 200:
+                            chain_parts.append(text)
             except Exception as e:
                 logger.warning(f"Failed to download {name}: {e}")
 
@@ -611,11 +985,17 @@ async def get_renewal_schedule(authorization: str = Header(None)):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
     conn = await get_database_connection()
     try:
+        # Issue #35: expose the challenge method/provider (via the originating order/account) so the
+        # UI can distinguish manual DNS-01 certs, which cannot auto-renew unattended. LEFT JOINs keep
+        # legacy certs (no order link / pre-DNS-01 columns) rendering as http-01.
         certs = await conn.fetch("""
-            SELECT id, name, primary_domain, expiry_date, auto_renew, days_until_expiry
-            FROM ssl_certificates
-            WHERE source = 'letsencrypt' AND is_active = TRUE
-            ORDER BY expiry_date ASC NULLS LAST
+            SELECT c.id, c.name, c.primary_domain, c.expiry_date, c.auto_renew, c.days_until_expiry,
+                   o.challenge_type, a.dns_provider
+            FROM ssl_certificates c
+            LEFT JOIN letsencrypt_orders o ON o.id = c.letsencrypt_order_id
+            LEFT JOIN letsencrypt_accounts a ON a.id = o.account_id
+            WHERE c.source = 'letsencrypt' AND c.is_active = TRUE
+            ORDER BY c.expiry_date ASC NULLS LAST
         """)
         return [dict(c) for c in certs]
     finally:
@@ -663,6 +1043,18 @@ async def _complete_certificate(order_id: int) -> dict:
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
         primary_domain = domains[0] if domains else 'unknown'
 
+        # Issue #35: a manual DNS-01 certificate cannot be auto-renewed unattended (the renewal task
+        # skips it — see main.py), so persist auto_renew=FALSE rather than storing a misleading
+        # "Enabled" that the user trusts while the cert silently expires. http-01 and automated
+        # DNS-01 (e.g. Cloudflare) keep auto_renew=TRUE, preserving existing behaviour.
+        auto_renew_value = True
+        if order.get('challenge_type') == 'dns-01':
+            acct_provider = await conn.fetchval(
+                "SELECT dns_provider FROM letsencrypt_accounts WHERE id = $1", order['account_id']
+            )
+            if (acct_provider or 'manual') == 'manual':
+                auto_renew_value = False
+
         # Commit 5g: guard against empty cert_private_key.
         # Inserting an SSL certificate row with an empty private_key would silently
         # produce an unusable certificate (HAProxy would fail to load on Apply, or
@@ -749,12 +1141,13 @@ async def _complete_certificate(order_id: int) -> dict:
                     certificate_content = $1, private_key_content = $2, chain_content = $3,
                     all_domains = $4::jsonb, expiry_date = $5, days_until_expiry = $6,
                     issuer = $7, fingerprint = $8, letsencrypt_order_id = $9,
-                    auto_renew = TRUE, is_active = TRUE, last_config_status = 'PENDING',
+                    auto_renew = $11, is_active = TRUE, last_config_status = 'PENDING',
                     updated_at = NOW()
                 WHERE id = $10
             """, cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id,
+                auto_renew_value)
             logger.info(f"ACME RENEWAL: Updated certificate {cert_id} for {primary_domain}, status=PENDING")
         else:
             cert_row = await conn.fetchrow("""
@@ -764,11 +1157,12 @@ async def _complete_certificate(order_id: int) -> dict:
                      usage_type, source, letsencrypt_order_id, auto_renew, is_active,
                      last_config_status)
                 VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10,
-                        'frontend', 'letsencrypt', $11, TRUE, TRUE, 'PENDING')
+                        'frontend', 'letsencrypt', $11, $12, TRUE, 'PENDING')
                 RETURNING id
             """, f"le-{primary_domain}", cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), primary_domain, all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id,
+                auto_renew_value)
             cert_id = cert_row['id']
 
         await conn.execute(
@@ -815,12 +1209,18 @@ async def _complete_certificate(order_id: int) -> dict:
             if mapped:
                 effective_cluster_ids = [r['cluster_id'] for r in mapped]
             else:
-                # Audit Tur 6 / Commit 5j: only fall back to ACME-enabled clusters.
-                # Applying renewal to ACME-disabled clusters could re-introduce Issue #11
-                # patterns and risks deploying certs to clusters where they can't be renewed.
-                all_clusters = await conn.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                # Audit Tur 6 / Commit 5j: http-01 only falls back to ACME-enabled clusters.
+                # Issue #35: a dns-01 cert needs NO challenge routing / port 80, so it can renew on
+                # any active cluster — resolve to all active clusters for dns-01.
+                ch_type = await conn.fetchval(
+                    "SELECT challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
                 )
+                if ch_type == 'dns-01':
+                    all_clusters = await conn.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    all_clusters = await conn.fetch(
+                        "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                    )
                 effective_cluster_ids = [r['id'] for r in all_clusters]
             if effective_cluster_ids:
                 logger.info(f"ACME RENEWAL: Resolved {len(effective_cluster_ids)} cluster(s) for global cert {cert_id}")

backend/routers/mfa.py

@@ -0,0 +1,727 @@
+"""MFA router — Issue #18, v1.6.0.
+
+All endpoints are additive; nothing here breaks existing JWT or apply_service flows.
+Authentication is JWT-based (Bearer). Admin endpoints additionally require
+``users.is_admin == True`` (canonical super-admin flag).
+"""
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from middleware.mfa_rate_limit_key import mfa_rate_limit_key
+from middleware.mfa_rate_limits import MFA_LIMITS
+from middleware.rate_limiter import limiter
+from models.mfa import (
+    MfaAdminResetAllRequest,
+    MfaAdminResetRequest,
+    MfaDisableRequest,
+    MfaEnrollConfirmRequest,
+    MfaEnrollConfirmResponse,
+    MfaEnrollStartResponse,
+    MfaRegenerateBackupRequest,
+    MfaRegenerateBackupResponse,
+    MfaStatusResponse,
+)
+from services import mfa_service
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/api/mfa", tags=["MFA"])
+
+# Lifecycle constants (Plan section 5)
+PENDING_ENROLLMENT_TTL_SECONDS = 600  # 10 minutes — QR scan + verify window
+PENDING_ENROLLMENT_MAX_ATTEMPTS = 5
+
+
+# ---------------------------------------------------------------------------
+# Authentication helpers
+# ---------------------------------------------------------------------------
+
+
+async def _require_user(authorization: Optional[str]) -> dict:
+    user = await get_current_user_from_token(authorization)
+    if not user:
+        raise HTTPException(status_code=401, detail="Not authenticated")
+    return user
+
+
+async def _require_admin(authorization: Optional[str]) -> dict:
+    user = await _require_user(authorization)
+    if not user.get("is_admin"):
+        raise HTTPException(status_code=403, detail="Admin privileges required")
+    return user
+
+
+async def _cleanup_expired_pending_enrollments(conn) -> None:
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_enrollments WHERE expires_at < NOW()"
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-enrollment cleanup skipped: {exc}")
+
+
+# ---------------------------------------------------------------------------
+# Self status
+# ---------------------------------------------------------------------------
+
+
+@router.get(
+    "/status",
+    summary="MFA status for the authenticated user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_status(authorization: str = Header(None)):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            current["id"],
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            current["id"],
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+@router.get(
+    "/admin/status/{user_id}",
+    summary="Admin: MFA status of any user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_admin_status(user_id: int, authorization: str = Header(None)):
+    await _require_admin(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            user_id,
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Enrollment
+# ---------------------------------------------------------------------------
+
+
+@router.post(
+    "/enroll/start",
+    summary="Begin TOTP enrollment (returns secret + otpauth URI)",
+    response_model=MfaEnrollStartResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_start, key_func=mfa_rate_limit_key)
+async def mfa_enroll_start(request: Request, authorization: str = Header(None)):
+    current = await _require_user(authorization)
+
+    # Round 7 audit fix — REFUSE re-enrollment if the user is already MFA-on.
+    # Without this guard a stolen JWT could silently rotate the victim's TOTP
+    # secret + invalidate all their backup codes via /enroll/start ->
+    # /enroll/confirm (overwriting `users.mfa_secret_encrypted` and replacing
+    # `mfa_backup_codes`). To re-enroll, the user must first call /api/mfa/disable
+    # (which requires a fresh TOTP) or an admin must run /api/mfa/admin-reset.
+    secret_plain = mfa_service.generate_totp_secret()
+    secret_encrypted = mfa_service.encrypt_secret(secret_plain)
+
+    conn = await get_database_connection()
+    blocked = False
+    try:
+        # Single transaction with SELECT FOR UPDATE closes the TOCTOU window
+        # between the mfa_enabled check and the pending_enrollment upsert.
+        async with conn.transaction():
+            row = await conn.fetchrow(
+                "SELECT mfa_enabled FROM users WHERE id = $1 FOR UPDATE",
+                current["id"],
+            )
+            if row and row["mfa_enabled"]:
+                blocked = True
+            else:
+                await _cleanup_expired_pending_enrollments(conn)
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_enrollments
+                        (user_id, secret_encrypted, attempts, expires_at)
+                    VALUES ($1, $2, 0, NOW() + ($3 || ' seconds')::interval)
+                    ON CONFLICT (user_id) DO UPDATE
+                      SET secret_encrypted = EXCLUDED.secret_encrypted,
+                          attempts = 0,
+                          expires_at = EXCLUDED.expires_at,
+                          created_at = CURRENT_TIMESTAMP
+                    """,
+                    current["id"],
+                    secret_encrypted,
+                    str(PENDING_ENROLLMENT_TTL_SECONDS),
+                )
+    finally:
+        await close_database_connection(conn)
+
+    if blocked:
+        raise HTTPException(
+            status_code=400,
+            detail="MFA is already enabled. Disable it first (via /api/mfa/disable or admin reset) to re-enroll.",
+        )
+
+    hostname_hint = request.url.hostname if request.url else None
+    label = mfa_service.build_account_label(current["username"], hostname_hint)
+    otpauth_uri = mfa_service.build_otpauth_uri(label, secret_plain)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.started",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"secret_len": len(secret_plain)},  # NEVER log the secret itself
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+
+    return MfaEnrollStartResponse(
+        secret=secret_plain,
+        otpauth_uri=otpauth_uri,
+        expires_in=PENDING_ENROLLMENT_TTL_SECONDS,
+    )
+
+
+@router.post(
+    "/enroll/confirm",
+    summary="Confirm enrollment with a TOTP code; returns 10 backup codes once",
+    response_model=MfaEnrollConfirmResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_confirm, key_func=mfa_rate_limit_key)
+async def mfa_enroll_confirm(
+    payload: MfaEnrollConfirmRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # Pre-generate plain codes & hashes outside the DB transaction so we
+    # never hold a row lock for ~2-3s of bcrypt work.
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    # failure: { reason, attempts, http_status, detail }; success when None.
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            # Lock the pending row so concurrent /enroll/confirm calls for the
+            # same user can't both consume the same pending enrollment.
+            pending = await conn.fetchrow(
+                """
+                SELECT secret_encrypted, attempts, expires_at
+                  FROM mfa_pending_enrollments
+                 WHERE user_id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+
+            if not pending:
+                failure = {
+                    "reason": "no_pending",
+                    "http_status": 410,
+                    "detail": "No pending enrollment; start again",
+                }
+            else:
+                from datetime import datetime as _dt
+                if pending["expires_at"] and pending["expires_at"] < _dt.utcnow():
+                    await conn.execute(
+                        "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                        current["id"],
+                    )
+                    failure = {
+                        "reason": "expired",
+                        "http_status": 410,
+                        "detail": "Enrollment expired; start again",
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(pending["secret_encrypted"])
+                    if not secret_plain:
+                        await conn.execute(
+                            "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                            current["id"],
+                        )
+                        failure = {
+                            "reason": "unreadable",
+                            "http_status": 500,
+                            "detail": "Pending enrollment unreadable; start again",
+                        }
+                    else:
+                        ok, _step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, None
+                        )
+                        if not ok:
+                            new_attempts = (pending["attempts"] or 0) + 1
+                            if new_attempts >= PENDING_ENROLLMENT_MAX_ATTEMPTS:
+                                await conn.execute(
+                                    "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "too_many_attempts",
+                                    "attempts": new_attempts,
+                                    "http_status": 410,
+                                    "detail": "Enrollment invalidated; start again",
+                                }
+                            else:
+                                await conn.execute(
+                                    "UPDATE mfa_pending_enrollments SET attempts = $1 WHERE user_id = $2",
+                                    new_attempts,
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "invalid_code",
+                                    "attempts": new_attempts,
+                                    "http_status": 401,
+                                    "detail": "Invalid code",
+                                }
+                        else:
+                            # Verified — finalize state inside the transaction.
+                            await conn.execute(
+                                """
+                                UPDATE users
+                                   SET mfa_enabled = TRUE,
+                                       mfa_method = 'totp',
+                                       mfa_secret_encrypted = $1,
+                                       mfa_enrolled_at = NOW(),
+                                       mfa_last_used_totp_step = NULL,
+                                       mfa_last_used_at = NULL
+                                 WHERE id = $2
+                                """,
+                                pending["secret_encrypted"],
+                                current["id"],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                current["id"],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        # Log AFTER commit so the audit row reflects what actually persisted.
+        await log_user_activity(
+            user_id=current["id"],
+            action="mfa.enrollment.failed",
+            resource_type="mfa",
+            resource_id=str(current["id"]),
+            details={
+                "reason": failure["reason"],
+                "attempts": failure.get("attempts"),
+            },
+            ip_address=ip,
+            user_agent=ua,
+        )
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.confirmed",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"method": "totp"},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaEnrollConfirmResponse(enabled=True, backup_codes=plain_codes, method="totp")
+
+
+# ---------------------------------------------------------------------------
+# Disable + regenerate
+# ---------------------------------------------------------------------------
+
+
+async def _verify_user_code(conn, user_row: dict, code: str) -> Optional[str]:
+    """Verify a TOTP-or-backup code against the user's stored secret.
+
+    Returns the method used ('totp' / 'backup') on success, None on failure.
+
+    On TOTP success the step counter is bumped *atomically* — the UPDATE
+    only succeeds if no other request consumed the same (or a newer) step
+    in between. On backup success the consumed row's used_at is set with
+    an atomic ``WHERE used_at IS NULL RETURNING id`` pattern.
+    """
+    secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+    if secret_plain:
+        ok, step = mfa_service.verify_totp_with_replay_guard(
+            secret_plain, code, user_row["mfa_last_used_totp_step"]
+        )
+        if ok:
+            bumped = await conn.fetchval(
+                """
+                UPDATE users
+                   SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                 WHERE id = $2
+                   AND (mfa_last_used_totp_step IS NULL
+                        OR mfa_last_used_totp_step < $1)
+                 RETURNING id
+                """,
+                step,
+                user_row["id"],
+            )
+            if bumped:
+                return "totp"
+
+    rows = await conn.fetch(
+        """
+        SELECT id, code_hash FROM mfa_backup_codes
+         WHERE user_id = $1 AND used_at IS NULL
+        """,
+        user_row["id"],
+    )
+    for row in rows:
+        if await mfa_service.check_backup_code(code, row["code_hash"]):
+            consumed = await conn.fetchval(
+                """
+                UPDATE mfa_backup_codes
+                   SET used_at = NOW()
+                 WHERE id = $1 AND used_at IS NULL
+                 RETURNING id
+                """,
+                row["id"],
+            )
+            if consumed:
+                return "backup"
+    return None
+
+
+@router.post("/disable", summary="Disable MFA (requires current TOTP or backup)")
+@limiter.limit(MFA_LIMITS.disable, key_func=mfa_rate_limit_key)
+async def mfa_disable(
+    payload: MfaDisableRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        user_row = await conn.fetchrow(
+            """
+            SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users WHERE id = $1
+            """,
+            current["id"],
+        )
+        if not user_row or not user_row["mfa_enabled"]:
+            raise HTTPException(status_code=400, detail="MFA is not enabled")
+
+        method_used = await _verify_user_code(conn, dict(user_row), payload.code)
+        if not method_used:
+            await log_user_activity(
+                user_id=current["id"],
+                action="mfa.disable.failed",
+                resource_type="mfa",
+                resource_id=str(current["id"]),
+                details={"reason": "invalid_code"},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get("user-agent"),
+            )
+            raise HTTPException(status_code=401, detail="Invalid code")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                current["id"],
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", current["id"]
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.disabled.self",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"verified_via": method_used},
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"enabled": False}
+
+
+@router.post(
+    "/backup-codes/regenerate",
+    summary="Issue 10 fresh backup codes (TOTP required)",
+    response_model=MfaRegenerateBackupResponse,
+)
+@limiter.limit(MFA_LIMITS.regenerate_backup_codes, key_func=mfa_rate_limit_key)
+async def mfa_regenerate_backup_codes(
+    payload: MfaRegenerateBackupRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # bcrypt-hash the new codes outside the DB transaction (~2-3s of CPU work).
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            user_row = await conn.fetchrow(
+                """
+                SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+                  FROM users WHERE id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+            if not user_row or not user_row["mfa_enabled"]:
+                failure = {"http_status": 400, "detail": "MFA is not enabled"}
+            else:
+                secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+                if not secret_plain:
+                    failure = {"http_status": 500, "detail": "MFA secret unreadable"}
+                else:
+                    ok, step = mfa_service.verify_totp_with_replay_guard(
+                        secret_plain, payload.code, user_row["mfa_last_used_totp_step"]
+                    )
+                    if not ok:
+                        failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                    else:
+                        bumped = await conn.fetchval(
+                            """
+                            UPDATE users
+                               SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                             WHERE id = $2
+                               AND (mfa_last_used_totp_step IS NULL
+                                    OR mfa_last_used_totp_step < $1)
+                             RETURNING id
+                            """,
+                            step,
+                            current["id"],
+                        )
+                        if not bumped:
+                            failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                        else:
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.backup_codes.regenerated",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"codes_count": len(plain_codes)},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaRegenerateBackupResponse(backup_codes=plain_codes)
+
+
+# ---------------------------------------------------------------------------
+# Admin operations
+# ---------------------------------------------------------------------------
+
+
+@router.post("/admin-reset/{user_id}", summary="Admin: reset a single user's MFA")
+@limiter.limit(MFA_LIMITS.admin_reset, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset(
+    user_id: int,
+    payload: MfaAdminResetRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    if user_id == admin["id"]:
+        raise HTTPException(status_code=400, detail="Use /api/mfa/disable for self-reset")
+
+    conn = await get_database_connection()
+    try:
+        target = await conn.fetchrow(
+            "SELECT id, username, mfa_enabled FROM users WHERE id = $1", user_id
+        )
+        if not target:
+            raise HTTPException(status_code=404, detail="User not found")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                user_id,
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", user_id
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_reset",
+        resource_type="mfa",
+        resource_id=str(user_id),
+        details={
+            "target_user_id": user_id,
+            "target_username": target["username"],
+            "admin_user_id": admin["id"],
+            "reason": payload.reason,
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset": True, "user_id": user_id}
+
+
+@router.post(
+    "/admin-reset-all",
+    summary="Admin: emergency reset of MFA for all users (double confirm)",
+)
+@limiter.limit(MFA_LIMITS.admin_reset_all, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset_all(
+    payload: MfaAdminResetAllRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    # Pydantic's Literal already enforces the magic string, but check defensively too.
+    if payload.confirm != "RESET ALL MFA":
+        raise HTTPException(status_code=400, detail="Invalid confirmation string")
+
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            reset_count = await conn.fetchval(
+                """
+                WITH affected AS (
+                    UPDATE users
+                       SET mfa_enabled = FALSE,
+                           mfa_method = NULL,
+                           mfa_secret_encrypted = NULL,
+                           mfa_enrolled_at = NULL,
+                           mfa_last_used_at = NULL,
+                           mfa_last_used_totp_step = NULL
+                     WHERE mfa_enabled = TRUE
+                     RETURNING id
+                )
+                SELECT COUNT(*) FROM affected
+                """
+            )
+            await conn.execute("DELETE FROM mfa_backup_codes")
+            await conn.execute("DELETE FROM mfa_pending_logins")
+            await conn.execute("DELETE FROM mfa_pending_enrollments")
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_bulk_reset",
+        resource_type="mfa",
+        resource_id=str(admin["id"]),
+        details={
+            "reset_count": int(reset_count or 0),
+            "reason": payload.reason,
+            "admin_user_id": admin["id"],
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset_count": int(reset_count or 0)}

backend/routers/request_logs.py

@@ -0,0 +1,514 @@
+"""v1.11.0 — the read/administration API for the unified request/response log.
+
+Endpoints (declaration order matters — see below):
+
+    GET    /api/request-logs/settings   requestlog.manage
+    PUT    /api/request-logs/settings   requestlog.manage
+    GET    /api/request-logs/stats      requestlog.read
+    POST   /api/request-logs/purge      requestlog.manage
+    GET    /api/request-logs            requestlog.read
+    GET    /api/request-logs/{log_id}   requestlog.read
+
+`/{log_id}` is a single-segment path, so FastAPI — which matches in declaration
+order — would shadow `/settings`, `/stats` and `/purge` if it came first. The
+literals are therefore declared before it. (This is the mirror image of the
+trap in routers/settings.py, where `GET /{category}` sits at the top of the
+file and swallows every literal route added after it.)
+
+Settings are stored in `system_settings` under the `requestlog` category, so
+`GET /api/settings/requestlog` still reads them, but writes go through THIS
+router: the generic `PUT /api/settings/{category}` stringifies values with
+`str(value)`, which turns `True` into `'True'` — not valid JSON, and the
+`::jsonb` cast then fails.
+"""
+import json
+import logging
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from fastapi import APIRouter, Header, HTTPException, Query
+from pydantic import BaseModel, Field, field_validator
+
+from auth_middleware import check_user_permission, get_current_user_from_token
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_settings import (
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATH_LENGTH,
+    SETTINGS_CATEGORY,
+    config_from_mapping,
+    get_config,
+    load_settings_rows,
+    refresh_config,
+    set_config,
+)
+from utils.request_log_sink import TARGET_INBOUND_AGENT, request_log_sink
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/request-logs", tags=["Request Logs"])
+
+# Columns returned by the list endpoint. Bodies and headers are detail-only:
+# a 200-row page carrying two 8 KB JSONB blobs per row is a 3 MB response.
+_LIST_COLUMNS = """
+    id, request_id, direction, target, method, url, path, status_code,
+    status_class, duration_ms, user_id, username, host(client_ip) AS client_ip,
+    error, request_body_bytes, response_body_bytes, truncated, created_at
+"""
+
+_JSONB_COLUMNS = ("query_params", "request_headers", "request_body",
+                  "response_headers", "response_body")
+
+
+class RequestLogSettings(BaseModel):
+    """Operator-tunable capture + retention policy."""
+
+    enabled: bool = True
+    capture_inbound: bool = True
+    capture_outbound: bool = True
+    capture_bodies: bool = True
+    capture_get: bool = True
+    capture_agent_success: bool = False
+    max_body_bytes: int = Field(8192, ge=0, le=262144)
+    sample_rate: float = Field(1.0, ge=0.0, le=1.0)
+    exclude_paths: List[str] = Field(
+        default_factory=lambda: list(DEFAULT_EXCLUDE_PATHS),
+        max_length=MAX_EXCLUDE_PATHS,
+    )
+    success_retention_days: int = Field(7, ge=1, le=365)
+    error_retention_days: int = Field(30, ge=1, le=365)
+    max_rows: int = Field(500000, ge=1000, le=50_000_000)
+    prune_interval_minutes: int = Field(60, ge=5, le=1440)
+
+    @field_validator("exclude_paths")
+    @classmethod
+    def _validate_paths(cls, value: List[str]) -> List[str]:
+        for entry in value:
+            if not entry.startswith("/"):
+                raise ValueError("exclude_paths entries must start with '/'")
+            if len(entry) > MAX_EXCLUDE_PATH_LENGTH:
+                raise ValueError(
+                    f"exclude_paths entries must be <= {MAX_EXCLUDE_PATH_LENGTH} characters"
+                )
+        return value
+
+
+async def _require(authorization: Optional[str], action: str) -> Dict[str, Any]:
+    """Authenticate, then enforce `requestlog.<action>`.
+
+    `current_user=` is passed through so the admin bypass in
+    check_user_permission short-circuits without a second DB round-trip.
+    """
+    current_user = await get_current_user_from_token(authorization)
+    allowed = await check_user_permission(
+        current_user["id"], "requestlog", action, current_user=current_user
+    )
+    if not allowed:
+        raise HTTPException(
+            status_code=403,
+            detail=f"Insufficient permissions: requestlog.{action} required",
+        )
+    return current_user
+
+
+async def _can_manage(current_user: Dict[str, Any]) -> bool:
+    return await check_user_permission(
+        current_user["id"], "requestlog", "manage", current_user=current_user
+    )
+
+
+def _parse_jsonb(value: Any) -> Any:
+    """asyncpg has no JSONB codec on this pool, so JSONB comes back as raw
+    text. This router is a new contract, so it parses server-side and returns
+    real JSON rather than pushing a JSON.parse() into the UI."""
+    if isinstance(value, str):
+        try:
+            return json.loads(value)
+        except (json.JSONDecodeError, ValueError):
+            return value
+    return value
+
+
+def _row_to_dict(row) -> Dict[str, Any]:
+    out = dict(row)
+    for key in _JSONB_COLUMNS:
+        if key in out:
+            out[key] = _parse_jsonb(out[key])
+    created = out.get("created_at")
+    if isinstance(created, datetime):
+        out["created_at"] = created.isoformat()
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Literal paths FIRST — see the module docstring.
+# ---------------------------------------------------------------------------
+
+
+@router.get("/settings")
+async def get_request_log_settings(authorization: Optional[str] = Header(None)):
+    """Current capture + retention policy, plus the shipped defaults so the UI
+    can offer a 'reset' without hardcoding them."""
+    await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        values = await load_settings_rows(conn)
+        config = config_from_mapping(values) if values else DEFAULT_CONFIG
+        return {
+            "settings": config.as_dict(),
+            "defaults": DEFAULT_CONFIG.as_dict(),
+            "category": SETTINGS_CATEGORY,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.put("/settings")
+async def update_request_log_settings(
+    body: RequestLogSettings,
+    authorization: Optional[str] = Header(None),
+):
+    """Persist the policy and apply it immediately.
+
+    `refresh_config()` at the end is what makes an operator's change take
+    effect on the very next request instead of up to 30 seconds later, when
+    the writer loop would otherwise pick it up.
+    """
+    current_user = await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        updated = []
+        for suffix, value in body.model_dump().items():
+            await conn.execute(
+                """
+                INSERT INTO system_settings (key, value, category, updated_at, updated_by)
+                VALUES ($1, $2::jsonb, $3, $4, $5)
+                ON CONFLICT (key) DO UPDATE SET
+                    value = EXCLUDED.value,
+                    updated_at = EXCLUDED.updated_at,
+                    updated_by = EXCLUDED.updated_by
+                """,
+                f"{SETTINGS_CATEGORY}.{suffix}",
+                json.dumps(value),
+                SETTINGS_CATEGORY,
+                datetime.utcnow(),
+                current_user.get("id"),
+            )
+            updated.append(suffix)
+
+        # Apply in-process right away, then re-read so this worker's snapshot
+        # is exactly what is on disk.
+        set_config(config_from_mapping(body.model_dump()))
+        await refresh_config()
+
+        logger.info(
+            f"Request log settings updated by {current_user.get('username')}: {len(updated)} keys"
+        )
+        return {"message": f"Updated {len(updated)} settings", "settings": get_config().as_dict()}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error updating request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to update request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/stats")
+async def get_request_log_stats(
+    authorization: Optional[str] = Header(None),
+    hours: int = Query(24, ge=1, le=720),
+):
+    """Volume and error breakdown over a window, plus table-level totals and
+    this worker's sink counters (so a saturated queue is visible)."""
+    await _require(authorization, "read")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        by_direction = await conn.fetch(
+            """
+            SELECT direction,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors,
+                   COALESCE(ROUND(AVG(duration_ms))::int, 0)                   AS avg_duration_ms,
+                   COALESCE(MAX(duration_ms), 0)                               AS max_duration_ms
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY direction
+            """,
+            str(hours),
+        )
+
+        by_status = await conn.fetch(
+            """
+            SELECT status_class, COUNT(*) AS total
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY status_class
+            ORDER BY status_class
+            """,
+            str(hours),
+        )
+
+        by_target = await conn.fetch(
+            """
+            SELECT target,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors
+            FROM request_logs
+            WHERE target IS NOT NULL
+              AND created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY target
+            ORDER BY total DESC
+            LIMIT 20
+            """,
+            str(hours),
+        )
+
+        totals = await conn.fetchrow(
+            "SELECT COUNT(*) AS total_rows, MIN(created_at) AS oldest_at, "
+            "MAX(created_at) AS newest_at FROM request_logs"
+        )
+
+        return {
+            "window_hours": hours,
+            "by_direction": [dict(r) for r in by_direction],
+            "by_status_class": [dict(r) for r in by_status],
+            "by_target": [dict(r) for r in by_target],
+            "total_rows": (totals or {}).get("total_rows", 0),
+            "oldest_at": totals["oldest_at"].isoformat() if totals and totals["oldest_at"] else None,
+            "newest_at": totals["newest_at"].isoformat() if totals and totals["newest_at"] else None,
+            # THIS WORKER only. The sink is a module global, so with
+            # UVICORN_WORKERS > 1 each process keeps its own queue and its own
+            # counters, and whichever worker happens to serve this request is
+            # the one being reported. Labelled rather than aggregated: there is
+            # no cross-process channel here, and a number that looks fleet-wide
+            # but is not would understate drops by exactly the worker count.
+            "sink": {**request_log_sink.stats, "scope": "this worker only"},
+            "retention": {
+                "success_retention_days": get_config().success_retention_days,
+                "error_retention_days": get_config().error_retention_days,
+                "max_rows": get_config().max_rows,
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log stats: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log stats")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.post("/purge")
+async def purge_request_logs(authorization: Optional[str] = Header(None)):
+    """Run a retention pass now, ignoring the watermark.
+
+    This applies the CONFIGURED retention — it is not a 'delete everything'
+    button. It exists so an operator who has just lowered the retention does
+    not have to wait for the next scheduled pass to reclaim the space.
+    """
+    current_user = await _require(authorization, "manage")
+    from utils.request_log_prune import prune_request_logs_if_due
+
+    counts = await prune_request_logs_if_due(force=True)
+    logger.info(f"Manual request log purge by {current_user.get('username')}: {counts}")
+    return {
+        "message": "Retention pass completed",
+        "removed": {
+            "success": counts.get("success", 0),
+            "error": counts.get("error", 0),
+            "overflow": counts.get("overflow", 0),
+        },
+        "ran": bool(counts.get("ran")),
+    }
+
+
+# ---------------------------------------------------------------------------
+# List, then the catch-all detail route LAST.
+# ---------------------------------------------------------------------------
+
+
+@router.get("")
+async def list_request_logs(
+    authorization: Optional[str] = Header(None),
+    direction: Optional[str] = Query(None, pattern="^(inbound|outbound)$"),
+    status_class: Optional[int] = Query(None, ge=0, le=5),
+    method: Optional[str] = Query(None, max_length=10),
+    target: Optional[str] = Query(None, max_length=32),
+    user_id: Optional[int] = Query(None, ge=1),
+    path_prefix: Optional[str] = Query(None, max_length=200),
+    q: Optional[str] = Query(None, max_length=200),
+    request_id: Optional[str] = Query(None, max_length=64),
+    errors_only: bool = Query(False),
+    since: Optional[datetime] = Query(None),
+    until: Optional[datetime] = Query(None),
+    min_duration_ms: Optional[int] = Query(None, ge=0),
+    limit: int = Query(50, ge=1, le=500),
+    offset: int = Query(0, ge=0),
+):
+    """Filtered, server-paginated list. Bodies are not included — use the
+    detail endpoint for those."""
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    where: List[str] = []
+    params: List[Any] = []
+
+    def add(clause_template: str, value: Any) -> None:
+        params.append(value)
+        where.append(clause_template.format(n=len(params)))
+
+    if direction:
+        add("direction = ${n}", direction)
+    if status_class is not None:
+        add("status_class = ${n}", status_class)
+    if method:
+        add("method = ${n}", method.upper())
+    if target:
+        add("target = ${n}", target)
+    if user_id is not None:
+        add("user_id = ${n}", user_id)
+    if path_prefix:
+        add("path LIKE ${n} || '%'", path_prefix)
+    if q:
+        # Substring search has no index to lean on; it is the deliberately slow
+        # filter and should be combined with a time window.
+        add("url ILIKE '%' || ${n} || '%'", q)
+    if request_id:
+        add("request_id = ${n}", request_id)
+    if errors_only:
+        where.append("(status_class = 0 OR status_class >= 4)")
+    if since:
+        add("created_at >= ${n}", since)
+    if until:
+        add("created_at <= ${n}", until)
+    if min_duration_ms is not None:
+        add("duration_ms >= ${n}", min_duration_ms)
+
+    # Self-scoping. Captured bodies are a broader disclosure surface than the
+    # existing activity log, so a caller holding only `requestlog.read` sees
+    # their OWN inbound requests, plus the fleet's. `requestlog.manage` (and the
+    # is_admin bypass inside it) lifts the restriction.
+    #
+    # The agent clause is not a widening for its own sake, it is what makes the
+    # `operator` grant do what the migration says it is for: "operators debug
+    # failing applies and ACME orders, so they get read access to the request
+    # log". An apply fails on the NODE, and the node reports that back over its
+    # own API key - so the row carrying the diagnosis is an agent row with
+    # `user_id IS NULL`, which own-rows-only scoping hid from exactly the role
+    # the grant was written for. Scoped on `target`, not on `user_id IS NULL`:
+    # anonymous inbound traffic (failed logins and their usernames, unauthorised
+    # probes) is NOT agent traffic and stays admin-only.
+    if not can_manage:
+        params.append(current_user["id"])
+        own = f"user_id = ${len(params)}"
+        params.append(TARGET_INBOUND_AGENT)
+        where.append(
+            f"(direction = 'inbound' AND ({own} OR target = ${len(params)}))"
+        )
+
+    where_sql = (" WHERE " + " AND ".join(where)) if where else ""
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        rows = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs{where_sql} "
+            f"ORDER BY id DESC LIMIT ${len(params) + 1} OFFSET ${len(params) + 2}",
+            *params, limit, offset,
+        )
+
+        # Bounded count: an unfiltered COUNT(*) over a multi-million-row table
+        # is a sequential scan on every page change. Cap it and tell the client
+        # the number is a floor.
+        count_cap = 10001
+        counted = await conn.fetchval(
+            f"SELECT COUNT(*) FROM (SELECT 1 FROM request_logs{where_sql} LIMIT {count_cap}) t",
+            *params,
+        )
+        total = int(counted or 0)
+
+        return {
+            "logs": [_row_to_dict(r) for r in rows],
+            "total": total,
+            "total_is_estimate": total >= count_cap,
+            "limit": limit,
+            "offset": offset,
+            "scoped_to_self": not can_manage,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing request logs: {e}")
+        raise HTTPException(status_code=500, detail="Failed to list request logs")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/{log_id}")
+async def get_request_log(log_id: int, authorization: Optional[str] = Header(None)):
+    """One exchange in full, plus every other row sharing its `request_id`.
+
+    That `related` list is the point of the feature: one inbound API call and
+    the ACME / DNS / agent calls it triggered read as a single trace.
+    """
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            "SELECT *, host(client_ip) AS client_ip_text FROM request_logs WHERE id = $1",
+            log_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        record = _row_to_dict(row)
+        record["client_ip"] = record.pop("client_ip_text", None)
+
+        if not can_manage and not (
+            record.get("direction") == "inbound"
+            and (
+                record.get("user_id") == current_user["id"]
+                or record.get("target") == TARGET_INBOUND_AGENT
+            )
+        ):
+            # Same self-scoping rule as the list endpoint. 404 rather than 403
+            # so the endpoint does not confirm that a given id exists.
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        related = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs "
+            "WHERE request_id = $1 AND id <> $2 ORDER BY id ASC LIMIT 100",
+            record["request_id"], log_id,
+        )
+
+        return {"log": record, "related": [_row_to_dict(r) for r in related]}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log {log_id}: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log entry")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)

backend/routers/settings.py

@@ -1,10 +1,12 @@
 from fastapi import APIRouter, HTTPException, Header
 from pydantic import BaseModel
 from typing import Dict, Any
+import json
 import logging
 from datetime import datetime
 
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
 
 logger = logging.getLogger(__name__)
 
@@ -50,6 +52,39 @@ async def get_settings_by_category(category: str, authorization: str = Header(No
         await close_database_connection(conn)
 
 
+def _validate_acme_challenge_backend_url(value):
+    """Validate `acme.challenge_backend_url` exactly as the config renderer reads it.
+
+    Settings values are stored as jsonb, so the renderer json.loads them before use
+    (services/haproxy_config.py). Validating the raw column text instead of the
+    decoded string would check the quoting rather than the URL.
+    """
+    decoded = value
+    if isinstance(decoded, str):
+        try:
+            decoded = json.loads(decoded)
+        except (json.JSONDecodeError, TypeError):
+            pass
+    if decoded is None:
+        return
+    try:
+        validate_acme_backend_url(str(decoded))
+    except AcmeBackendUrlError as exc:
+        raise HTTPException(
+            status_code=422,
+            detail=f"acme.challenge_backend_url: {exc}",
+        ) from None
+
+
+# Per-key validators for settings that end up in generated configuration or in
+# outbound requests. Everything else is still written through unchecked; this is a
+# deliberate allow-list of the keys where a bad value causes silent breakage rather
+# than an obvious one.
+_SETTING_VALIDATORS = {
+    "acme.challenge_backend_url": _validate_acme_challenge_backend_url,
+}
+
+
 @router.put("/{category}")
 async def update_settings_by_category(
     category: str,
@@ -59,6 +94,13 @@ async def update_settings_by_category(
     current_user = await _get_admin_user(authorization)
     conn = await get_database_connection()
     try:
+        # Validate the whole batch before writing any of it, so a rejected key cannot
+        # leave the category half-applied.
+        for key_suffix, value in body.settings.items():
+            validator = _SETTING_VALIDATORS.get(f"{category}.{key_suffix}")
+            if validator is not None:
+                validator(value)
+
         updated = []
         for key_suffix, value in body.settings.items():
             full_key = f"{category}.{key_suffix}"
@@ -104,19 +146,52 @@ async def test_acme_connection(authorization: str = Header(None), directory_url:
         finally:
             await close_database_connection(conn)
 
+    # SECURITY (GHSA-3vh4-gvxx-wm2p): validate the URL before any outbound request
+    # (https-only; block loopback/RFC1918/link-local/cloud-metadata after DNS),
+    # pin the connector to IPv4, and never follow redirects. Also do NOT reflect
+    # arbitrary upstream JSON keys back to the caller — that was an information-
+    # disclosure oracle. Only report presence of the FIXED, known ACME directory
+    # field names (never attacker-controlled data).
+    from utils.ssrf_guard import assert_public_url, safe_connector, SSRFValidationError
+
+    directory_url = str(directory_url)
+    try:
+        await assert_public_url(directory_url)
+    except SSRFValidationError as e:
+        return {"success": False, "error": f"Refused to fetch directory URL: {e}"}
+
+    _KNOWN_ACME_FIELDS = ["newNonce", "newAccount", "newOrder", "newAuthz", "revokeCert", "keyChange"]
     try:
         import aiohttp
-        async with aiohttp.ClientSession() as session:
-            async with session.get(str(directory_url), timeout=aiohttp.ClientTimeout(total=10)) as resp:
-                if resp.status == 200:
-                    data = await resp.json()
-                    return {
-                        "success": True,
-                        "directory": str(directory_url),
-                        "endpoints": list(data.keys()) if isinstance(data, dict) else []
-                    }
-                else:
-                    return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
+        # v1.11.0: this handler returns str(e) to the caller and logs nothing —
+        # the span gives the failed probe a durable record.
+        from utils.http_instrumentation import outbound_span, TARGET_SETTINGS_PROBE
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
+            async with outbound_span(
+                target=TARGET_SETTINGS_PROBE, method="GET", url=directory_url
+            ) as span:
+                async with session.get(
+                    directory_url,
+                    timeout=aiohttp.ClientTimeout(total=10),
+                    allow_redirects=False,
+                ) as resp:
+                    if resp.status == 200:
+                        data = await resp.json(content_type=None)
+                        span.set_response(resp.status, getattr(resp, "headers", None), data)
+                        if not isinstance(data, dict):
+                            return {"success": False, "error": "Directory URL did not return a JSON object"}
+                        present = [k for k in _KNOWN_ACME_FIELDS if k in data]
+                        if not present:
+                            return {"success": False, "error": "Response is not a valid ACME directory"}
+                        return {
+                            "success": True,
+                            "directory": directory_url,
+                            "endpoints": present,
+                        }
+                    else:
+                        span.set_response(resp.status, getattr(resp, "headers", None))
+                        return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
     except HTTPException:
         raise
     except Exception as e:

backend/routers/ssl.py

@@ -9,7 +9,7 @@ from datetime import datetime, timezone
 
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from models.ssl import SSLCertificate, SSLCertificateCreate, SSLCertificateUpdate, SSLCertificateResponse
 from utils.ssl_parser import parse_ssl_certificate, validate_private_key, validate_certificate_chain, format_certificate_info
 from utils.activity_log import log_user_activity
@@ -825,7 +825,8 @@ async def get_ssl_certificate(cert_id: int, authorization: str = Header(None)):
         logger.error(f"Error getting SSL certificate: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/certificates/{cert_id}/config-versions")
+@router.get("/certificates/{cert_id}/config-versions",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was unauthenticated
 async def get_ssl_certificate_config_versions(cert_id: int):
     """Get config version history for specific SSL certificate"""
     try:

backend/routers/user.py

@@ -38,18 +38,19 @@ async def get_users(authorization: str = Header(None)):
         # Get users with their roles (only active users)
         try:
             users = await conn.fetch("""
-                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active, 
-                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at
+                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active,
+                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at,
+                       COALESCE(u.mfa_enabled, FALSE) AS mfa_enabled
                 FROM users u
                 WHERE u.is_active = TRUE
                 ORDER BY u.username
             """)
         except Exception as schema_error:
             logger.warning(f"Schema error in users query, using fallback: {schema_error}")
-            # Fallback query with minimal columns
+            # Fallback query with minimal columns (pre-MFA-migration deploys)
             users = await conn.fetch("""
                 SELECT id, username, email, is_active, is_admin, created_at
-                FROM users 
+                FROM users
                 WHERE is_active = TRUE
                 ORDER BY username
             """)

backend/routers/waf.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Request, Header
+from fastapi import APIRouter, HTTPException, Request, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 import logging
 import time
@@ -182,7 +183,8 @@ async def get_waf_stats(cluster_id: Optional[int] = None, authorization: str = H
         logger.error(f"Error fetching WAF stats: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules")
+@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): WAF rule definitions aid bypass crafting
 async def get_waf_rules(cluster_id: Optional[int] = None):
     """
     # Get WAF Rules

backend/services/acme_diagnostics.py

@@ -8,9 +8,13 @@ suitable for an Antd Tabs/Steps display.
 Key constraints (Section 3.3 of the v1.5.0 plan):
 - DNS resolution uses stdlib socket.gethostbyname_ex via run_in_executor (we
   intentionally avoid pulling aiodns as a runtime dep for v1.5.0).
-- Port-80 probe is HEAD-only, target locked to the order's domains, success on
-  HTTP 200 OR 404, warns on egress timeout (don't fail-hard — corp egress
-  policies often blackhole outbound 80).
+- Port-80 probe is a GET (not HEAD) locked to the order's domains, because the
+  status code alone cannot tell a working challenge endpoint from a web UI: a
+  reverse proxy that has lost its /.well-known/acme-challenge/ location serves
+  its SPA with HTTP 200. The body's shape decides. Warns rather than fails on
+  egress timeout (corp egress policies often blackhole outbound 80) and on a
+  wrong responder (this probe sees the PUBLIC domain, not the challenge backend,
+  so it is evidence rather than a verdict).
 - All checks have hard wall-clock timeouts (asyncio.wait_for) to bound impact
   on the API event loop.
 - humanize_error_detail covers >= 11 RFC8555 problem types and is backwards
@@ -116,6 +120,10 @@ _PROBLEM_HUMANIZED: Dict[str, Dict[str, str]] = {
         "title": "HTTP-01 challenge response mismatch",
         "hint": "The CA fetched the challenge URL but received the wrong key authorization. Confirm the challenge was served from the right backend.",
     },
+    "urn:ietf:params:acme:error:externalAccountRequired": {
+        "title": "External Account Binding (EAB) required",
+        "hint": "This CA (e.g. ZeroSSL, Google) requires EAB. Enter the EAB Key ID and HMAC Key from your CA account when registering.",
+    },
     "urn:ietf:params:acme:error:invalidContact": {
         "title": "Invalid contact email",
         "hint": "The ACME account email is malformed. Update the LE account email.",
@@ -199,6 +207,23 @@ def humanize_error_detail(error_detail: Any) -> Dict[str, Any]:
     status = parsed.get("status")
     subproblems = parsed.get("subproblems") or []
 
+    # Issue #35: DNS-01 failures are recorded as {stage, reason, timestamp} (no RFC8555 "type"),
+    # so without this fallback the humanized alert would show a bare "ACME error" with no message.
+    # Surface the reason and a targeted hint so the operator knows exactly what to fix.
+    if not problem_type and parsed.get("reason"):
+        reason = str(parsed.get("reason"))
+        message = message or reason
+        title = "DNS-01 validation failed"
+        rlow = reason.lower()
+        if "decrypt" in rlow or "credential" in rlow:
+            hint = hint or "Re-enter the DNS provider credentials for this account in ACME Automation."
+        elif "zone" in rlow:
+            hint = hint or "Confirm the domain's DNS zone is managed by the configured provider and the token has access to it."
+        elif "deadline" in rlow or "expired" in rlow or "confirm" in rlow:
+            hint = hint or "The manual confirmation window passed. Create a new certificate request and publish the TXT record promptly."
+        else:
+            hint = hint or "Check the DNS TXT record and provider credentials, then retry."
+
     out = {
         "title": title,
         "message": message,
@@ -309,10 +334,41 @@ async def check_dns(domains: List[str]) -> Dict[str, Any]:
     )
 
 
+# Enough to classify a response without turning the diagnostic into a way to pull
+# arbitrary amounts of a third party's content into our JSON.
+_PROBE_BODY_LIMIT = 65536
+
+
+def _classify_probe_body(body: bytes, content_type: str) -> str:
+    """Coarse shape of a probe response: html | json | text | empty | binary.
+
+    The body itself is deliberately NOT retained anywhere — the shape is all that is
+    needed to tell "served me a web page" from "served me a token", and keeping the
+    bytes would open a new read surface onto whatever is behind the address.
+    """
+    if not body:
+        return "empty"
+    ct = (content_type or "").lower()
+    head = body[:512].lstrip().lower()
+    if ct.startswith("text/html") or head.startswith((b"<!doctype", b"<html")):
+        return "html"
+    if ct.startswith("application/json") or head[:1] in (b"{", b"["):
+        return "json"
+    try:
+        body.decode("utf-8")
+    except UnicodeDecodeError:
+        return "binary"
+    return "text"
+
+
 async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict[str, Any]:
-    """Probe HTTP-01 readiness on port 80 with a HEAD request to a synthetic
-    challenge URL. Success on 200 OR 404 (404 means the well-known path is
-    served but no challenge yet — fine).
+    """Probe HTTP-01 readiness on port 80 with a GET to a synthetic challenge URL.
+
+    404 means the path is served but no challenge is outstanding, which is fine. A
+    200 is only fine if the body is NOT a web page: a proxy that has lost its
+    /.well-known/acme-challenge/ route falls through to its catch-all and answers
+    200 with index.html, which a status-code-only check accepts as healthy while
+    every real validation fails.
 
     On egress timeout we WARN rather than FAIL because many corporate egress
     policies blackhole port 80 outbound; that does not impair LE's ingress
@@ -377,12 +433,84 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
                 continue
             url = f"http://{d}/.well-known/acme-challenge/diagnostic-probe"
             try:
-                async with session.head(url, allow_redirects=False) as resp:
-                    targets.append({
-                        "domain": d,
-                        "status": resp.status,
-                        "ok": resp.status in (200, 404),
-                    })
+                # v1.11.0: recorded as an outbound row so a failing port-80 probe is
+                # diagnosable after the fact, not only while the panel is open.
+                #
+                # MERGE NOTE: the feature branch instrumented a `session.head(...)`
+                # probe, which is what this function did when that branch was cut.
+                # It has since become a GET, because the status code alone cannot
+                # tell a working challenge endpoint from a SPA catch-all serving
+                # index.html with HTTP 200 (v1.10.x). Reverting to HEAD to gain the
+                # log row would put that bug straight back, so the GET probe below
+                # is the one that is wrapped.
+                #
+                # GET, not HEAD: the status code alone cannot tell a working challenge
+                # endpoint from a SPA. A reverse proxy that has lost its
+                # /.well-known/acme-challenge/ location falls through to its catch-all
+                # and serves index.html with HTTP 200 — which the old
+                # `status in (200, 404)` rule accepted as healthy while every real
+                # validation failed. Only the body distinguishes them.
+                from utils.http_instrumentation import outbound_span, TARGET_ACME_DIAG
+
+                async with outbound_span(
+                    target=TARGET_ACME_DIAG, method="GET", url=url, capture_body=False
+                ) as span:
+                    async with session.get(url, allow_redirects=False) as resp:
+                        # The body is EVIDENCE, not a precondition. If it cannot be read —
+                        # connection reset mid-response, a server that hangs after headers —
+                        # fall back to the status-only semantics this check has always had
+                        # rather than turning a healthy 404 into a hard failure. The stricter
+                        # rule below applies only when there is something to judge.
+                        try:
+                            body = await resp.content.read(_PROBE_BODY_LIMIT)
+                        except Exception:
+                            body = None
+                        content_type = (resp.headers.get("content-type") or "").split(";")[0].strip()
+                        body_class = (
+                            _classify_probe_body(body, content_type) if body is not None else "unread"
+                        )
+                        target = {
+                            "domain": d,
+                            "status": resp.status,
+                            "content_type": content_type or None,
+                            "body_len": len(body) if body is not None else None,
+                            "body_class": body_class,
+                        }
+                        # The VERDICT goes in the log row, not the body. `body` here is
+                        # up to 64 KB of a third party's page (`_PROBE_BODY_LIMIT`);
+                        # storing it would put an arbitrary remote document into the
+                        # audit table for every probed domain. The classification is
+                        # what an operator reads back anyway.
+                        span.set_response(
+                            resp.status,
+                            getattr(resp, "headers", None),
+                            {
+                                "probe": "acme-http01",
+                                "content_type": content_type or None,
+                                "body_len": len(body) if body is not None else None,
+                                "body_class": body_class,
+                            },
+                        )
+                        if resp.status == 200 and body_class == "html":
+                            # Reachable, wrong responder. Reported as a warning rather than
+                            # a failure: this check probes the PUBLIC domain and cannot see
+                            # the challenge backend, so it is evidence, not a verdict — and
+                            # a new `fail` here would block the site wizard on upgrade day
+                            # for every install.
+                            target["warn"] = True
+                            target["diagnosis"] = (
+                                "responded 200 with an HTML page, not a challenge token — "
+                                "the request is reaching a web UI instead of the ACME endpoint"
+                            )
+                        elif resp.status in (301, 302, 303, 307, 308):
+                            target["warn"] = True
+                            target["redirect_location"] = resp.headers.get("location")
+                            target["diagnosis"] = (
+                                "redirected instead of serving the challenge path"
+                            )
+                        else:
+                            target["ok"] = resp.status in (200, 404)
+                        targets.append(target)
             except asyncio.TimeoutError:
                 targets.append({"domain": d, "error": "egress timeout", "warn": True})
                 skip_reason = "egress timeout"
@@ -404,7 +532,11 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
             details={"targets": targets},
             duration_ms=duration_ms,
         )
-    if warns and not [t for t in targets if t.get("ok")]:
+    # Surface warnings even when OTHER domains answered correctly. The old condition
+    # ("warn only if nothing succeeded") hid the single most diagnostic outcome there
+    # is: a multi-domain certificate where one name reaches a web UI instead of the
+    # challenge endpoint reported a clean pass.
+    if warns:
         # R18b audit fix (round 7): branch the rollup message on the
         # actual cause. Pre-fix the message was always "Egress to
         # port 80 appears blocked" — even when every target was
@@ -414,6 +546,36 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
         # corporate firewall logs while the real cause was an
         # internal-only DNS A record. Also harden against
         # `skip_reason=None` so the message never reads "(None)".
+        # Wrong-responder warnings take priority over every other cause: they are the
+        # only ones that mean "your server answered, and answered wrong", which is a
+        # different problem from "we could not test".
+        wrong_responder = [t for t in targets if t.get("diagnosis")]
+        if wrong_responder:
+            first = wrong_responder[0]
+            if first.get("body_class") == "html":
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} with an HTML "
+                    f"page ({first.get('content_type') or 'unknown type'}, "
+                    f"{first.get('body_len')} bytes) instead of a challenge token. The "
+                    "path is reaching a web interface, not the ACME endpoint — check "
+                    "that the reverse proxy in front of OpenManager routes "
+                    "/.well-known/acme-challenge/ to the API."
+                )
+            else:
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} "
+                    f"({first.get('diagnosis')})"
+                )
+            return _check_result(
+                "port80",
+                "Port 80 reachability",
+                "warn",
+                human,
+                severity="warn",
+                details={"targets": targets},
+                duration_ms=duration_ms,
+            )
+
         ssrf_skip = any(
             "non-public" in (t.get("skip") or "")
             or "SSRF" in (t.get("skip") or "")
@@ -470,11 +632,20 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             duration_ms=int((time.time() - started) * 1000),
         )
 
+    # The WHERE clause is deliberately identical to the pre-existing one, so `not rows`
+    # still means exactly what it meant before and the `fail` branch below cannot fire
+    # in any situation where it previously passed. Narrowing it here (e.g. by adding a
+    # mode filter) would turn a tcp-only port-80 cluster from "ok" into "fail", and the
+    # site wizard blocks submit on any failing check — locking those installs the day
+    # this ships. Mode is examined afterwards, in Python, and only ever downgrades to
+    # `warn`.
     rows = await conn.fetch(
         """
-        SELECT id, name, bind_address, bind_port, mode, default_backend
-        FROM frontends
-        WHERE cluster_id = ANY($1::int[]) AND is_active = TRUE AND bind_port = 80
+        SELECT f.id, f.name, f.bind_address, f.bind_port, f.mode, f.default_backend,
+               f.cluster_id, c.acme_enabled
+        FROM frontends f
+        JOIN haproxy_clusters c ON c.id = f.cluster_id
+        WHERE f.cluster_id = ANY($1::int[]) AND f.is_active = TRUE AND f.bind_port = 80
         """,
         cluster_ids,
     )
@@ -489,13 +660,144 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             details={"cluster_ids": cluster_ids},
             duration_ms=duration_ms,
         )
+
+    # Character-for-character the renderer's normalisation (services/haproxy_config.py),
+    # so this can never disagree with what actually gets emitted.
+    http_rows = [r for r in rows if (r["mode"] or "http").strip().lower() == "http"]
+    if not http_rows:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                "The only port-80 frontend(s) in the target cluster(s) are in tcp mode. "
+                "A tcp-mode frontend cannot carry the /.well-known/acme-challenge/ ACL, "
+                "so HTTP-01 cannot be served — use DNS-01, or add an http-mode frontend "
+                "on port 80."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows]},
+            duration_ms=duration_ms,
+        )
+    rows = http_rows
+
+    # A frontend row proves only that the DATABASE describes port-80 routing. The
+    # renderer gates the challenge ACL on `acme_enabled`, and the nodes run whatever
+    # config was last APPLIED — so the row said "ok" during an incident where the
+    # live config had no usable challenge route at all. Check the two things the row
+    # cannot tell us. Both report `warn`, never `fail`: the site wizard blocks submit
+    # on any `fail`, so a new failing condition would lock every install on the day
+    # it ships.
+    # `.get()` rather than `[]`: the column gates a WARNING, so a row shape without
+    # it should not blow up the whole diagnostic. Absent means "assume enabled" —
+    # the applied-config check below is the authoritative one either way.
+    acme_off = sorted({r["cluster_id"] for r in rows if not r.get("acme_enabled", True)})
+    if acme_off:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {acme_off} have ACME Challenge Routing disabled, so the "
+                "generated config contains no /.well-known/acme-challenge/ route. "
+                "Enable it in Cluster Management, then apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "acme_disabled_clusters": acme_off},
+            duration_ms=duration_ms,
+        )
+
+    missing_in_applied = []
+    challenge_backends = {}
+    for cluster_id in sorted({r["cluster_id"] for r in rows}):
+        # Selector matched to the one the AGENT uses to fetch its config
+        # (routers/agent.py: status='APPLIED' AND is_active=TRUE), because the question
+        # here is "what are the nodes running right now?". Without `is_active` this can
+        # read a superseded row and report on a config that was never delivered.
+        # Extracting in SQL rather than pulling whole configs back per cluster: these
+        # files run to hundreds of KB on real installs.
+        applied = await conn.fetchrow(
+            """
+            SELECT position('use_backend _acme_challenge_backend' in config_content) > 0
+                       AS has_route,
+                   substring(config_content from 'server _acme_mgmt [^\\n]*') AS server_line
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+              AND config_content IS NOT NULL
+            ORDER BY created_at DESC LIMIT 1
+            """,
+            cluster_id,
+        )
+        if not applied or not applied["has_route"]:
+            missing_in_applied.append(cluster_id)
+            continue
+        server_line = (applied["server_line"] or "").strip()
+        challenge_backends[cluster_id] = (
+            server_line[len("server _acme_mgmt "):].strip() if server_line else None
+        )
+
+    if missing_in_applied:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {missing_in_applied} have no applied configuration carrying "
+                "the challenge route. The change exists in the database but the nodes are "
+                "still running an older config — apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_not_applied": missing_in_applied},
+            duration_ms=duration_ms,
+        )
+
+    # A backend section with no `server` line: the route exists, `haproxy -c` passes,
+    # and every challenge request gets a 503 from an empty backend. Without this branch
+    # the falsy target slips past the loopback filter below and the check reports "ok".
+    serverless = sorted(cid for cid, target in challenge_backends.items() if not target)
+    if serverless:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {serverless} route the challenge path to a backend that has "
+                "no server line, so every request returns 503. The configured ACME "
+                "Challenge Backend URL could not be resolved into an address — check it "
+                "in Cluster Management, or Settings > ACME for the global value."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_without_server": serverless},
+            duration_ms=duration_ms,
+        )
+
+    loopback = {
+        cid: target for cid, target in challenge_backends.items()
+        if target and target.split(":")[0].strip("[]").lower()
+        in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+    }
+    if loopback:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"The applied config points the challenge backend at {sorted(loopback.values())}. "
+                "HAProxy resolves that on the HAProxy node, so it means the node itself, not "
+                "this management server. Set ACME Challenge Backend URL to a routable address."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
+            duration_ms=duration_ms,
+        )
+
     return _check_result(
         "routing",
         "HAProxy routing",
         "ok",
-        f"Found {len(rows)} HTTP frontend(s) on port 80",
+        f"Found {len(rows)} HTTP frontend(s) on port 80; challenge route present in applied config",
         severity="info",
-        details={"frontends": [dict(r) for r in rows]},
+        details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
         duration_ms=duration_ms,
     )
 
@@ -632,6 +934,61 @@ async def check_agents(conn, cluster_ids: List[int]) -> Dict[str, Any]:
 CHECK_IDS = ("dns", "port80", "routing", "account", "agents")
 
 
+def _coerce_cluster_ids(raw) -> List[int]:
+    """Coerce a cluster_ids list to ints, dropping non-integer-compatible
+    values. JSONB-stored lists occasionally land as ["1", "2"] (string form)
+    due to legacy paths; asyncpg's `$1::int[]` cast then fails the diagnostic
+    query with InvalidTextRepresentationError. We normalise here so the
+    diagnostic surface is the same regardless of how the order was written.
+    """
+    out: List[int] = []
+    if not raw:
+        return out
+    for v in raw:
+        try:
+            out.append(int(v))
+        except (TypeError, ValueError):
+            continue
+    return out
+
+
+# Bulgu #94 (Round-25 audit) — the entire point of the diagnostic panel
+# is to SHOW the operator what went wrong. Pre-fix, a single check raising
+# an uncaught exception (e.g. an asyncpg cast error from a malformed
+# cluster_ids JSONB, a DNS resolver outage, an SSRF-guard glitch) would
+# propagate up to the router's `try/finally` block, which had no `except`
+# clause, and return HTTP 500 with no body. The operator saw only
+# "Internal Server Error" in DevTools — the inverse of what a diagnostic
+# panel should ever produce. We now wrap every check inside `run_checks`
+# so that a check crash becomes a structured `fail` row instead of
+# bubbling up; the operator gets the exception type + message in the
+# UI and can carry it forward, and the rest of the panel still renders.
+async def _safe_check(check_id: str, label: str, coro):
+    """Run an awaitable that produces a check result; swallow exceptions
+    and convert them to a structured `fail` result so the diagnostic
+    response is never short-circuited by a single broken check."""
+    started = time.time()
+    try:
+        return await coro
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        duration_ms = int((time.time() - started) * 1000)
+        logger.exception(
+            "ACME diagnostic check %s raised", check_id
+        )
+        return _check_result(
+            check_id,
+            label,
+            "fail",
+            f"Diagnostic check crashed: {exc.__class__.__name__}: {exc}",
+            severity="error",
+            details={
+                "exception_type": exc.__class__.__name__,
+                "exception_message": str(exc),
+            },
+            duration_ms=duration_ms,
+        )
+
+
 async def run_checks(
     conn,
     *,
@@ -639,22 +996,53 @@ async def run_checks(
     cluster_ids: List[int],
     account_id: Optional[int],
     only: Optional[List[str]] = None,
+    challenge_type: str = "http-01",
 ) -> List[Dict[str, Any]]:
     """Execute the full pre-flight check suite. `only` lets callers re-run a
     subset (per-check rerun in the UI).
+
+    Every individual check is wrapped in `_safe_check` so the diagnostic
+    endpoint NEVER 500s because of one broken check — the operator gets
+    a structured `fail` row identifying which check crashed and why.
     """
     selected = set(only) if only else set(CHECK_IDS)
     results: List[Dict[str, Any]] = []
 
+    # Normalise inputs once so the per-check error stays in the right
+    # bucket (a malformed cluster_ids should not crash routing/agents).
+    safe_domains = [d for d in (domains or []) if isinstance(d, str) and d]
+    safe_cluster_ids = _coerce_cluster_ids(cluster_ids)
+    try:
+        safe_account_id = int(account_id) if account_id is not None else None
+    except (TypeError, ValueError):
+        safe_account_id = None
+
+    # Issue #35: DNS-01 validates via a TXT record, so the HTTP-01 reachability checks
+    # (public A record, inbound port 80, ACME Challenge Routing) do not apply — report them
+    # as `skipped` rather than failing an internal/isolated host that is actually fine.
+    is_dns01 = (challenge_type == "dns-01")
     if "dns" in selected:
-        results.append(await check_dns(domains))
+        if is_dns01:
+            results.append(_check_result("dns", "DNS resolution", "skipped",
+                                         "DNS-01: a public A record is not required (validation is via a TXT record).",
+                                         severity="info"))
+        else:
+            results.append(await _safe_check("dns", "DNS resolution", check_dns(safe_domains)))
     if "port80" in selected:
-        results.append(await check_port80(domains))
+        if is_dns01:
+            results.append(_check_result("port80", "Port 80 reachability", "skipped",
+                                         "DNS-01: inbound port 80 is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("port80", "Port 80 reachability", check_port80(safe_domains)))
     if "routing" in selected:
-        results.append(await check_routing(conn, domains, cluster_ids))
+        if is_dns01:
+            results.append(_check_result("routing", "HAProxy routing", "skipped",
+                                         "DNS-01: ACME Challenge Routing is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("routing", "HAProxy routing", check_routing(conn, safe_domains, safe_cluster_ids)))
     if "account" in selected:
-        results.append(await check_account(conn, account_id))
+        results.append(await _safe_check("account", "ACME account", check_account(conn, safe_account_id)))
     if "agents" in selected:
-        results.append(await check_agents(conn, cluster_ids))
+        results.append(await _safe_check("agents", "HAProxy agents", check_agents(conn, safe_cluster_ids)))
 
     return results

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -65,26 +69,61 @@ class ACMEService:
             if cached.get('_fetched_at', 0) > time.time() - 3600:
                 return cached
 
-        async with aiohttp.ClientSession() as session:
-            async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
-                if resp.status != 200:
-                    raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
-                data = await resp.json()
-                if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
-                data['_fetched_at'] = time.time()
-                self._directory_cache[directory_url] = data
-                return data
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): directory_url can come from a stored
+        # account row; validate it (https + public IP, no redirects) before the
+        # server-side fetch so it cannot be pointed at internal/metadata targets.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(directory_url)
+
+        # v1.11.0: recorded in request_logs as an outbound call so an operator
+        # can see exactly which CA was contacted and what it answered.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(target=TARGET_ACME, method="GET", url=directory_url) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    if resp.status != 200:
+                        span.set_response(resp.status, dict(resp.headers))
+                        raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
+                    data = await resp.json()
+                    span.set_response(resp.status, dict(resp.headers), data)
+                    if 'Replay-Nonce' in resp.headers:
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
+                    data['_fetched_at'] = time.time()
+                    self._directory_cache[directory_url] = data
+                    return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
-            nonce = self._nonce
-            self._nonce = None
-            return nonce
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         directory = await self.get_directory(directory_url)
-        async with aiohttp.ClientSession() as session:
-            async with session.head(directory['newNonce']) as resp:
-                return resp.headers['Replay-Nonce']
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): newNonce is taken from the (attacker-
+        # influenceable) directory JSON and is fetched here BEFORE the guarded
+        # _signed_request POST, so it must be guarded too — otherwise a directory
+        # that returns an internal newNonce (and omits Replay-Nonce) is a live SSRF.
+        # https + public IP only, IPv4-pinned connector, no redirects, bounded timeout.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        nonce_url = directory['newNonce']
+        await assert_public_url(nonce_url)
+
+        # v1.11.0: a HEAD with no body and no status check — capture the status
+        # and the allowlisted headers only. `Replay-Nonce` itself is redacted by
+        # the header rules: it is a single-use credential.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(
+            target=TARGET_ACME, method="HEAD", url=nonce_url, capture_body=False
+        ) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.head(nonce_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    span.set_response(resp.status, dict(resp.headers))
+                    return resp.headers['Replay-Nonce']
 
     def _generate_account_key(self) -> Tuple[str, dict]:
         private_key = rsa.generate_private_key(
@@ -128,6 +167,20 @@ class ACMEService:
         digest = hashlib.sha256(ordered.encode('utf-8')).digest()
         return _b64url(digest)
 
+    @staticmethod
+    def _dns_txt_value(key_authorization: str) -> str:
+        """RFC 8555 §8.4: the DNS-01 TXT value is base64url(SHA256(key_authorization)) over the
+        RAW 32-byte digest (NOT the hexdigest)."""
+        return _b64url(hashlib.sha256(key_authorization.encode('utf-8')).digest())
+
+    @staticmethod
+    def _challenge_dns_name(identifier: str) -> str:
+        """The `_acme-challenge.<base>` record name for an ACME identifier. A leading wildcard
+        `*.` is stripped, so both `*.example.com` and bare `example.com` map to the SAME name
+        `_acme-challenge.example.com` (which is why apex+wildcard need two coexisting TXT values)."""
+        base = identifier[2:] if identifier.startswith('*.') else identifier
+        return f"_acme-challenge.{base}"
+
     def _sign_jws(self, private_key, protected: dict, payload: Any) -> dict:
         protected_b64 = _b64url(json.dumps(protected).encode('utf-8'))
         if payload == "":
@@ -165,39 +218,82 @@ class ACMEService:
 
         body = self._sign_jws(private_key, protected, payload)
 
-        async with aiohttp.ClientSession() as session:
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): `url` is taken from the CA directory /
+        # order responses. The directory is already fetched from a validated
+        # public CA, but guard the follow-up POST target too (defence in depth)
+        # so a tampered/malicious directory cannot steer the request internally.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(url)
+
+        # v1.11.0: instrument each ATTEMPT separately (the span goes inside the
+        # retry loop, the session stays outside it) so a badNonce retry shows up
+        # as its own row instead of being folded into the successful one.
+        #
+        # capture_body=False is mandatory here. The JWS body is
+        # {protected, payload, signature}: `protected` carries the nonce and the
+        # account kid/jwk, and `signature` is made with the account private key.
+        # The key itself never crosses the wire, but a stored (protected,
+        # signature) pair is a REPLAYABLE ACME credential for the lifetime of the
+        # nonce. We log a description of the request instead of the request.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
             for attempt in range(3):
-                async with session.post(
-                    url,
-                    json=body,
-                    headers={"Content-Type": "application/jose+json"},
-                    timeout=aiohttp.ClientTimeout(total=30),
-                ) as resp:
-                    if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                jws_summary = {
+                    "jws": True,
+                    "acme_url": protected.get("url"),
+                    "kid_present": bool(protected.get("kid")),
+                    "jwk_present": bool(protected.get("jwk")),
+                    "payload_empty": payload == "",
+                    "attempt": attempt + 1,
+                }
+                async with outbound_span(
+                    target=TARGET_ACME,
+                    method="POST",
+                    url=url,
+                    request_body=jws_summary,
+                    capture_body=False,
+                ) as span:
+                    async with session.post(
+                        url,
+                        json=body,
+                        headers={"Content-Type": "application/jose+json"},
+                        timeout=aiohttp.ClientTimeout(total=30),
+                        allow_redirects=False,
+                    ) as resp:
+                        if 'Replay-Nonce' in resp.headers:
+                            self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
-                        err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
-                            nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
-                            protected['nonce'] = nonce
-                            body = self._sign_jws(private_key, protected, payload)
-                            continue
+                        if resp.status == 400 and attempt < 2:
+                            err = await resp.json()
+                            etype = (err.get('type') or '')
+                            edetail = (err.get('detail') or '').lower()
+                            # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                            # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                            # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                            # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                            if etype.endswith('badNonce') or 'nonce' in edetail:
+                                span.set_response(resp.status, dict(resp.headers), err)
+                                nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
+                                protected['nonce'] = nonce
+                                body = self._sign_jws(private_key, protected, payload)
+                                continue
 
-                    resp_data = {}
-                    content_type = resp.headers.get('Content-Type', '')
-                    if 'json' in content_type:
-                        resp_data = await resp.json()
-                    elif resp.status < 300:
-                        text = await resp.text()
-                        if text:
-                            try:
-                                resp_data = json.loads(text)
-                            except json.JSONDecodeError:
-                                resp_data = {"raw": text}
+                        resp_data = {}
+                        content_type = resp.headers.get('Content-Type', '')
+                        if 'json' in content_type:
+                            resp_data = await resp.json()
+                        elif resp.status < 300:
+                            text = await resp.text()
+                            if text:
+                                try:
+                                    resp_data = json.loads(text)
+                                except json.JSONDecodeError:
+                                    resp_data = {"raw": text}
 
-                    headers = dict(resp.headers)
-                    return resp.status, resp_data, headers
+                        headers = dict(resp.headers)
+                        span.set_response(resp.status, headers, resp_data)
+                        return resp.status, resp_data, headers
 
         raise Exception(f"ACME request to {url} failed after retries")
 
@@ -208,6 +304,8 @@ class ACMEService:
         tos_agreed: bool = True,
         eab_kid: Optional[str] = None,
         eab_hmac_key: Optional[str] = None,
+        challenge_type: str = 'http-01',
+        dns_provider: Optional[str] = None,
     ) -> dict:
         directory = await self.get_directory(directory_url)
         pem, jwk = self._generate_account_key()
@@ -250,13 +348,14 @@ class ACMEService:
         conn = await get_database_connection()
         try:
             row = await conn.fetchrow("""
-                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid, challenge_type, dns_provider)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 ON CONFLICT (email, directory_url) DO UPDATE SET
-                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6, updated_at = NOW()
-                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at
+                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6,
+                    challenge_type = $8, dns_provider = $9, updated_at = NOW()
+                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at, challenge_type, dns_provider
             """, email, directory_url, account_url, pem,
-                data.get('status') or 'valid', tos_agreed, eab_kid)
+                data.get('status') or 'valid', tos_agreed, eab_kid, challenge_type, dns_provider)
             return dict(row)
         finally:
             await close_database_connection(conn)
@@ -302,8 +401,10 @@ class ACMEService:
         account_id: int,
         domains: List[str],
         cluster_ids: Optional[List[int]] = None,
+        challenge_type: str = 'http-01',
+        created_by: Optional[int] = None,
     ) -> dict:
-        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}")
+        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}, challenge_type={challenge_type}")
         conn = await get_database_connection()
         try:
             account = await conn.fetchrow(
@@ -341,12 +442,12 @@ class ACMEService:
 
             order_row = await conn.fetchrow("""
                 INSERT INTO letsencrypt_orders
-                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids, challenge_type, created_by)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 RETURNING id
             """, account_id, order_url, data.get('status') or 'pending',
                 json.dumps(domains), data.get('finalize') or '', expires_at,
-                json.dumps(cluster_ids or []))
+                json.dumps(cluster_ids or []), challenge_type, created_by)
 
             order_id = order_row['id']
 
@@ -383,18 +484,24 @@ class ACMEService:
                 domain = (auth_data.get('identifier') or {}).get('value', '')
                 http01_for_domain = False
                 for challenge in (auth_data.get('challenges') or []):
-                    if challenge.get('type') == 'http-01':
+                    # Store only the challenge of the CHOSEN method (default 'http-01' keeps the
+                    # existing behaviour byte-identical; 'dns-01' selects the TXT challenge instead).
+                    if challenge.get('type') == challenge_type:
                         token = challenge['token']
                         jwk = self._get_jwk(private_key)
                         thumbprint = self._jwk_thumbprint(jwk)
                         key_auth = f"{token}.{thumbprint}"
+                        dns_txt = self._dns_txt_value(key_auth) if challenge_type == 'dns-01' else None
 
                         await conn.execute("""
-                            INSERT INTO acme_challenges (order_id, domain, token, key_authorization, challenge_url, status)
-                            VALUES ($1, $2, $3, $4, $5, $6)
+                            INSERT INTO acme_challenges
+                                (order_id, domain, token, key_authorization, challenge_url, status,
+                                 challenge_type, dns_txt_value)
+                            VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
                         """, order_id, domain, token, key_auth,
-                            challenge.get('url') or '', challenge.get('status') or 'pending')
-                        logger.info(f"ACME: Challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
+                            challenge.get('url') or '', challenge.get('status') or 'pending',
+                            challenge_type, dns_txt)
+                        logger.info(f"ACME: {challenge_type} challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
                         http01_for_domain = True
                 if http01_for_domain and domain:
                     domains_with_http01.add(domain)
@@ -413,7 +520,7 @@ class ACMEService:
                     error_payload, order_id
                 )
                 raise Exception(
-                    f"ACME order {order_id} created but no http-01 challenges available "
+                    f"ACME order {order_id} created but no {challenge_type} challenges available "
                     f"(auth fetch failures: {len(auth_fetch_failures)}). See order.error_detail for diagnostics."
                 )
             elif auth_fetch_failures:
@@ -449,10 +556,15 @@ class ACMEService:
         try:
             # Issue #12 / Commit 5a: include 'failed' challenges so they can be retried,
             # but rate-limit per challenge: max 5 attempts in last 5 minutes.
+            # DNS-01 skip-gate (the single safe choke point): never POST a challenge response for a
+            # dns-01 row whose TXT record has not been published yet — that would make the CA validate
+            # against a missing record and burn the order. http-01 rows (challenge_type 'http-01'/NULL)
+            # are never excluded, so the existing flow is byte-identical.
             challenges = await conn.fetch(
                 """SELECT * FROM acme_challenges
                    WHERE order_id = $1
-                     AND (status IN ('pending', 'failed') OR status IS NULL)""",
+                     AND (status IN ('pending', 'failed') OR status IS NULL)
+                     AND NOT (COALESCE(challenge_type, 'http-01') = 'dns-01' AND COALESCE(dns_record_published, FALSE) = FALSE)""",
                 order_id
             )
             order = await conn.fetchrow(

backend/services/csr_service.py

@@ -0,0 +1,461 @@
+"""
+csr_service: CSR (Certificate Signing Request) generation + signed-certificate
+import (v1.9.0).
+
+Flow:
+  1. `generate_csr_bundle` builds a private key + CSR locally (pure crypto,
+     no DB/IO — callers MUST run it via `asyncio.to_thread`: RSA-4096
+     generation takes seconds and would stall the single-worker event loop).
+  2. The bundle is persisted to `ssl_csrs` (`insert_csr_row`); the operator
+     downloads the CSR PEM and has it signed by an external CA.
+  3. `import_signed_certificate` pairs the CA response with the stored key,
+     creates a normal `ssl_certificates` row (source='csr',
+     last_config_status='PENDING' — agents never see it before Apply) and
+     NULLs the key copy on the CSR row.
+
+The CSR builder generalises the in-repo ACME reference
+(services/acme_service.py finalize_order): PEM output instead of DER, full
+subject instead of CN-only, ECDSA support, same PKCS8/NoEncryption key
+serialisation (the agent concatenates cert+key+chain into one PEM and HAProxy
+cannot read passphrase-protected keys).
+
+Private keys are ENCRYPTED AT REST from v1.10.1 (Issue #53): the Fernet token
+replaces the PEM in the same `ssl_csrs.private_key_pem` column, so there is no
+schema change and no SCHEMA_VERSION bump. Rows written earlier hold a raw PEM
+and are still read transparently — see utils/csr_key_crypto.py for the format
+discriminator and the key-rotation caveat. The pending CSR key is the one key
+in the system worth encrypting: it sits idle for the whole signing window and
+is never transmitted, unlike ssl_certificates.private_key_content and the ACME
+order keys, which agents must receive in plaintext on every poll.
+The key is NEVER returned by any CSR API response — `csr_row_to_dict` strips
+it unconditionally.
+"""
+
+import json
+import logging
+from typing import Any, Dict, List, Optional
+from types import SimpleNamespace
+
+import asyncpg
+from fastapi import HTTPException
+
+from cryptography import x509
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import NameOID
+
+from services import ssl_service
+from utils.csr_key_crypto import decrypt_csr_private_key, encrypt_csr_private_key
+
+logger = logging.getLogger(__name__)
+
+
+_KEY_FACTORIES = {
+    'rsa-2048': lambda: rsa.generate_private_key(public_exponent=65537, key_size=2048),
+    'rsa-4096': lambda: rsa.generate_private_key(public_exponent=65537, key_size=4096),
+    'ecdsa-p256': lambda: ec.generate_private_key(ec.SECP256R1()),
+    'ecdsa-p384': lambda: ec.generate_private_key(ec.SECP384R1()),
+}
+
+# (payload attribute, x509 OID, subject-JSON key)
+_SUBJECT_OID_MAP = [
+    ('organization', NameOID.ORGANIZATION_NAME, 'O'),
+    ('organizational_unit', NameOID.ORGANIZATIONAL_UNIT_NAME, 'OU'),
+    ('locality', NameOID.LOCALITY_NAME, 'L'),
+    ('state', NameOID.STATE_OR_PROVINCE_NAME, 'ST'),
+    ('country', NameOID.COUNTRY_NAME, 'C'),
+    ('email', NameOID.EMAIL_ADDRESS, 'emailAddress'),
+]
+
+
+def generate_csr_bundle(payload: Any) -> Dict[str, Any]:
+    """Generate a private key + CSR for a validated SSLCSRCreate payload.
+
+    Pure CPU-bound crypto — no DB, no network. Callers must offload via
+    `asyncio.to_thread` (see module docstring).
+
+    Returns {'csr_pem', 'private_key_pem', 'sans', 'subject'}.
+    """
+    key = _KEY_FACTORIES[payload.key_algorithm]()
+
+    attrs = [x509.NameAttribute(NameOID.COMMON_NAME, payload.common_name)]
+    subject_json: Dict[str, str] = {}
+    for attr_name, oid, json_key in _SUBJECT_OID_MAP:
+        value = getattr(payload, attr_name, None)
+        if value and str(value).strip():
+            cleaned = str(value).strip()
+            attrs.append(x509.NameAttribute(oid, cleaned))
+            subject_json[json_key] = cleaned
+
+    # CN always first in the SAN list, then the extra names, deduped with
+    # order preserved (mirrors the ACME flow where domains[0] is the CN).
+    sans = list(dict.fromkeys([payload.common_name, *(payload.sans or [])]))
+
+    builder = (
+        x509.CertificateSigningRequestBuilder()
+        .subject_name(x509.Name(attrs))
+        .add_extension(
+            x509.SubjectAlternativeName([x509.DNSName(d) for d in sans]),
+            critical=False,
+        )
+    )
+    csr = builder.sign(key, hashes.SHA256())
+
+    return {
+        'csr_pem': csr.public_bytes(serialization.Encoding.PEM).decode('utf-8'),
+        'private_key_pem': key.private_bytes(
+            serialization.Encoding.PEM,
+            serialization.PrivateFormat.PKCS8,
+            serialization.NoEncryption(),
+        ).decode('utf-8'),
+        'sans': sans,
+        'subject': subject_json,
+    }
+
+
+def diff_domains(csr_sans: Optional[List[str]], cert_domains: Optional[List[str]]) -> List[str]:
+    """Human-readable warnings for SAN drift between the CSR and the signed
+    certificate (case-insensitive set diff). CAs legitimately add/normalise
+    SANs, so drift is WARN-only — the hard gate is the key match."""
+    csr_set = {d.lower() for d in (csr_sans or []) if d}
+    cert_set = {d.lower() for d in (cert_domains or []) if d}
+    warnings: List[str] = []
+    added = sorted(cert_set - csr_set)
+    dropped = sorted(csr_set - cert_set)
+    if added:
+        warnings.append(
+            f"The CA added domains that were not in the CSR: {', '.join(added)}"
+        )
+    if dropped:
+        warnings.append(
+            f"The CA dropped domains that were requested in the CSR: {', '.join(dropped)}"
+        )
+    return warnings
+
+
+def _maybe_json_list(value: Any) -> List[str]:
+    """asyncpg returns JSONB columns as str unless a codec is registered."""
+    if isinstance(value, str):
+        try:
+            parsed = json.loads(value)
+            return parsed if isinstance(parsed, list) else []
+        except Exception:
+            return []
+    return list(value) if value else []
+
+
+def csr_row_to_dict(row: Any, include_pem: bool = False) -> Dict[str, Any]:
+    """Row → API dict. ALWAYS strips private_key_pem — the key never leaves
+    the server via a CSR endpoint. csr_pem included only on demand
+    (detail/create responses, not lists)."""
+    d = dict(row)
+    d.pop('private_key_pem', None)
+    if not include_pem:
+        d.pop('csr_pem', None)
+    for key in ('subject', 'sans'):
+        if key in d and isinstance(d[key], str):
+            try:
+                d[key] = json.loads(d[key])
+            except Exception:
+                pass
+    return d
+
+
+async def assert_csr_name_available(conn, name: str) -> None:
+    """Reject a CSR name that is already taken by an ACTIVE certificate or
+    another PENDING CSR. Called BEFORE key generation (cheap fail-fast) and
+    re-run inside `insert_csr_row` (the unique index closes the race)."""
+    existing_cert = await conn.fetchval(
+        "SELECT id FROM ssl_certificates WHERE name = $1 AND is_active = TRUE",
+        name,
+    )
+    if existing_cert:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"An active SSL certificate named '{name}' already exists. "
+                "The CSR name becomes the certificate name at import — choose "
+                "a different name or remove the existing certificate first."
+            ),
+        )
+    existing_csr = await conn.fetchval(
+        "SELECT id FROM ssl_csrs WHERE name = $1 AND status = 'pending'",
+        name,
+    )
+    if existing_csr:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{name}' already exists (id={existing_csr}). "
+                "Import or delete it first, or choose a different name."
+            ),
+        )
+
+
+async def insert_csr_row(conn, payload: Any, bundle: Dict[str, Any], user_id: Optional[int]) -> int:
+    """Persist a freshly generated CSR bundle. Returns the new csr id.
+
+    Issue #53 (v1.10.1): the private key is Fernet-encrypted before it is stored. The token goes
+    into the SAME private_key_pem TEXT column — no schema change — and is only ever decrypted
+    in-process by import_signed_certificate. No CSR endpoint returns the column either way.
+    """
+    await assert_csr_name_available(conn, payload.name)
+    stored_key = encrypt_csr_private_key(bundle['private_key_pem'])
+    try:
+        csr_id = await conn.fetchval(
+            """
+            INSERT INTO ssl_csrs
+            (name, common_name, subject, sans, key_algorithm, csr_pem,
+             private_key_pem, status, created_by)
+            VALUES ($1, $2, $3::jsonb, $4::jsonb, $5, $6, $7, 'pending', $8)
+            RETURNING id
+            """,
+            payload.name,
+            payload.common_name,
+            json.dumps(bundle['subject']),
+            json.dumps(bundle['sans']),
+            payload.key_algorithm,
+            bundle['csr_pem'],
+            stored_key,
+            user_id,
+        )
+    except asyncpg.exceptions.UniqueViolationError:
+        # uq_ssl_csrs_name_pending — a concurrent request won the name.
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{payload.name}' was just created by a "
+                "concurrent request — choose a different name."
+            ),
+        )
+    return csr_id
+
+
+async def import_signed_certificate(conn, csr_id: int, imp: Any, user_id: Optional[int]) -> Dict[str, Any]:
+    """Pair the CA-signed certificate with the stored CSR key and create the
+    ssl_certificates row. Atomic: cert row + CSR state change commit together.
+
+    Returns {'certificate_id', 'certificate_name', 'primary_domain',
+    'warnings', 'reactivated'}. Raises HTTPException on every failure
+    (404 missing, 409 already completed, 400 validation).
+    """
+    async with conn.transaction():
+        # Row lock serialises concurrent imports AND a concurrent DELETE of
+        # the same CSR; works across multiple uvicorn workers (DB-level lock).
+        row = await conn.fetchrow(
+            "SELECT * FROM ssl_csrs WHERE id = $1 FOR UPDATE", csr_id
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        if row['status'] == 'completed':
+            raise HTTPException(
+                status_code=409,
+                detail=(
+                    f"CSR '{row['name']}' is already completed — certificate "
+                    f"id {row['ssl_certificate_id']} was imported from it. "
+                    "Create a new CSR to reissue."
+                ),
+            )
+        if not row['private_key_pem']:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Stored CSR private key is missing — the CSR row is "
+                    "corrupt. Delete it and create a new CSR."
+                ),
+            )
+        # Issue #53: the column holds a Fernet token from v1.10.1 on, and a raw PEM for rows
+        # written before it. decrypt_csr_private_key accepts both, so no data migration is
+        # needed. A None here means the token cannot be decrypted — SECRET_KEY was rotated
+        # without CSR_ENCRYPTION_KEY set. Fail loudly: the key is gone, so the CA's certificate
+        # can never be paired with it, and silently falling through would surface as the far
+        # more confusing "certificate does not match this CSR's private key".
+        stored_key = decrypt_csr_private_key(row['private_key_pem'])
+        if not stored_key:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    f"The stored private key for CSR '{row['name']}' cannot be decrypted. This "
+                    "happens when SECRET_KEY was rotated while CSR_ENCRYPTION_KEY was not set. "
+                    "The key is unrecoverable, so this CSR can no longer be completed — delete "
+                    "it and create a new one (then have the new CSR signed)."
+                ),
+            )
+
+        effective_name = getattr(imp, 'name', None) or row['name']
+
+        # Parse the pasted certificate FIRST so a malformed/truncated CA
+        # response gets the manual flow's 400, not a 500 from the key-match
+        # step below (verify_certificate_key_match reports an unparseable
+        # cert as match=None, which we treat as an integrity failure).
+        from utils.ssl_parser import parse_ssl_certificate, verify_certificate_key_match
+        precheck = parse_ssl_certificate(imp.certificate_content)
+        if precheck.get('error'):
+            raise HTTPException(
+                status_code=400,
+                detail=f"Invalid SSL certificate: {precheck['error']}",
+            )
+
+        # THE defining check of this feature: the CA response must match the
+        # key we generated. Deliberately stricter than create_cert_row's
+        # lenient fallback — we generated this key ourselves, so an
+        # unverifiable pair is an integrity failure, not operator input.
+        match_result = verify_certificate_key_match(imp.certificate_content, stored_key)
+        if match_result.get('match') is False:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    "The signed certificate does not match this CSR's private "
+                    "key — the CA response likely belongs to a different "
+                    "CSR/key. Verify you pasted the certificate that was "
+                    "issued for this exact CSR."
+                ),
+            )
+        if match_result.get('match') is not True:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Could not verify the certificate/key pair: "
+                    f"{match_result.get('reason', 'unknown')}"
+                ),
+            )
+
+        # Full parse/validation pipeline shared with the manual + wizard
+        # flows: invalid PEM, bad chain and already-expired certs all 400.
+        payload = SimpleNamespace(
+            name=effective_name,
+            certificate_content=imp.certificate_content,
+            private_key_content=stored_key,
+            chain_content=getattr(imp, 'chain_content', None),
+            usage_type=getattr(imp, 'usage_type', 'frontend') or 'frontend',
+        )
+        fields = ssl_service._prepare_cert_fields(payload)
+
+        # Global name uniqueness (ssl_certificates.cluster_id is always NULL
+        # under the R38 schema, so name is effectively a global namespace).
+        existing = await conn.fetchrow(
+            "SELECT id, is_active FROM ssl_certificates WHERE name = $1 LIMIT 1",
+            effective_name,
+        )
+        if existing and existing['is_active']:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    f"An active SSL certificate named '{effective_name}' "
+                    "already exists (created after this CSR). Delete or "
+                    "rename it, or pass a different `name` in the import "
+                    "request — the CSR stays pending and can be re-imported."
+                ),
+            )
+
+        reactivated = False
+        if existing and not existing['is_active']:
+            # Reactivate the soft-deleted row (mirrors create_cert_row):
+            # preserves the row id so historical references keep working.
+            await conn.execute(
+                "DELETE FROM ssl_certificate_clusters WHERE ssl_certificate_id = $1",
+                existing['id'],
+            )
+            await conn.execute(
+                """
+                UPDATE ssl_certificates
+                SET is_active = TRUE,
+                    last_config_status = 'PENDING',
+                    certificate_content = $2,
+                    private_key_content = $3,
+                    chain_content = $4,
+                    primary_domain = $5,
+                    all_domains = $6::jsonb,
+                    expiry_date = $7,
+                    usage_type = $8,
+                    issuer = $9,
+                    fingerprint = $10,
+                    status = $11,
+                    days_until_expiry = $12,
+                    source = 'csr',
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = $1
+                """,
+                existing['id'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['primary_domain'],
+                json.dumps(fields['all_domains']),
+                fields['expiry_date'],
+                fields['usage_type'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+            )
+            cert_id = existing['id']
+            reactivated = True
+            logger.info(
+                f"csr_service.import_signed_certificate: reactivated "
+                f"soft-deleted cert '{effective_name}' (id={cert_id}) for CSR {csr_id}"
+            )
+        else:
+            cert_id = await conn.fetchval(
+                """
+                INSERT INTO ssl_certificates (
+                    name, primary_domain, certificate_content, private_key_content,
+                    chain_content, expiry_date, issuer, fingerprint, status,
+                    days_until_expiry, all_domains, is_active, cluster_id,
+                    last_config_status, usage_type, source
+                ) VALUES (
+                    $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb,
+                    TRUE, NULL, 'PENDING', $12, 'csr'
+                )
+                RETURNING id
+                """,
+                effective_name,
+                fields['primary_domain'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['expiry_date'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+                json.dumps(fields['all_domains']),
+                fields['usage_type'],
+            )
+
+        # Cluster bindings: global = zero junction rows (existing convention).
+        if not getattr(imp, 'is_global', False):
+            for cluster_id in (getattr(imp, 'cluster_ids', None) or []):
+                await ssl_service.ensure_cluster_junction(conn, cert_id, cluster_id)
+
+        # Complete the CSR and destroy the key copy — the key now lives on
+        # the certificate row only, like every other key in the system.
+        await conn.execute(
+            """
+            UPDATE ssl_csrs
+            SET status = 'completed',
+                ssl_certificate_id = $2,
+                private_key_pem = NULL,
+                completed_at = CURRENT_TIMESTAMP,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $1
+            """,
+            csr_id,
+            cert_id,
+        )
+
+    warnings = diff_domains(_maybe_json_list(row['sans']), fields['all_domains'])
+    if reactivated:
+        warnings.append(
+            f"A soft-deleted certificate named '{effective_name}' was "
+            f"reactivated (row id {cert_id}) — existing entities that still "
+            "reference that id now serve the newly imported certificate."
+        )
+
+    return {
+        'certificate_id': cert_id,
+        'certificate_name': effective_name,
+        'primary_domain': fields['primary_domain'],
+        'warnings': warnings,
+        'reactivated': reactivated,
+    }

backend/services/dns01_orchestrator.py

@@ -0,0 +1,321 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): non-blocking per-cycle orchestration.
+
+Driven by the existing `complete_pending_acme_orders` background task (which already claims
+in-progress orders with `FOR UPDATE SKIP LOCKED`). For a dns-01 order this module advances AT MOST
+ONE step per 60s cycle — publish TXT, then (after a short min-age) respond — so the serial claim
+loop is never blocked by a multi-minute wait, and NO DNS library is needed (the CA is the source of
+truth; a propagation-lag `invalid` is recovered by a bounded fresh-order chain).
+
+Design invariants (from the hardening review):
+- Additive at the RRset level: publish/cleanup operate on a single (name, value), so wildcard+apex
+  (two values at one name) coexist.
+- No new order-status value: a failed order stays `invalid`; a boolean `dns01_retry_claimed` does the
+  winner-only CAS + claim exclusion, so existing `status` consumers are untouched.
+- Secrets (provider API tokens) are NEVER logged or written to error_detail/events.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta, timezone
+from typing import Dict, List, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+from services.acme_service import acme_service as acme_svc, ACMEService
+from services.dns_providers import get_provider, is_supported, DnsProviderError
+from utils.dns_credentials import decrypt_dns_credentials
+from utils.activity_log import record_event
+
+logger = logging.getLogger(__name__)
+
+# Tunables (kept conservative vs Let's Encrypt rate limits: 5 failed-validations/host/hour,
+# 300 new-orders/account/3h).
+PROPAGATION_GRACE_SECONDS = 25          # min age before we tell the CA to validate
+MANUAL_CONFIRM_TTL = timedelta(hours=48)
+MAX_RETRIES = 3                          # bounded fresh-order chain (1 original + 3 retries = 4 orders)
+# Retry backoff floor (minutes) indexed by the order's current dns01_attempts: [15, 30, 60].
+# The AUTHORITATIVE implementation is the SQL CASE in main.py's claim query
+# (complete_pending_acme_orders), so the backoff is evaluated atomically with the
+# FOR UPDATE SKIP LOCKED claim. Documented here only — do not reintroduce a second copy.
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _aware(dt) -> Optional[datetime]:
+    if dt is None:
+        return None
+    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+
+
+async def _load_credentials(conn, account_id: int) -> Tuple[Optional[Dict[str, str]], bool]:
+    """Return (credentials_dict_or_None, row_exists). credentials None + row_exists True means the
+    stored token could not be decrypted (e.g. SECRET_KEY rotated)."""
+    row = await conn.fetchrow(
+        "SELECT credentials_encrypted FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+        account_id,
+    )
+    if not row:
+        return None, False
+    return decrypt_dns_credentials(row["credentials_encrypted"]), True
+
+
+async def _fail_order(conn, order_id: int, reason: str) -> None:
+    """Mark an order invalid with a sanitized reason (no secrets) + event."""
+    import json
+    payload = json.dumps({"stage": "dns01", "reason": reason, "timestamp": _now().isoformat()})
+    await conn.execute(
+        "UPDATE letsencrypt_orders SET status = 'invalid', error_detail = $1, updated_at = NOW() WHERE id = $2",
+        payload, order_id,
+    )
+    await record_event(order_id, "acme.dns01.validation", severity="ERROR", message=reason, conn=conn)
+
+
+async def advance_dns01_order(order_id: int) -> None:
+    """One non-blocking step for a claimed pending/processing dns-01 order. No-op for http-01 or
+    orders not in a publishable state. Safe to call every cycle (idempotent via CAS flags)."""
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, o.account_id, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if order["status"] not in ("pending", "processing"):
+            return
+
+        challenges = await conn.fetch(
+            "SELECT * FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'", order_id
+        )
+        if not challenges:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+
+        # --- Manual provider: the user publishes + confirms; we only enforce the deadline. ---
+        if provider_name == "manual" or not is_supported(provider_name):
+            deadline = _aware(challenges[0]["manual_confirm_deadline"])
+            if deadline is None:
+                new_deadline = _now() + MANUAL_CONFIRM_TTL
+                await conn.execute(
+                    "UPDATE acme_challenges SET manual_confirm_deadline = $1 WHERE order_id = $2 AND manual_confirm_deadline IS NULL",
+                    new_deadline, order_id,
+                )
+            elif _now() > deadline:
+                await _fail_order(conn, order_id,
+                                  "Manual DNS-01 confirmation deadline passed without confirmation.")
+            return  # respond happens via the dns-confirm endpoint
+
+        # --- Automated provider (e.g. Cloudflare). ---
+        creds, row_exists = await _load_credentials(conn, order["account_id"])
+        if not row_exists:
+            await _fail_order(conn, order_id,
+                              f"No DNS provider credentials configured for provider '{provider_name}'.")
+            return
+        if creds is None:
+            await _fail_order(conn, order_id,
+                              "DNS provider credentials could not be decrypted; re-enter them in Settings.")
+            return
+
+        provider = get_provider(provider_name, creds)
+
+        # Publish any not-yet-published challenge (CAS so two replicas can't double-publish).
+        for ch in challenges:
+            if ch["dns_record_published"]:
+                continue
+            flipped = await conn.fetchval(
+                """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = NOW()
+                   WHERE id = $1 AND dns_record_published = FALSE RETURNING id""",
+                ch["id"],
+            )
+            if not flipped:
+                continue
+            name = ACMEService._challenge_dns_name(ch["domain"])
+            try:
+                await provider.add_txt_record(name, ch["dns_txt_value"])
+                await record_event(order_id, "acme.dns01.publish",
+                                   message=f"Published TXT {name}; waiting for DNS propagation before asking the CA to validate.",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+            except DnsProviderError as exc:
+                # Revert so the next cycle retries the publish; keep the order pending. exc is sanitized.
+                await conn.execute(
+                    "UPDATE acme_challenges SET dns_record_published = FALSE, dns_published_at = NULL WHERE id = $1",
+                    ch["id"],
+                )
+                await record_event(order_id, "acme.dns01.publish", severity="WARNING",
+                                   message=f"Publish failed for {name}: {exc}",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+                return
+
+        # All published? Then respond once the min-age gate has elapsed (across cycles, no sleep).
+        rows = await conn.fetch(
+            "SELECT dns_record_published, dns_published_at FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+            order_id,
+        )
+        if any(not r["dns_record_published"] for r in rows):
+            return
+        published_ats = [_aware(r["dns_published_at"]) for r in rows if r["dns_published_at"]]
+        if not published_ats:
+            return
+        if (_now() - min(published_ats)).total_seconds() < PROPAGATION_GRACE_SECONDS:
+            return  # wait one more cycle
+
+        # Only POST the challenge response if something still needs validating — avoids re-POSTing
+        # every cycle (and bumping dns01_last_attempt_at) once the CA already has them processing.
+        still_pending = await conn.fetchval(
+            """SELECT 1 FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'
+               AND (status IN ('pending', 'failed') OR status IS NULL) LIMIT 1""",
+            order_id,
+        )
+        if not still_pending:
+            return
+
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Told the CA to validate the DNS-01 challenge(s).", conn=conn)
+    finally:
+        await close_database_connection(conn)
+
+
+async def confirm_manual_dns01(order_id: int) -> Dict:
+    """Called by POST /orders/{id}/dns-confirm for the manual provider: mark the TXT published and
+    tell the CA to validate. Returns a small status dict."""
+    conn = await get_database_connection()
+    try:
+        await conn.execute(
+            """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = COALESCE(dns_published_at, NOW())
+               WHERE order_id = $1 AND challenge_type = 'dns-01'""",
+            order_id,
+        )
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Manual DNS-01 confirmed; told the CA to validate.", conn=conn)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+async def retry_invalid_dns01(order_id: int) -> None:
+    """Bounded fresh-order recovery for a dns-01 order that went `invalid` (e.g. propagation lag).
+    Winner-only CAS on `dns01_retry_claimed`; cleans the old TXT, mints a child order. No-op for
+    http-01 or when the budget is exhausted."""
+    conn = await get_database_connection()
+    child_created = False
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.*, a.dns_provider FROM letsencrypt_orders o
+               JOIN letsencrypt_accounts a ON o.account_id = a.id WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if (order["dns01_attempts"] or 0) >= MAX_RETRIES:
+            return  # budget exhausted; stays terminal invalid
+
+        # Winner-only claim (closes the cross-replica double-mint race).
+        claimed = await conn.fetchval(
+            """UPDATE letsencrypt_orders SET dns01_retry_claimed = TRUE, updated_at = NOW()
+               WHERE id = $1 AND status = 'invalid' AND dns01_retry_claimed = FALSE RETURNING id""",
+            order_id,
+        )
+        if not claimed:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+        # Best-effort cleanup of this order's TXT before minting the replacement.
+        if provider_name != "manual" and is_supported(provider_name):
+            creds, _exists = await _load_credentials(conn, order["account_id"])
+            if creds:
+                provider = get_provider(provider_name, creds)
+                chs = await conn.fetch(
+                    "SELECT domain, dns_txt_value FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+                    order_id,
+                )
+                for ch in chs:
+                    try:
+                        await provider.remove_txt_record(ACMEService._challenge_dns_name(ch["domain"]), ch["dns_txt_value"])
+                    except DnsProviderError:
+                        pass  # tolerate; the reconcile sweep will retry
+        await conn.execute(
+            "UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE order_id = $1 AND dns_record_published = TRUE",
+            order_id,
+        )
+
+        import json
+        domains = json.loads(order["domains"]) if isinstance(order["domains"], str) else (order["domains"] or [])
+        cluster_ids = json.loads(order["cluster_ids"]) if isinstance(order["cluster_ids"], str) else (order["cluster_ids"] or [])
+        next_attempts = (order["dns01_attempts"] or 0) + 1
+        child = await acme_svc.create_order(
+            order["account_id"], domains, cluster_ids, challenge_type="dns-01",
+            created_by=order["created_by"],
+        )
+        child_created = True
+        await conn.execute(
+            """UPDATE letsencrypt_orders
+               SET dns01_attempts = $1, dns01_parent_order_id = $2, dns01_last_attempt_at = NOW()
+               WHERE id = $3""",
+            next_attempts, order_id, child["order_id"],
+        )
+        await record_event(order_id, "acme.dns01.validation", severity="WARNING",
+                           message=f"DNS-01 order invalid; minted retry #{next_attempts} (order {child['order_id']}).",
+                           details={"child_order_id": child["order_id"], "attempt": next_attempts}, conn=conn)
+    except Exception as exc:  # noqa: BLE001
+        logger.error(f"[DNS01-RETRY] order {order_id}: {exc}")
+        # A transient failure (e.g. CA rate limit) BEFORE the child was minted must NOT permanently
+        # burn the retry slot — reset the claim so the next cycle can retry. If the child was already
+        # created, leave the claim set (resetting would double-mint).
+        if not child_created:
+            try:
+                await conn.execute(
+                    "UPDATE letsencrypt_orders SET dns01_retry_claimed = FALSE WHERE id = $1 AND status = 'invalid'",
+                    order_id,
+                )
+            except Exception:
+                pass
+    finally:
+        await close_database_connection(conn)
+
+
+async def reconcile_dns01_cleanup() -> None:
+    """Best-effort sweep that removes any TXT records left published for terminal orders (covers a
+    cleanup that failed, or the kill-switch being flipped off mid-flight). NOT gated by the
+    kill-switch. Runs once per completion cycle."""
+    conn = await get_database_connection()
+    try:
+        rows = await conn.fetch(
+            """SELECT c.id AS chal_id, c.order_id, c.domain, c.dns_txt_value, o.account_id, a.dns_provider
+               FROM acme_challenges c
+               JOIN letsencrypt_orders o ON c.order_id = o.id
+               JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE c.challenge_type = 'dns-01'
+                 AND c.dns_record_published = TRUE
+                 AND COALESCE(c.dns_record_cleaned, FALSE) = FALSE
+                 AND o.status IN ('valid', 'invalid', 'cancelled')
+               LIMIT 50""",
+        )
+        for r in rows:
+            provider_name = (r["dns_provider"] or "manual").strip()
+            if provider_name == "manual" or not is_supported(provider_name):
+                # Manual: nothing to call; mark cleaned so we stop revisiting.
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                continue
+            creds, _exists = await _load_credentials(conn, r["account_id"])
+            if creds is None:
+                continue  # can't clean without creds; leave for a later pass
+            provider = get_provider(provider_name, creds)
+            try:
+                await provider.remove_txt_record(ACMEService._challenge_dns_name(r["domain"]), r["dns_txt_value"])
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                await record_event(r["order_id"], "acme.dns01.cleanup",
+                                   message=f"Cleaned up TXT for {r['domain']}", conn=conn)
+            except DnsProviderError:
+                pass  # retry next sweep
+    except Exception as exc:  # noqa: BLE001
+        logger.debug(f"[DNS01-RECONCILE] skipped: {exc}")
+    finally:
+        await close_database_connection(conn)

backend/services/dns_providers/__init__.py

@@ -0,0 +1,14 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): pluggable DNS provider package.
+
+A small adapter layer so DNS-01 challenges can publish/clean up the
+`_acme-challenge.<domain>` TXT record via different DNS providers. The interface is
+additive at the RRset level (add/remove a single value by name+content, never
+overwrite-by-name) so multiple coexisting values at one name (wildcard + apex) work.
+
+Providers: manual (user publishes the TXT themselves), Cloudflare, and GoDaddy (v1.10.0). New
+providers plug in via the registry without touching the orchestration.
+"""
+from .base import DnsProvider, DnsProviderError
+from .registry import get_provider, list_providers, is_supported
+
+__all__ = ["DnsProvider", "DnsProviderError", "get_provider", "list_providers", "is_supported"]

backend/services/dns_providers/base.py

@@ -0,0 +1,53 @@
+"""Abstract DNS provider interface for ACME DNS-01 (Issue #35)."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Dict, List
+
+
+class DnsProviderError(Exception):
+    """A DNS provider failure with a SANITIZED, user-safe message.
+
+    The message must NEVER contain API tokens, request headers, or other secrets — it is
+    persisted to acme_order_events / order error_detail and shown in the UI. Raise this (not a
+    raw aiohttp/json error) so credentials can't leak into logs or the order timeline.
+    """
+
+
+class DnsProvider(ABC):
+    """Base class for a pluggable DNS provider.
+
+    RRset semantics are ADDITIVE: ``add_txt_record`` ensures a (name, value) TXT exists WITHOUT
+    removing other values at the same name, and ``remove_txt_record`` deletes ONLY the record
+    matching (name, value). This is required because a cert for ``example.com`` + ``*.example.com``
+    publishes two distinct values at the SAME name ``_acme-challenge.example.com``.
+    """
+
+    # Stable machine name (used in DB + API); human label; whether the provider automates publishing.
+    name: str = "base"
+    label: str = "Base"
+    automated: bool = True
+
+    # Declarative schema the UI renders to collect credentials. Each field:
+    #   {"key", "label", "type" ("text"|"password"), "required" (bool), "max_length" (int), "help" (str)}
+    credential_fields: List[Dict] = []
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        self.credentials = credentials or {}
+
+    @abstractmethod
+    async def verify_credentials(self) -> Dict:
+        """Validate the stored credentials against the provider. Returns
+        ``{"ok": bool, "detail": str}`` (detail is user-safe). Must not raise on auth failure —
+        return ``ok=False`` with a sanitized detail; may raise DnsProviderError on transport errors.
+        """
+
+    @abstractmethod
+    async def add_txt_record(self, name: str, value: str) -> None:
+        """Ensure a TXT record (name, value) exists. Idempotent; must not remove other values
+        at the same name. Raise DnsProviderError (sanitized) on failure."""
+
+    @abstractmethod
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        """Remove ONLY the TXT record matching (name, value). Tolerate 'already gone'.
+        Raise DnsProviderError (sanitized) on a real failure."""

backend/services/dns_providers/cloudflare.py

@@ -0,0 +1,195 @@
+"""Cloudflare DNS provider for ACME DNS-01 (Issue #35).
+
+Uses the Cloudflare API v4 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name influences which zone is used). Errors are wrapped in DnsProviderError with a
+sanitized message so the API token never reaches logs / order events.
+
+Token scope required: Zone:DNS:Edit + Zone:Read.
+"""
+from __future__ import annotations
+
+import logging
+import re
+from typing import Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+CLOUDFLARE_API_BASE = "https://api.cloudflare.com/client/v4"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# Characters NOT valid in an HTTP bearer credential (RFC 6750 token68: A-Za-z0-9-._~+/=).
+# Cloudflare API tokens are a strict subset of this set, so removing anything outside it can
+# never corrupt a valid token, but it does strip the paste artifacts that make Cloudflare
+# reject the Authorization header with HTTP 400 "Invalid request headers" (CF code 6003):
+# surrounding/embedded quotes, interior spaces/tabs, zero-width/unicode chars, and CR/LF
+# (the latter would otherwise make aiohttp raise client-side before the request is even sent).
+_NON_TOKEN68 = re.compile(r"[^A-Za-z0-9._~+/=-]")
+
+
+def _strip_quotes(s: str) -> str:
+    s = (s or "").strip()
+    if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
+        return s[1:-1]
+    return s
+
+
+def _sanitize_token(s: str) -> str:
+    """Strip surrounding quotes/whitespace, then drop every character outside the token68 set."""
+    return _NON_TOKEN68.sub("", _strip_quotes(s))
+
+
+class CloudflareDNSProvider(DnsProvider):
+    name = "cloudflare"
+    label = "Cloudflare"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_token",
+            "label": "API Token",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": "Scoped API token with Zone:DNS:Edit and Zone:Read permissions.",
+        }
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        self._raw_token = (self.credentials.get("api_token") or "").strip()
+        # Sanitize to the token68 set so a pasted token with quotes/spaces/control/unicode chars
+        # cannot produce an invalid Authorization header (CF 6003 "Invalid request headers").
+        self._token = _sanitize_token(self._raw_token)
+
+    def _headers(self) -> Dict[str, str]:
+        return {"Authorization": f"Bearer {self._token}", "Content-Type": "application/json"}
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> dict:
+        """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
+        DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
+        url = f"{CLOUDFLARE_API_BASE}{path}"
+        # v1.11.0: single funnel for every Cloudflare call, so instrumenting here
+        # covers all five logical endpoints. `safe_error_only=True` records only
+        # the exception TYPE — the same stance the handlers below already take,
+        # because a raw message can carry the request URL and through it the zone
+        # identifier. The Authorization header is dropped to a presence marker by
+        # the header allowlist.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_CLOUDFLARE
+
+        try:
+            async with outbound_span(
+                target=TARGET_DNS_CLOUDFLARE,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        body = await resp.json()
+                    except Exception:  # noqa: BLE001
+                        body = {}
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    if resp.status in (401, 403):
+                        raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                    if resp.status >= 400 or not body.get("success", False):
+                        # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                        # human message text, never the request (which carries the token header).
+                        msgs = "; ".join(
+                            str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                        )
+                        raise DnsProviderError(
+                            f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                        )
+                    return body
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Do NOT include exc verbatim everywhere; aiohttp client errors are URL/transport only
+            # (no token), but keep the message generic and stable.
+            raise DnsProviderError(f"Could not reach the Cloudflare API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected Cloudflare API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._token:
+            return {"ok": False, "detail": "No Cloudflare API token provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                body = await self._request(session, "GET", "/zones?per_page=1")
+            total = ((body.get("result_info") or {}).get("total_count"))
+            detail = "Cloudflare token valid."
+            if isinstance(total, int):
+                detail = f"Cloudflare token valid; {total} zone(s) visible."
+            return {"ok": True, "detail": detail}
+        except DnsProviderError as exc:
+            # Always surface the real Cloudflare reason (e.g. token scope). If sanitizing also changed
+            # the token, append a hint that stray characters were stripped (never echo the token).
+            detail = str(exc)
+            if self._raw_token != self._token:
+                detail += (" Note: the token contained characters that were stripped; if it still "
+                           "fails, re-copy it from Cloudflare without quotes or spaces.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the Cloudflare token."}
+
+    async def _resolve_zone(self, session: aiohttp.ClientSession, record_name: str) -> Tuple[str, str]:
+        """Find the most-specific (longest-suffix) managed zone for a record name.
+        Returns (zone_id, zone_name). Raises DnsProviderError if no zone matches."""
+        labels = record_name.split(".")
+        # Walk suffixes from longest to shortest; a zone needs at least 2 labels.
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break
+            body = await self._request(
+                session, "GET", f"/zones?name={quote(candidate)}&status=active&per_page=50"
+            )
+            results = body.get("result") or []
+            if results:
+                return results[0]["id"], candidate
+        raise DnsProviderError(f"No managed Cloudflare zone found for {record_name}.")
+
+    async def _find_record_id(
+        self, session: aiohttp.ClientSession, zone_id: str, name: str, value: str
+    ) -> Optional[str]:
+        body = await self._request(
+            session, "GET", f"/zones/{zone_id}/dns_records?type=TXT&name={quote(name)}&per_page=100"
+        )
+        for rec in body.get("result") or []:
+            if _strip_quotes(rec.get("content", "")) == value:
+                return rec.get("id")
+        return None
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            zone_id, _zone_name = await self._resolve_zone(session, name)
+            # Idempotent: only create if (name, value) is not already present (preserves coexisting values).
+            existing = await self._find_record_id(session, zone_id, name, value)
+            if existing:
+                return
+            await self._request(
+                session,
+                "POST",
+                f"/zones/{zone_id}/dns_records",
+                json={"type": "TXT", "name": name, "content": value, "ttl": 120},
+            )
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            try:
+                zone_id, _zone_name = await self._resolve_zone(session, name)
+            except DnsProviderError:
+                # Zone gone / not resolvable — nothing we can clean up.
+                return
+            record_id = await self._find_record_id(session, zone_id, name, value)
+            if not record_id:
+                return  # already gone — tolerate
+            await self._request(session, "DELETE", f"/zones/{zone_id}/dns_records/{record_id}")

backend/services/dns_providers/godaddy.py

@@ -0,0 +1,527 @@
+"""GoDaddy DNS provider for ACME DNS-01 (Issue #35 follow-up, v1.10.0).
+
+Uses the GoDaddy Domains API v1 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name selects which zone is touched), which is the same reason cloudflare.py is exempt from
+utils/ssrf_guard.py. Every failure is wrapped in DnsProviderError with a SANITIZED message: the
+API Key and Secret are scrubbed out of any text that could reach a log, an order event, or
+letsencrypt_orders.error_detail.
+
+Two GoDaddy-specific hazards drive the shape of this module — neither exists on Cloudflare:
+
+1. NO PER-VALUE WRITE. `PUT /v1/domains/{d}/records/TXT/{name}` REPLACES the entire RRset at that
+   type+name; it does not merge. A certificate for `example.com` + `*.example.com` publishes two
+   DIFFERENT TXT values at the SAME name `_acme-challenge.example.com` (base.py's additive
+   contract), so a naive single-value PUT would silently destroy the sibling and fail the wildcard
+   authorization. Every mutation here is therefore read-modify-write: GET the current RRset, merge,
+   PUT the whole list back. An EMPTY array is rejected (422 INVALID_BODY, "Records must be
+   specified"), so removing the LAST value must use DELETE — never `PUT []`.
+
+2. ZONE-DESTRUCTIVE SIBLING PATHS. `PUT /v1/domains/{d}/records/TXT` (three segments, no name)
+   wipes EVERY TXT in the zone — SPF, DKIM, DMARC, Microsoft/Google verification — and
+   `PUT /v1/domains/{d}/records` wipes the whole zone (this is dehydrated issue #430 verbatim).
+   The record path is built only by _rrset_path(), which refuses an empty zone or relative name so
+   a URL can never collapse onto one of those endpoints.
+
+Concurrency: v1 has no ETag, no If-Match and no per-record id, so read-modify-write can lose an
+update if two mutations at one name overlap. Today they cannot: orders are advanced sequentially
+(`for oid in claimed_ids: await advance_dns01_order(oid)` in main.py) and an order's challenges are
+published sequentially (`for ch in challenges: await provider.add_txt_record(...)` in
+dns01_orchestrator.py), so the apex+wildcard pair is strictly ordered and the second publish sees
+the first. _rrset_lock() makes that safety structural rather than incidental. Across REPLICAS the
+window is real but narrow (two orders publishing at the same record name in overlapping cycles) and
+self-healing: a lost publish ends `invalid` and the bounded retry chain mints a fresh order, a lost
+cleanup is retried by the reconcile sweep, and an orphaned `_acme-challenge` TXT is inert. The real
+fix is the v3 API (POST + DELETE by recordId, natively per-value), which is PAT-only and a
+follow-up; it is deliberately not used here because v1 + sso-key is what operators can use today.
+
+Credentials: an API Key + Secret pair from https://developer.godaddy.com/keys. It must be a
+PRODUCTION key — the first key the dashboard issues is an OTE (test) key and an OTE credential
+against api.godaddy.com returns 401. A Personal Access Token also works: paste it as the API Key
+and leave the Secret blank, and the Authorization header becomes `Bearer <token>`. That path is not
+cosmetic — GoDaddy marks sso-key "deprecated, supported through 2026" and the current v1 OpenAPI
+advertises only bearer auth, so the PAT is the migration target, not an alternative.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any, Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+GODADDY_API_BASE = "https://api.godaddy.com/v1"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# GoDaddy enforces a 600s (10 min) TTL floor at request time. The published v1 OpenAPI declares no
+# minimum, so a smaller value is not caught by the schema — it fails with
+# 422 {"code":"INVALID_BODY","fields":[{"message":"must have a minimum value of 600", ...}]}.
+# Pin the floor; DNS-01 has no reason to want anything longer.
+_TXT_TTL = 600
+
+# Read-modify-write serialization, keyed by the RRset (record name), not the zone — the RRset is the
+# actual unit of contention, and keying on it avoids serializing unrelated subdomains of one zone.
+# The orchestrator is sequential today (see the module docstring), so this is defence in depth: it
+# is what stops a future `asyncio.gather()` over the publish loop from silently breaking every
+# wildcard+apex certificate. Bounded in practice by the certificate inventory of one process, so
+# there is no eviction; the entries are empty Lock objects.
+_RRSET_LOCKS: Dict[str, asyncio.Lock] = {}
+
+
+def _rrset_lock(record_name: str) -> asyncio.Lock:
+    key = (record_name or "").rstrip(".").lower()
+    lock = _RRSET_LOCKS.get(key)
+    if lock is None:
+        # Safe without a guard: a single event loop never preempts between the get and the assign.
+        lock = _RRSET_LOCKS[key] = asyncio.Lock()
+    return lock
+
+
+def _scrub(text: str, *secrets: str) -> str:
+    """Remove credential substrings from a message before it can reach a log or an order event.
+
+    GoDaddy error bodies do not echo the Authorization header, so this is belt-and-braces — but it
+    makes base.py's "never leak a secret" invariant structural instead of a matter of care. Short
+    strings are skipped so a 1-2 char credential fragment cannot blank out ordinary prose.
+    """
+    out = text or ""
+    for secret in secrets:
+        if secret and len(secret) >= 4:
+            out = out.replace(secret, "***")
+    return out[:300]
+
+
+def _relative_name(fqdn: str, zone: str) -> str:
+    """Convert an absolute record name to the zone-relative form GoDaddy's API requires.
+
+    GoDaddy record names are RELATIVE to the zone with NO trailing dot, and the zone apex is the
+    literal "@" — never an empty string (which would collapse the URL onto the zone-wide TXT
+    endpoint) and never the domain name itself.
+
+      ("_acme-challenge.example.com",         "example.com") -> "_acme-challenge"
+      ("_acme-challenge.foo.bar.example.com", "example.com") -> "_acme-challenge.foo.bar"
+      ("example.com",                         "example.com") -> "@"
+    """
+    f = (fqdn or "").rstrip(".").lower()
+    z = (zone or "").rstrip(".").lower()
+    if z and f == z:
+        return "@"
+    if z and f.endswith("." + z):
+        return f[: -(len(z) + 1)]
+    # Defensive: callers always pass a zone that _resolve_domain derived from this very name.
+    return f or "@"
+
+
+def _rrset_path(zone: str, rel_name: str) -> str:
+    """Build the 4-segment record path `/domains/{zone}/records/TXT/{name}`.
+
+    SAFETY GATE: an empty rel_name would collapse the URL to `/domains/{zone}/records/TXT` — the
+    endpoint that replaces EVERY TXT record in the zone (SPF, DKIM, DMARC, domain verifications).
+    A "." or ".." segment does the same thing one step later: `quote()` leaves both untouched
+    (they are unreserved) and yarl normalizes dot segments away when it builds the URL, so
+    ".../records/TXT/.." would resolve to ".../records" — the whole-zone endpoint. Refuse both
+    rather than build them. `safe=''` percent-encodes the apex "@" as "%40" (accepted bare too,
+    but safer through proxies); "_", "-" and "." are unreserved and pass through unchanged, so a
+    multi-label relative name stays one readable path segment.
+    """
+    if not zone or not rel_name:
+        raise DnsProviderError("Internal error: refusing to build a zone-wide GoDaddy TXT record path.")
+    if rel_name.strip(".") == "" or any(part in (".", "..") for part in rel_name.split("/")):
+        raise DnsProviderError("Internal error: refusing to build a GoDaddy TXT path from a dot segment.")
+    return f"/domains/{quote(zone, safe='')}/records/TXT/{quote(rel_name, safe='')}"
+
+
+def _live_values(records: List[Dict]) -> List[str]:
+    """The non-empty `data` values in an RRset read.
+
+    GoDaddy leaves tombstone rows with `"data": ""` behind at a name after some removals. Echoing
+    one back in a PUT body is rejected with 422 INVALID_BODY, so every field implementation
+    (lego, acme.sh, Posh-ACME) filters them independently — so do we.
+    """
+    out: List[str] = []
+    for rec in records or []:
+        data = (rec or {}).get("data") or ""
+        if data:
+            out.append(data)
+    return out
+
+
+def _merge_add(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that adds `value` while preserving every coexisting sibling value.
+
+    Returns None when `value` is already present — an idempotent no-op, which is where an ACME
+    retry cycle lands.
+    """
+    live = _live_values(existing)
+    if value in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live] + [{"data": value, "ttl": _TXT_TTL}]
+
+
+def _merge_remove(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that removes ONLY `value`, keeping every sibling.
+
+    Three-state result, because GoDaddy needs three different calls:
+      None -> `value` is not there; already gone, tolerate (base.py's remove contract).
+      []   -> it was the last value; the caller must DELETE, since `PUT []` is rejected.
+      list -> PUT this body.
+    """
+    live = _live_values(existing)
+    if value not in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live if d != value]
+
+
+def _require_rrset(body: Any) -> List[Dict]:
+    """The RRset read, or a refusal.
+
+    FAIL CLOSED. A read that did not come back as a JSON array must never be treated as "the RRset
+    is empty" — the very next call is a full-RRset PUT, so coercing an unreadable read to [] would
+    replace every coexisting sibling value with just ours. Failing instead is free: the orchestrator
+    reverts the publish flag and retries next cycle, while a destructive PUT is unrecoverable.
+    """
+    if not isinstance(body, list):
+        raise DnsProviderError(
+            "GoDaddy returned an unreadable TXT record list; refusing to replace the record set."
+        )
+    return body
+
+
+def _error_fields(body: Any) -> Tuple[str, str]:
+    """The whitelisted (code, message) pair from a GoDaddy error body.
+
+    Only these two string fields are ever read; the raw body is never interpolated into a
+    user-facing message.
+    """
+    if not isinstance(body, dict):
+        return "", ""
+    code = body.get("code")
+    message = body.get("message")
+    return (code if isinstance(code, str) else ""), (message if isinstance(message, str) else "")
+
+
+def _retry_after_seconds(headers, body: Any) -> int:
+    """Seconds to wait after a 429.
+
+    The current platform sends `Retry-After` and `ratelimit-reset` headers with no body, while the
+    legacy v1 OpenAPI documents an `ErrorLimit` body carrying `retryAfterSec`. All three shapes are
+    live in the wild — and so is none of them, hence the 60s default.
+    """
+    for key in ("Retry-After", "ratelimit-reset"):
+        raw = (headers or {}).get(key)
+        if raw:
+            try:
+                return max(1, int(str(raw).strip()))
+            except (TypeError, ValueError):
+                pass
+    if isinstance(body, dict):
+        raw = body.get("retryAfterSec")
+        if isinstance(raw, int) and raw > 0:
+            return raw
+    return 60
+
+
+class _GoDaddyHTTPError(DnsProviderError):
+    """A DnsProviderError that also carries the HTTP status and GoDaddy `code`.
+
+    Callers INSIDE this module branch on the status (tolerate a 404 read-back, fall through a
+    zone probe), while everything outside — dns01_orchestrator, letsencrypt.py — still sees a
+    plain sanitized DnsProviderError and needs no change.
+    """
+
+    def __init__(self, message: str, status: int, code: str = ""):
+        super().__init__(message)
+        self.status = status
+        self.code = code
+
+
+class GoDaddyDNSProvider(DnsProvider):
+    name = "godaddy"
+    label = "GoDaddy"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_key",
+            "label": "API Key",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": ("Production API Key from developer.godaddy.com/keys — the first key the dashboard "
+                     "issues is an OTE (test) key and will be rejected. A Personal Access Token also "
+                     "works: paste it here and leave the Secret blank."),
+        },
+        {
+            "key": "api_secret",
+            "label": "API Secret",
+            "type": "password",
+            "required": False,
+            "max_length": 200,
+            "help": ("The Secret half of the same API Key pair. Leave blank ONLY if the field above "
+                     "holds a Personal Access Token. The account also needs at least one registered "
+                     "domain for GoDaddy to allow DNS API access at all."),
+        },
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        # Normalize, never validate: dns01_orchestrator.py calls get_provider() OUTSIDE any
+        # DnsProviderError guard, so a constructor that raised on malformed credentials would escape
+        # as an unhandled exception in the 60s background cycle. The UI drops blank fields before
+        # submitting, so a left-blank field arrives as a MISSING key rather than "" — `.get() or ""`
+        # covers both.
+        self._api_key = (self.credentials.get("api_key") or "").strip()
+        self._api_secret = (self.credentials.get("api_secret") or "").strip()
+        # Per-INSTANCE zone cache. A module-level cache would leak one ACME account's zone visibility
+        # into another's; an instance lives for exactly one orchestrator step, which is precisely the
+        # scope where caching pays off (apex + wildcard resolve the same zone from the same name).
+        self._zone_cache: Dict[str, str] = {}
+
+    def _auth_header(self) -> str:
+        """`sso-key <key>:<secret>` when a Secret is present, else `Bearer <token>` for a PAT.
+
+        Literal prefix, one space, a single colon — no base64, no URL-encoding, no quotes. Keeping
+        this as one swappable string is what makes GoDaddy's sso-key sunset a credential change
+        rather than a code change.
+        """
+        if self._api_secret:
+            return f"sso-key {self._api_key}:{self._api_secret}"
+        return f"Bearer {self._api_key}"
+
+    def _headers(self) -> Dict[str, str]:
+        # Accept is not optional: these endpoints content-negotiate application/xml and
+        # text/javascript. Content-Type is required on every write or GoDaddy answers 400/415.
+        return {
+            "Authorization": self._auth_header(),
+            "Accept": "application/json",
+            "Content-Type": "application/json",
+        }
+
+    def _http_error(self, status: int, code: str, message: str, retry_after: Optional[int]) -> _GoDaddyHTTPError:
+        """Map an HTTP status to a sanitized, operator-actionable DnsProviderError.
+
+        These strings land in acme_order_events and letsencrypt_orders.error_detail and are shown
+        in the order timeline, so each one names what to fix. GoDaddy's own `code`/`message` is
+        appended when present because the two 403 causes — account not eligible for the DNS API vs.
+        a PAT missing `domains.dns:update` — are indistinguishable by status alone. Scrubbing
+        happens HERE, at the single point where provider-supplied text enters a message, so a new
+        caller cannot forget it.
+        """
+        code = _scrub(code, self._api_key, self._api_secret)
+        message = _scrub(message, self._api_key, self._api_secret)
+        if status == 401:
+            detail = ("GoDaddy rejected the API credentials. Check they are a PRODUCTION Key/Secret pair "
+                      "from developer.godaddy.com/keys — the first key the dashboard issues is an OTE "
+                      "(test) key and is not valid here.")
+        elif status == 403:
+            detail = ("GoDaddy denied access to the DNS API. The account needs at least one registered "
+                      "domain, and a Personal Access Token needs the domains.domain:read and "
+                      "domains.dns:update scopes.")
+        elif status == 404:
+            detail = ("GoDaddy has no zone for this domain (check it is registered in this account and "
+                      "uses GoDaddy nameservers).")
+        elif status == 409:
+            detail = "GoDaddy reports this domain is not eligible to have its DNS records changed."
+        elif status == 422:
+            detail = "GoDaddy rejected the record change as invalid (HTTP 422)."
+        elif status == 429:
+            detail = f"GoDaddy rate limit reached; retry in ~{retry_after or 60}s."
+        else:
+            detail = f"GoDaddy API error (HTTP {status})."
+        if code or message:
+            detail += f" (GoDaddy: {code}{': ' + message if message else ''})"
+        return _GoDaddyHTTPError(detail, status=status, code=code)
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> Any:
+        """One GoDaddy API call. Returns the parsed JSON body, or None for the empty-bodied writes.
+
+        Raises a SANITIZED _GoDaddyHTTPError / DnsProviderError — never the credentials, never the
+        request, never a response body verbatim.
+        """
+        url = f"{GODADDY_API_BASE}{path}"
+        # v1.11.0: single funnel for every GoDaddy call. `safe_error_only=True`
+        # keeps the recorded error to the exception TYPE, matching the stance the
+        # handlers below already take — a raw message can carry the request URL.
+        # The `Authorization: sso-key <key>:<secret>` header never reaches the log:
+        # the header allowlist reduces it to a presence marker.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_GODADDY
+
+        try:
+            async with outbound_span(
+                target=TARGET_DNS_GODADDY,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        # content_type=None: every GoDaddy write answers 200/204 with an EMPTY body, and
+                        # aiohttp would otherwise raise on the missing/other content type before parsing.
+                        body = await resp.json(content_type=None)
+                    except ValueError:
+                        # ONLY a decode failure (JSONDecodeError subclasses ValueError) is swallowed —
+                        # an empty write body, or an HTML error page on a >=400. A transport failure
+                        # mid-read (ClientPayloadError, TimeoutError) must NOT land here: it would look
+                        # identical to "empty body", and a caller that reads an RRset would then see
+                        # None and could mistake it for an empty RRset. Those propagate to the handlers
+                        # below and become a real DnsProviderError.
+                        body = None
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    # 2xx only. Redirects are deliberately not followed (aiohttp would forward the
+                    # Authorization header), so a 3xx is a failed call — treating `< 400` as success
+                    # would report a redirected write as a silent no-op.
+                    if 200 <= resp.status < 300:
+                        return body
+                    code, message = _error_fields(body)
+                    retry_after = _retry_after_seconds(resp.headers, body) if resp.status == 429 else None
+                    raise self._http_error(resp.status, code, message, retry_after)
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Only the exception TYPE is interpolated: an aiohttp client error's str() can carry the
+            # request URL, and the message is persisted to the order timeline.
+            raise DnsProviderError(f"Could not reach the GoDaddy API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected GoDaddy API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._api_key:
+            return {"ok": False, "detail": "No GoDaddy API Key provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                # Cheapest read-only check: one request, no zone needed. Deliberately NOT
+                # GET /v1/domains/{domain} — GoDaddy has rejected that details call for small
+                # accounts since 2024-05 while record-level calls keep working, so verifying with it
+                # produces false negatives on accounts where DNS-01 would succeed.
+                body = await self._request(session, "GET", "/domains?limit=1")
+            if not isinstance(body, list):
+                return {"ok": False, "detail": "GoDaddy returned an unexpected response to the credential check."}
+            if not body:
+                # An empty list is NOT a failure: sub-zones delegated to GoDaddy nameservers are
+                # manageable via the records API but never appear in the domain listing.
+                return {"ok": True, "detail": ("GoDaddy credentials valid, but no domains are visible in this "
+                                               "account — the domain you validate must be registered here, or "
+                                               "be a zone delegated to GoDaddy nameservers.")}
+            return {"ok": True, "detail": "GoDaddy credentials valid."}
+        except DnsProviderError as exc:
+            detail = str(exc)
+            if not self._api_secret:
+                # The Bearer path is silent otherwise, and a half-filled form is the likeliest cause.
+                detail += (" Note: no API Secret was entered, so the API Key was sent as a Personal Access "
+                           "Token (Bearer). If you have a Key + Secret pair, enter both halves.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the GoDaddy credentials."}
+
+    async def _resolve_domain(self, session: aiohttp.ClientSession, record_name: str) -> str:
+        """Find the most-specific (longest-suffix) GoDaddy-managed zone for an absolute record name.
+
+        GoDaddy has no `/zones?name=` equivalent, so this walks suffixes longest-to-shortest and
+        probes `GET /v1/domains/{candidate}/records/NS`. That probe (rather than the domain listing
+        or the domain-details call) is deliberate: it finds sub-zones delegated to GoDaddy
+        nameservers, which never appear in `GET /v1/domains` at all, and it does not depend on the
+        details endpoint that small accounts are rejected from.
+        """
+        cached = self._zone_cache.get(record_name)
+        if cached:
+            return cached
+        labels = record_name.rstrip(".").lower().split(".")
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break  # a zone needs at least two labels
+            try:
+                body = await self._request(
+                    session, "GET", f"/domains/{quote(candidate, safe='')}/records/NS"
+                )
+            except _GoDaddyHTTPError as exc:
+                if exc.status in (404, 422):
+                    continue  # not a zone in this account — keep walking
+                # 401/403/409/429/5xx are credential, eligibility or platform failures, not
+                # "wrong zone". Continuing would burn the rate-limit budget re-failing on every
+                # remaining suffix and would bury the real cause under "no managed domain".
+                raise
+            if isinstance(body, list) and body:
+                self._zone_cache[record_name] = candidate
+                return candidate
+        raise DnsProviderError(f"No managed GoDaddy domain found for {record_name}.")
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                zone = await self._resolve_domain(session, name)
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status != 404:
+                        raise
+                    # Some accounts 404 reading back a record set in a zone whose WRITES succeed
+                    # (acme.sh #6517). Reachable only when the NS probe resolved the zone but the
+                    # TXT read 404s — if the NS probe itself 404s we never get here and the caller
+                    # sees "No managed GoDaddy domain found", which is the honest answer. We cannot
+                    # merge what we cannot read, and a single-value PUT would destroy any coexisting
+                    # sibling, so PATCH is the only correct recovery: it is the one genuinely
+                    # ADDITIVE primitive in v1 ("Appends DNS records ... Existing records with the
+                    # same type and name are preserved"). It cannot dedupe, but a duplicate
+                    # identical TXT is harmless for validation and cleanup removes the whole RRset.
+                    await self._request(
+                        session, "PATCH", f"/domains/{quote(zone, safe='')}/records",
+                        json=[{"type": "TXT", "name": _relative_name(name, zone),
+                               "data": value, "ttl": _TXT_TTL}],
+                    )
+                    return
+                body = _merge_add(_require_rrset(existing), value)
+                if body is None:
+                    return  # already published — idempotent, this is where ACME retries land
+                await self._request(session, "PUT", path, json=body)
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                try:
+                    zone = await self._resolve_domain(session, name)
+                except _GoDaddyHTTPError as exc:
+                    # Raise only what a later sweep could plausibly succeed at. reconcile_dns01_cleanup
+                    # swallows the error and leaves dns_record_cleaned FALSE, so the row is re-selected
+                    # every cycle — and its query takes a bare LIMIT 50, so rows that can NEVER succeed
+                    # (revoked key, account lost DNS-API eligibility) would monopolise the whole
+                    # cleanup budget and starve every other account. For those terminal statuses we
+                    # give up quietly: the orphaned `_acme-challenge` TXT is inert, and the same
+                    # credential failure is already loud on the publish path, where it is actionable.
+                    if exc.status == 429 or exc.status >= 500:
+                        raise
+                    return
+                except DnsProviderError:
+                    return  # zone genuinely not resolvable — nothing we could clean up
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status == 404:
+                        return  # RRset (or the read) is gone — tolerate
+                    raise
+                body = _merge_remove(_require_rrset(existing), value)
+                if body is None:
+                    return  # our value is not there — already gone, tolerate
+                if not body:
+                    # The LAST value at this name. `PUT []` is rejected (422 INVALID_BODY, "Records
+                    # must be specified"), so emptying an RRset REQUIRES DELETE. This removes only
+                    # TXT at this exact name; other names and other record types are preserved.
+                    # Do NOT fall back to the "write an empty string to delete" folklore — that hack
+                    # is what creates the tombstone rows _live_values has to filter.
+                    try:
+                        await self._request(session, "DELETE", path)
+                    except _GoDaddyHTTPError as exc:
+                        if exc.status == 404:
+                            return  # raced with another cleanup — tolerate
+                        raise
+                    return
+                await self._request(session, "PUT", path, json=body)

backend/services/dns_providers/manual.py

@@ -0,0 +1,31 @@
+"""Manual DNS provider for ACME DNS-01 (Issue #35).
+
+The user publishes the `_acme-challenge` TXT record in their own DNS (any provider, including
+fully internal/isolated DNS that no API can reach) and then confirms via the UI. There is no API
+to call, so add/remove are no-ops and the orchestration waits for an explicit `dns-confirm`.
+CNAME delegation works implicitly here: the CA follows a CNAME, so a user who delegates
+`_acme-challenge` elsewhere just publishes the value there and confirms.
+"""
+from __future__ import annotations
+
+from typing import Dict, List
+
+from .base import DnsProvider
+
+
+class ManualDNSProvider(DnsProvider):
+    name = "manual"
+    label = "Manual (publish the TXT record yourself)"
+    automated = False
+    credential_fields: List[Dict] = []  # no credentials needed
+
+    async def verify_credentials(self) -> Dict:
+        return {"ok": True, "detail": "Manual mode needs no credentials. You will publish the TXT record yourself."}
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user publishes the record and confirms via the UI.
+        return None
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user may remove the record manually after issuance.
+        return None

backend/services/dns_providers/registry.py

@@ -0,0 +1,46 @@
+"""DNS provider registry for ACME DNS-01 (Issue #35).
+
+Single source of truth mapping a provider name -> class. The API serves the credential-field
+schema from here (so the UI has no hardcoded provider fields) and validates inbound provider
+names against this allow-list. Adding a provider = add it here; nothing else changes.
+"""
+from __future__ import annotations
+
+from typing import Dict, List, Type
+
+from .base import DnsProvider
+from .cloudflare import CloudflareDNSProvider
+from .godaddy import GoDaddyDNSProvider
+from .manual import ManualDNSProvider
+
+_PROVIDERS: Dict[str, Type[DnsProvider]] = {
+    ManualDNSProvider.name: ManualDNSProvider,
+    CloudflareDNSProvider.name: CloudflareDNSProvider,
+    GoDaddyDNSProvider.name: GoDaddyDNSProvider,
+}
+
+
+def is_supported(name: str) -> bool:
+    return name in _PROVIDERS
+
+
+def get_provider(name: str, credentials: Dict[str, str] | None = None) -> DnsProvider:
+    cls = _PROVIDERS.get(name)
+    if cls is None:
+        raise ValueError(f"Unsupported DNS provider: {name}")
+    return cls(credentials or {})
+
+
+def list_providers() -> List[Dict]:
+    """Return the UI-facing provider catalog: name, label, automated flag, and credential schema."""
+    out: List[Dict] = []
+    for name, cls in _PROVIDERS.items():
+        out.append(
+            {
+                "name": cls.name,
+                "label": cls.label,
+                "automated": cls.automated,
+                "credential_fields": cls.credential_fields,
+            }
+        )
+    return out

backend/services/frontend_service.py

@@ -56,14 +56,14 @@ async def create_frontend_row(
             acl_rules, redirect_rules, use_backend_rules,
             request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
             rate_limit, compression, log_separate, monitor_uri,
-            cluster_id, maxconn, updated_at
+            cluster_id, maxconn, log_format, filters, updated_at
         ) VALUES (
             $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12,
             $13, $14, $15, $16, $17, $18, $19,
             $20, $21, $22,
             $23, $24, $25, $26, $27, $28,
             $29, $30, $31, $32,
-            $33, $34, CURRENT_TIMESTAMP
+            $33, $34, $35, $36, CURRENT_TIMESTAMP
         )
         RETURNING id
         """,
@@ -101,6 +101,8 @@ async def create_frontend_row(
         getattr(payload, "monitor_uri", None),
         cluster_id,
         getattr(payload, "maxconn", None),
+        getattr(payload, "log_format", None),  # Issue #38
+        getattr(payload, "filters", None),     # Issue #38
     )
 
     if mark_pending:

backend/services/haproxy_config.py

@@ -6,9 +6,98 @@ import json
 import urllib.parse
 from typing import Optional, List, Dict, Any
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import resolve_acme_backend_target
 
 logger = logging.getLogger(__name__)
 
+# Sentinel prefixes returned by `generate_haproxy_config_for_cluster` INSTEAD of a
+# configuration when generation fails. They are plain strings (not exceptions) for
+# historical reasons: the function's outer `except` swallows everything and returns
+# a one-line comment as the "config".
+#
+# That is a data-loss primitive on its own: an exception anywhere in the generator —
+# e.g. `urllib.parse.urlparse('http://host:99999').port` raising ValueError for an
+# out-of-range port in `acme_backend_url` — collapses a whole cluster's haproxy.cfg
+# into a single comment line, which the apply path then stores as APPLIED and pushes
+# to every agent. Callers that PERSIST the returned text MUST reject it first; use
+# `is_config_generation_error()` rather than matching the string by hand.
+CONFIG_GENERATION_ERROR_PREFIXES = (
+    "# Error generating configuration:",
+    "# Error: Cluster not found",
+)
+
+
+def is_config_generation_error(config_content: Optional[str]) -> bool:
+    """True when `config_content` is a generator failure sentinel, not a configuration.
+
+    Persisting or shipping a sentinel silently destroys a cluster's configuration, so
+    every call site that writes the generator's output to `config_versions` (or hands
+    it to an agent) must guard with this.
+    """
+    if not config_content:
+        return True
+    return config_content.lstrip().startswith(CONFIG_GENERATION_ERROR_PREFIXES)
+
+
+def select_acme_backend_source(candidates: List[tuple]):
+    """Pick the first candidate that resolves into a usable address.
+
+    `candidates` is an ordered list of ``(source_name, url)`` from most to least
+    specific. Returns ``(source_name, url, target, skipped)`` where ``skipped`` lists
+    the ``(source_name, url, target)`` of candidates that were rejected.
+
+    Falling through on UNUSABLE values, not just empty ones, is the point. Values
+    predating validation are common — the settings field was free text — and a
+    scheme-less ``10.90.1.4:8080`` cannot be resolved. Stopping at the first non-empty
+    candidate would emit a backend section with no ``server`` line: ``haproxy -c``
+    still passes because the section exists, Apply succeeds, and every challenge
+    request then 503s from an empty backend with nothing to show for it.
+    """
+    skipped = []
+    for source, url in candidates:
+        if not url:
+            continue
+        target = resolve_acme_backend_target(url)
+        if target.error_code:
+            skipped.append((source, url, target))
+            continue
+        return source, url, target, skipped
+
+    # Nothing usable. Report against the last non-empty candidate so the rendered
+    # comment and the log name a concrete value rather than an empty one.
+    if skipped:
+        source, url, target = skipped[-1]
+        return source, url, target, skipped[:-1]
+    source, url = candidates[0] if candidates else ('none', '')
+    return source, url, resolve_acme_backend_target(url), skipped
+
+
+def extract_acme_backend_target(config_content: Optional[str]) -> Optional[str]:
+    """Return the `server _acme_mgmt` argument string from a rendered config.
+
+    e.g. ``"10.90.1.4:80"`` or ``"mgmt.internal:443 ssl verify none"``; ``None`` when
+    the cluster renders no ACME challenge backend at all.
+
+    Used to decide whether an ACME-related edit actually CHANGES the shipped
+    configuration. Comparing whole config texts would report a difference on every
+    unrelated pending edit; comparing this one line answers the only question that
+    matters here — "would the HAProxy nodes start talking to a different address?"
+    """
+    if not config_content:
+        return None
+    in_section = False
+    for line in config_content.splitlines():
+        stripped = line.strip()
+        if stripped.startswith("backend "):
+            in_section = stripped == "backend _acme_challenge_backend"
+            continue
+        if stripped.startswith(("frontend ", "listen ", "defaults", "global")):
+            in_section = False
+            continue
+        if in_section and stripped.startswith("server _acme_mgmt "):
+            return stripped[len("server _acme_mgmt "):].strip()
+    return None
+
 
 def _format_redirect_rule(rule: Any) -> Optional[str]:
     """Render a single redirect rule into a HAProxy `redirect ...` line.
@@ -393,6 +482,12 @@ def _categorize_haproxy_directive(line: str) -> str:
         return "prelude"
     if s.startswith("acl "):
         return "acl"
+    # Issue #38: SPOE (and other) `filter` directives must be declared BEFORE
+    # the `http-request send-spoe-group` rules that use them, otherwise HAProxy
+    # fails with "unable to find SPOE engine". Own bucket, flushed right after
+    # `prelude` and before tcp_req/acl/http_req (see flush order below).
+    if s.startswith("filter "):
+        return "filter"
     if s.startswith("stick-table") or s.startswith("stick "):
         return "stick"
     if s.startswith("tcp-request"):
@@ -414,6 +509,7 @@ def _categorize_haproxy_directive(line: str) -> str:
         or s.startswith("compression ")
         or s.startswith("monitor-uri")
         or s.startswith("log ")
+        or s.startswith("log-format")  # Issue #38: log-format / log-format-sd
         or s.startswith("description ")
         or s.startswith("disabled")
         or s.startswith("enabled")
@@ -871,7 +967,15 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                         f"    bind {frontend['bind_address']}:{frontend['bind_port']}"
                     )
             
-            config_lines.append(f"    mode {frontend['mode']}")
+            # `frontends.mode` is `VARCHAR(10) DEFAULT 'http'` but NULLABLE, and rows
+            # can arrive with it unset via agent sync or config import. Interpolating
+            # the raw value then emits a literal `mode None`, which HAProxy rejects —
+            # taking down the whole cluster config, not just this frontend. Normalise
+            # once here and use the result everywhere below, so the ACME gate, the
+            # backend-mode check and the rendered line can never disagree with each
+            # other about what mode this frontend is in.
+            frontend_mode = (frontend.get('mode') or 'http').strip().lower()
+            config_lines.append(f"    mode {frontend_mode}")
 
             # ─────────────────────────────────────────────────────────────────
             # R2.3 / R3.3 (PR-1 hotfix): emit ordering buckets.
@@ -903,11 +1007,18 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # "stick-table already declared").
             # ─────────────────────────────────────────────────────────────────
             _fe_buckets: Dict[str, List[str]] = {
-                "prelude": [], "stick": [], "tcp_req": [],
+                "prelude": [], "filter": [], "stick": [], "tcp_req": [],
                 "acl": [], "http_req": [], "http_resp": [],
                 "redirect": [], "use_be": [], "default_be": [],
             }
             _stick_table_emitted = False
+            # Whether this frontend emits ANY stick-counter usage (`track-sc<N>` or an
+            # `sc_*_rate(...)` fetch). If it does but no `stick-table` is declared, HAProxy
+            # fatally rejects the WHOLE cluster config with "table '<frontend>' used but not
+            # configured". This happens with rate-limit directives baked into a frontend's
+            # stored fields (request_headers/options) by an older version or a config import.
+            # We track it here and inject a default stick-table before flushing if needed.
+            _sc_counter_used = False
             # Phase K Phase D follow-up (Bulgu #13) — same dedup
             # contract for `http-request track-sc<N> <fetch>` lines.
             # HAProxy only NEEDS one tracking call per
@@ -931,9 +1042,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 correct frontend-block bucket. Idempotent for stick-table
                 lines (R3.3 dedup) AND http-request track-sc<N> lines
                 (Bulgu #13 dedup)."""
-                nonlocal _stick_table_emitted
+                nonlocal _stick_table_emitted, _sc_counter_used
                 cat = _categorize_haproxy_directive(line)
                 stripped = line.strip()
+                # Any stick-counter usage (track-sc<N> write, or an sc_*_rate(...) fetch like
+                # sc_http_req_rate(0)) requires a stick-table in this frontend.
+                if "track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped):
+                    _sc_counter_used = True
                 if cat == "stick":
                     if _stick_table_emitted and stripped.startswith("stick-table"):
                         logger.debug(
@@ -972,7 +1087,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 _fe_buckets[cat].append(line)
 
             # ACME HTTP-01 Challenge routing (auto-managed)
-            if frontend['mode'] == 'http' and cluster_info.get('acme_enabled', False):
+            if frontend_mode == 'http' and cluster_info.get('acme_enabled', False):
                 _emit_fe("    acl is_acme_challenge path_beg /.well-known/acme-challenge/")
                 _emit_fe("    http-request allow if is_acme_challenge")
                 _emit_fe("    use_backend _acme_challenge_backend if is_acme_challenge")
@@ -985,20 +1100,31 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
                         _emit_fe(f"    {line_stripped}")
 
+            # Issue #38: emit frontend log-format and SPOE (etc.) filter directives.
+            # `log_format` routes to the `prelude` bucket, `filters` to the `filter`
+            # bucket (both via _emit_fe → _categorize_haproxy_directive), guaranteeing
+            # `filter ...` is rendered before the `http-request send-spoe-group` rules.
+            for _fld in ('log_format', 'filters'):
+                if frontend.get(_fld):
+                    for line in frontend[_fld].split('\n'):
+                        line_stripped = line.strip()
+                        if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
+                            _emit_fe(f"    {line_stripped}")
+
             # CRITICAL: Validate frontend-backend mode compatibility
             if frontend.get('default_backend'):
                 default_backend_name = frontend['default_backend'].strip() if frontend['default_backend'] else ''
                 if default_backend_name and default_backend_name not in ('[]', '{}', 'null', 'None'):
                     backend_mode = backend_modes.get(default_backend_name)
 
-                    if backend_mode and backend_mode != frontend['mode']:
-                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend['mode']}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
+                    if backend_mode and backend_mode != frontend_mode:
+                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend_mode}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
                         # FIX-10 marker: 'BACKEND-MODE-WARNING' keyword in
                         # the comment body routes it to the 'default_be'
                         # bucket via _categorize_haproxy_directive, so the
                         # warning emits next to the actual default_backend
                         # directive instead of at the top of the block.
-                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend['mode']}'")
+                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend_mode}'")
                         _emit_fe(f"    # BACKEND-MODE-WARNING: HAProxy will reject this configuration! Please fix the mode mismatch in UI.")
 
                     _emit_fe(f"    default_backend {default_backend_name}")
@@ -1188,6 +1314,22 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 else:
                     logger.warning(f"Config Generation: No config lines generated for WAF rule '{waf_rule['name']}' (ID: {waf_rule['id']}, Type: {waf_rule['rule_type']})")
 
+            # Robustness fix: if this frontend uses a stick counter (track-sc<N> or an
+            # sc_*_rate(...) fetch) but declared NO stick-table, inject a default one so HAProxy
+            # doesn't fatally reject the whole cluster config with "table '<frontend>' used but
+            # not configured". This rescues rate-limit directives baked into a frontend's stored
+            # request_headers/options by an older version or import. Purely additive — it only
+            # fires when a counter is used AND no table exists (a config that is invalid today),
+            # so it never changes a frontend that already has a stick-table or doesn't rate-limit.
+            if _sc_counter_used and not _stick_table_emitted:
+                _fe_buckets["stick"].insert(
+                    0, "    stick-table type ip size 100k expire 30s store http_req_rate(10s)")
+                _stick_table_emitted = True
+                logger.info(
+                    f"STICK-TABLE AUTO-INJECT: frontend '{frontend['name']}' uses a stick "
+                    f"counter (track-sc/sc_*_rate) but declared no stick-table; injected a "
+                    f"default so the config stays valid.")
+
             # ─────────────────────────────────────────────────────────────
             # Flush the per-frontend buckets in canonical HAProxy order.
             # The order below is the single source of truth for emit
@@ -1196,6 +1338,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # ─────────────────────────────────────────────────────────────
             for _bucket_key in (
                 "prelude",
+                "filter",
                 "stick",
                 "tcp_req",
                 "acl",
@@ -1543,36 +1686,82 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     f"frontend found (would create orphan backend section)."
                 )
             else:
-                acme_url = cluster_info.get('acme_backend_url') or ''
-                if not acme_url:
-                    try:
-                        acme_settings = await db_conn.fetchrow(
-                            "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
-                        )
-                        if acme_settings and acme_settings['value']:
-                            val = acme_settings['value']
-                            if isinstance(val, str):
-                                try:
-                                    val = json.loads(val)
-                                except (json.JSONDecodeError, TypeError):
-                                    pass
-                            if val:
-                                acme_url = str(val)
-                    except Exception:
-                        pass
-                if not acme_url:
-                    from config import MANAGEMENT_BASE_URL
-                    acme_url = MANAGEMENT_BASE_URL
+                # Track WHERE the effective URL came from. Support has no way today to
+                # tell an operator-set value from the shipped `localhost` default, and
+                # this whole block emits no log line at all (contrast the skip branches
+                # above), so a wrong challenge backend is invisible until Let's Encrypt
+                # fails. `acme_source` is logged with the rendered host:port below.
+                acme_candidates = [
+                    ('cluster.acme_backend_url', cluster_info.get('acme_backend_url') or '')
+                ]
+                _settings_url = ''
+                try:
+                    acme_settings = await db_conn.fetchrow(
+                        "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
+                    )
+                    if acme_settings and acme_settings['value']:
+                        val = acme_settings['value']
+                        if isinstance(val, str):
+                            try:
+                                val = json.loads(val)
+                            except (json.JSONDecodeError, TypeError):
+                                pass
+                        if val:
+                            _settings_url = str(val)
+                except Exception:
+                    pass
+                acme_candidates.append(
+                    ('system_settings.acme.challenge_backend_url', _settings_url)
+                )
+                from config import MANAGEMENT_BASE_URL
+                acme_candidates.append(('config.MANAGEMENT_BASE_URL', MANAGEMENT_BASE_URL))
 
-                parsed = urllib.parse.urlparse(acme_url)
-                host = parsed.hostname or 'localhost'
-                port = parsed.port or (443 if parsed.scheme == 'https' else 8080)
-                ssl_flag = ' ssl verify none' if parsed.scheme == 'https' else ''
+                acme_source, acme_url, target, _skipped = select_acme_backend_source(
+                    acme_candidates
+                )
+                for _s_source, _s_url, _s_target in _skipped:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} skipping unusable value from "
+                        f"{_s_source} (reason={_s_target.error_code}): "
+                        f"{_s_target.error_message} value={_s_url!r}"
+                    )
 
                 config_lines.append("# ACME Challenge Backend (auto-managed by HAProxy OpenManager)")
                 config_lines.append("backend _acme_challenge_backend")
                 config_lines.append("    mode http")
-                config_lines.append(f"    server _acme_mgmt {host}:{port}{ssl_flag}")
+                if target.error_code:
+                    # The value cannot produce an address. Emit the section without a
+                    # `server` line rather than guessing: every HTTP frontend already
+                    # carries `use_backend _acme_challenge_backend`, and a use_backend
+                    # with no matching backend is fatal to `haproxy -c`. The operator's
+                    # raw value is deliberately NOT echoed into the file — a value
+                    # containing a newline would inject directives into a config pushed
+                    # to every node. It goes to the log instead.
+                    config_lines.append(
+                        f"    # ACME challenge backend unavailable ({target.error_code}) — "
+                        f"see Cluster Management > ACME Challenge Backend URL"
+                    )
+                    logger.error(
+                        f"ACME-BACKEND: cluster {cluster_id} has an unusable challenge backend "
+                        f"URL (source={acme_source}, reason={target.error_code}): "
+                        f"{target.error_message} value={acme_url!r}"
+                    )
+                else:
+                    config_lines.append(
+                        f"    server _acme_mgmt {target.host}:{target.port}{target.ssl_flag}"
+                    )
+                    # One greppable line per render. `ACME-BACKEND` is the support
+                    # keyword: it answers "what address did we actually ship, and who
+                    # chose it?" without shell access to the node.
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} challenge backend rendered as "
+                        f"{target.host}:{target.port}{target.ssl_flag} "
+                        f"(source={acme_source}, url={acme_url!r})"
+                    )
+                for _warning in target.warnings:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} (source={acme_source}): {_warning}"
+                    )
                 config_lines.append("")
 
         # Only close the connection if it was created within this function

backend/services/keepalived_config.py

@@ -0,0 +1,192 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Standalone, DB-free renderer for a node's /etc/keepalived/keepalived.conf and the
+HAProxy health-check script, plus Fernet at-rest encryption for the VRRP secret.
+
+The router fetches DB rows and calls these pure functions; nothing here touches the
+database or logs secrets. Trivially unit-testable (see tests/test_keepalived_config.py).
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import re
+from typing import List, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# Written into every file we manage so the agent can tell "ours" from a
+# hand-maintained keepalived setup (ownership guard, B-3/T-2). Must match the
+# string the agent greps for in linux_install.sh.
+OWNERSHIP_MARKER = "# Managed by HAProxy OpenManager"
+CHECK_SCRIPT_PATH = "/etc/keepalived/check_haproxy.sh"
+
+
+# ---------------------------------------------------------------------------
+# VRRP secret at rest (mirrors backend/services/mfa_service.py)
+# ---------------------------------------------------------------------------
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit VIP_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so the secret survives restarts, like MFA)."""
+    explicit = os.getenv("VIP_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("VIP_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"vip-vrrp-secret-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_vrrp_secret(secret_plain: str) -> str:
+    return _get_fernet().encrypt(secret_plain.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_vrrp_secret(secret_encrypted: str) -> Optional[str]:
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt VRRP secret (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting VRRP secret: %s", exc)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Renderers
+# ---------------------------------------------------------------------------
+def build_haproxy_check_script(*, bin_path: Optional[str] = None,
+                               config_path: Optional[str] = None) -> str:
+    """Render the health-check the agent writes to CHECK_SCRIPT_PATH.
+
+    Derives the process name from the HAProxy binary basename (B-4) rather than a
+    blind hardcoded 'haproxy'. Returns non-zero when HAProxy isn't running so the
+    VRRP track_script lowers this node's priority and the VIP fails over.
+    """
+    proc = "haproxy"
+    if bin_path:
+        base = os.path.basename(bin_path.strip())
+        if re.match(r'^[A-Za-z0-9._-]{1,64}$', base):
+            proc = base
+    return (
+        "#!/bin/sh\n"
+        f"{OWNERSHIP_MARKER} — DO NOT EDIT\n"
+        "# Exits 0 while HAProxy is up; non-zero triggers VRRP failover.\n"
+        f"pidof {proc} >/dev/null 2>&1 || exit 1\n"
+        "exit 0\n"
+    )
+
+
+def _vrrp_instance_name(vip_id: int) -> str:
+    return f"VI_{int(vip_id)}"
+
+
+def _failover_weight(members: List[dict]) -> int:
+    """Negative weight so a failed MASTER drops strictly below every healthy BACKUP
+    (B-6). master_priority + weight < min(backup_priority)."""
+    master = next((m for m in members if str(m.get("role", "")).upper() == "MASTER"), None)
+    backups = [int(m["priority"]) for m in members if str(m.get("role", "")).upper() != "MASTER"]
+    if not master or not backups:
+        return -20
+    return -((int(master["priority"]) - min(backups)) + 1)
+
+
+def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
+                           peer_ips: List[str], auth_pass_plain: Optional[str]) -> str:
+    """Render one node's keepalived.conf from the VIP + member rows.
+
+    `vip` keys: id, name, virtual_ip, prefix_length, virtual_router_id, advert_int,
+                use_unicast, track_haproxy.
+    `this_agent` keys: role, priority, network_interface, ip_address (str).
+    `peer_ips`: the OTHER members' ip_address strings (already str()'d by the caller).
+    Caller must never log the returned string (it may contain auth_pass).
+    """
+    role = str(this_agent["role"]).upper()
+    iface = this_agent["network_interface"]
+    prio = int(this_agent["priority"])
+    track = bool(vip.get("track_haproxy", True))
+    use_unicast = bool(vip.get("use_unicast", True))
+    vrid = int(vip["virtual_router_id"])
+    advert = int(vip.get("advert_int", 1))
+    name = str(vip.get("name", ""))
+    inst = _vrrp_instance_name(vip["id"])
+
+    lines: List[str] = []
+    lines.append(f"{OWNERSHIP_MARKER} — DO NOT EDIT")
+    lines.append(f'# VIP "{name}" (id={vip["id"]}) — role {role}')
+    lines.append("global_defs {")
+    lines.append("    enable_script_security")
+    lines.append("    script_user root")
+    lines.append("}")
+    lines.append("")
+
+    if track:
+        weight = _failover_weight(members)
+        lines.append("vrrp_script chk_haproxy {")
+        lines.append(f'    script "{CHECK_SCRIPT_PATH}"')
+        lines.append("    interval 2")
+        lines.append("    fall 2")
+        lines.append("    rise 2")
+        lines.append(f"    weight {weight}")
+        lines.append("}")
+        lines.append("")
+
+    lines.append(f"vrrp_instance {inst} {{")
+    lines.append(f"    state {role}")
+    lines.append(f"    interface {iface}")
+    lines.append(f"    virtual_router_id {vrid}")
+    lines.append(f"    priority {prio}")
+    lines.append(f"    advert_int {advert}")
+    if auth_pass_plain:
+        lines.append("    authentication {")
+        lines.append("        auth_type PASS")
+        lines.append(f"        auth_pass {auth_pass_plain}")
+        lines.append("    }")
+    # Unicast only makes sense with at least one peer. For a single-node VIP (no peers)
+    # we deliberately omit the unicast block: keepalived treats a bare `unicast_src_ip`
+    # with no `unicast_peer` as deprecated, warns, and silently falls back to multicast —
+    # and `keepalived -t` flags it. Omitting it yields a clean multicast config that holds
+    # the VIP with no peer to talk to. Multi-node behaviour (peers present) is unchanged.
+    if use_unicast and peer_ips:
+        src = this_agent.get("ip_address")
+        if src:
+            lines.append(f"    unicast_src_ip {src}")
+        lines.append("    unicast_peer {")
+        for p in peer_ips:
+            lines.append(f"        {p}")
+        lines.append("    }")
+    lines.append("    virtual_ipaddress {")
+    lines.append(f'        {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
+    lines.append("    }")
+    if track:
+        lines.append("    track_script {")
+        lines.append("        chk_haproxy")
+        lines.append("    }")
+    lines.append("}")
+    return "\n".join(lines) + "\n"

backend/services/keepalived_parser.py

@@ -0,0 +1,533 @@
+"""Issue #27 follow-up — parse an EXISTING keepalived.conf so a hand-maintained VIP can be
+adopted into OpenManager's model (v1.10.4).
+
+Standalone and DB-free, like keepalived_config.py: the agent reports the file it found on a
+node, this module turns it into the fields `vip_instances` / `vip_members` need, and the
+adoption endpoint decides whether taking ownership is safe.
+
+WHY A PARSER AND NOT THE HEARTBEAT. The heartbeat carries two keepalived facts —
+`keepalive_state` (MASTER/BACKUP, best-effort from logs) and `keepalive_ip` (the first
+address grepped out of `virtual_ipaddress`). Rendering a node's config needs eleven:
+virtual_router_id, auth_pass, interface, priority, prefix_length, advert_int, unicast
+peers, track_haproxy, role and the address itself. Guessing the missing ones is not a
+cosmetic risk — a wrong VRID puts the nodes in two separate VRRP domains and a wrong
+auth_pass makes them reject each other, and either way both nodes claim the VIP.
+
+THE SAFETY CONTRACT. Adoption REPLACES the operator's file with our render, so anything in
+their file that `render_keepalived_conf` cannot reproduce would be silently destroyed on
+takeover — a `notify_master` failover hook, an LVS `virtual_server` section, a second
+address in one instance, a sync group. Extracting the fields is the easy half; the half
+that matters is `unsupported`, the list of directives we would drop. The caller must treat
+a non-empty `unsupported` as a refusal to adopt, not a warning to log.
+
+Secrets: a parsed instance carries `auth_pass` in cleartext because that is the only way to
+re-render an identical config. NEVER log a parse result. Callers persist it through
+`encrypt_vrrp_secret` and mask it in anything UI-facing, exactly as the VIP router already
+does for `auth_pass` in version diffs.
+"""
+from __future__ import annotations
+
+import ipaddress
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+# Directives `render_keepalived_conf` emits, and therefore the only ones a takeover can
+# reproduce. Anything else found in a vrrp_instance is reported in `unsupported`.
+_SUPPORTED_INSTANCE_KEYS = {
+    "state", "interface", "virtual_router_id", "priority", "advert_int",
+    "authentication", "unicast_src_ip", "unicast_peer", "virtual_ipaddress", "track_script",
+}
+# Top-level blocks we can account for. `vrrp_script` is reproduced only when it is the
+# check script we generate ourselves (see _classify_script).
+_SUPPORTED_TOP_KEYS = {"global_defs", "vrrp_script", "vrrp_instance"}
+
+# global_defs entries our render emits. An operator's file usually carries more (notification
+# email, router_id, ...) and losing those is a real change, so they are reported too.
+_SUPPORTED_GLOBAL_KEYS = {"enable_script_security", "script_user"}
+
+_IDENT_RE = re.compile(r"^[A-Za-z0-9._:-]+$")
+
+
+class KeepalivedParseError(ValueError):
+    """The text is not a keepalived.conf we can reason about (unbalanced braces etc.)."""
+
+
+# ---------------------------------------------------------------------------
+# Tokenizer / block reader
+# ---------------------------------------------------------------------------
+def _strip_comment(line: str) -> str:
+    """Drop a trailing comment. keepalived treats BOTH `#` and `!` as comment starters, and
+    neither is meaningful inside the quoted script paths we care about, so a quote-aware
+    scan is enough (a `#` inside quotes stays)."""
+    out: List[str] = []
+    quote: Optional[str] = None
+    for ch in line:
+        if quote:
+            out.append(ch)
+            if ch == quote:
+                quote = None
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            out.append(ch)
+            continue
+        if ch in ("#", "!"):
+            break
+        out.append(ch)
+    return "".join(out)
+
+
+def _split_tokens(line: str) -> List[str]:
+    """Whitespace split that keeps quoted strings whole and isolates braces, so
+    `virtual_ipaddress { 10.0.0.1/24 dev eth0 }` tokenizes the same as its multi-line form."""
+    tokens: List[str] = []
+    buf: List[str] = []
+    quote: Optional[str] = None
+
+    def flush() -> None:
+        if buf:
+            tokens.append("".join(buf))
+            buf.clear()
+
+    for ch in line:
+        if quote:
+            if ch == quote:
+                quote = None
+            else:
+                buf.append(ch)
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            continue
+        if ch.isspace():
+            flush()
+        elif ch in ("{", "}"):
+            flush()
+            tokens.append(ch)
+        else:
+            buf.append(ch)
+    flush()
+    return tokens
+
+
+def _read_blocks(text: str) -> List[Dict[str, Any]]:
+    """Parse the file into nested entries.
+
+    Each entry is either
+      {"kind": "block", "name": str, "args": [str], "body": [entries], "line": int}
+      {"kind": "line",  "tokens": [str], "line": int}
+
+    Line boundaries matter: inside `virtual_ipaddress` and `unicast_peer` each line is one
+    bare value, so a flat token stream could not tell two addresses apart.
+    """
+    root: List[Dict[str, Any]] = []
+    stack: List[List[Dict[str, Any]]] = [root]
+    # Blocks whose opening `{` we have seen, so a stray `}` can be reported with context.
+    open_blocks: List[str] = []
+
+    for lineno, raw in enumerate(text.splitlines(), start=1):
+        pending: List[str] = []
+        for tok in _split_tokens(_strip_comment(raw)):
+            if tok == "{":
+                name = pending[0] if pending else ""
+                args = pending[1:]
+                block = {"kind": "block", "name": name, "args": args, "body": [], "line": lineno}
+                stack[-1].append(block)
+                stack.append(block["body"])
+                open_blocks.append(name)
+                pending = []
+            elif tok == "}":
+                if pending:
+                    stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+                    pending = []
+                if len(stack) == 1:
+                    raise KeepalivedParseError(f"unbalanced '}}' on line {lineno}")
+                stack.pop()
+                open_blocks.pop()
+            else:
+                pending.append(tok)
+        if pending:
+            stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+
+    if len(stack) != 1:
+        raise KeepalivedParseError(f"unclosed block '{open_blocks[-1] or '?'}' at end of file")
+    return root
+
+
+# ---------------------------------------------------------------------------
+# Interpretation
+# ---------------------------------------------------------------------------
+def _as_int(tokens: List[str]) -> Optional[int]:
+    if len(tokens) < 2:
+        return None
+    try:
+        return int(tokens[1])
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_vip_entry(tokens: List[str]) -> Optional[Dict[str, Any]]:
+    """One `virtual_ipaddress` line: `<addr>[/<prefix>] [dev <iface>] [label ...]`.
+
+    Returns None when the first token is not an address — a shape we do not understand must
+    surface as unsupported rather than be silently dropped.
+    """
+    spec = tokens[0]
+    addr, _, prefix = spec.partition("/")
+    try:
+        ip = ipaddress.ip_address(addr)
+    except ValueError:
+        return None
+    entry: Dict[str, Any] = {
+        "address": str(ip),
+        "prefix_length": None,
+        "dev": None,
+        "extra": [],
+    }
+    if prefix:
+        try:
+            entry["prefix_length"] = int(prefix)
+        except ValueError:
+            return None
+    rest = tokens[1:]
+    i = 0
+    while i < len(rest):
+        if rest[i] == "dev" and i + 1 < len(rest):
+            entry["dev"] = rest[i + 1]
+            i += 2
+            continue
+        # `label`, `scope`, `brd`, ... — all real directives we do not render.
+        entry["extra"].append(rest[i])
+        i += 1
+    return entry
+
+
+def _classify_script(block: Dict[str, Any]) -> Tuple[str, Optional[str]]:
+    """Return (name, script_path) for a vrrp_script block."""
+    name = block["args"][0] if block["args"] else (block["name"] or "")
+    path = None
+    for entry in block["body"]:
+        if entry["kind"] == "line" and entry["tokens"] and entry["tokens"][0] == "script":
+            path = " ".join(entry["tokens"][1:]) or None
+    return name, path
+
+
+def _parse_instance(block: Dict[str, Any]) -> Dict[str, Any]:
+    """Interpret one `vrrp_instance` block into VIP-model fields plus its own unsupported list."""
+    inst: Dict[str, Any] = {
+        "instance_name": block["args"][0] if block["args"] else "",
+        "state": None,
+        "interface": None,
+        "virtual_router_id": None,
+        "priority": None,
+        "advert_int": None,
+        "auth_type": None,
+        "auth_pass": None,
+        "unicast_src_ip": None,
+        "unicast_peers": [],
+        "virtual_ips": [],
+        "track_scripts": [],
+        "unsupported": [],
+        "line": block["line"],
+    }
+
+    def unsupported(what: str, lineno: int) -> None:
+        inst["unsupported"].append({"directive": what, "line": lineno})
+
+    for entry in block["body"]:
+        if entry["kind"] == "line":
+            tokens = entry["tokens"]
+            key = tokens[0]
+            if key == "state":
+                inst["state"] = (tokens[1].upper() if len(tokens) > 1 else None)
+            elif key == "interface":
+                inst["interface"] = tokens[1] if len(tokens) > 1 else None
+            elif key == "virtual_router_id":
+                inst["virtual_router_id"] = _as_int(tokens)
+            elif key == "priority":
+                inst["priority"] = _as_int(tokens)
+            elif key == "advert_int":
+                # keepalived accepts sub-second floats; our model column is an integer.
+                raw = tokens[1] if len(tokens) > 1 else ""
+                try:
+                    val = float(raw)
+                except (TypeError, ValueError):
+                    val = None
+                if val is None:
+                    unsupported(f"advert_int {raw}", entry["line"])
+                elif val != int(val):
+                    # Rounding would change VRRP timing, so refuse rather than adopt-and-alter.
+                    unsupported(f"advert_int {raw} (fractional; model stores whole seconds)",
+                                entry["line"])
+                else:
+                    inst["advert_int"] = int(val)
+            elif key == "unicast_src_ip":
+                inst["unicast_src_ip"] = tokens[1] if len(tokens) > 1 else None
+            else:
+                unsupported(" ".join(tokens), entry["line"])
+            continue
+
+        name = entry["name"]
+        if name == "authentication":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    continue
+                k = sub["tokens"][0]
+                if k == "auth_type":
+                    inst["auth_type"] = (sub["tokens"][1].upper() if len(sub["tokens"]) > 1 else None)
+                elif k == "auth_pass":
+                    # Everything after the keyword: a VRRP password may contain spaces.
+                    inst["auth_pass"] = " ".join(sub["tokens"][1:]) or None
+                else:
+                    unsupported(f"authentication/{' '.join(sub['tokens'])}", sub["line"])
+        elif name == "unicast_peer":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["unicast_peers"].append(sub["tokens"][0])
+                else:
+                    unsupported("unicast_peer/<block>", entry["line"])
+        elif name == "virtual_ipaddress":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    unsupported("virtual_ipaddress/<block>", entry["line"])
+                    continue
+                parsed = _parse_vip_entry(sub["tokens"])
+                if parsed is None:
+                    unsupported(f"virtual_ipaddress/{' '.join(sub['tokens'])}", sub["line"])
+                else:
+                    if parsed["extra"]:
+                        unsupported(
+                            f"virtual_ipaddress/{parsed['address']} "
+                            f"({' '.join(parsed['extra'])})", sub["line"])
+                    inst["virtual_ips"].append(parsed)
+        elif name == "track_script":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["track_scripts"].append(sub["tokens"][0])
+        else:
+            unsupported(f"{name} {{...}}", entry["line"])
+
+    return inst
+
+
+def parse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse a keepalived.conf into VIP-model fields plus everything we could not model.
+
+    Raises KeepalivedParseError on structurally broken input. Never log the result: parsed
+    instances carry `auth_pass` in cleartext.
+    """
+    root = _read_blocks(text or "")
+    result: Dict[str, Any] = {
+        "instances": [],
+        "scripts": {},
+        "global_defs": {},
+        "unsupported": [],   # top-level directives our render would drop
+        "sync_groups": [],
+    }
+
+    for entry in root:
+        if entry["kind"] == "line":
+            # A bare top-level directive (e.g. `include /etc/keepalived/conf.d/*.conf`).
+            result["unsupported"].append(
+                {"directive": " ".join(entry["tokens"]), "line": entry["line"]})
+            continue
+        name = entry["name"]
+        if name == "global_defs":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    key = sub["tokens"][0]
+                    result["global_defs"][key] = " ".join(sub["tokens"][1:])
+                    if key not in _SUPPORTED_GLOBAL_KEYS:
+                        result["unsupported"].append(
+                            {"directive": f"global_defs/{' '.join(sub['tokens'])}",
+                             "line": sub["line"]})
+                else:
+                    result["unsupported"].append(
+                        {"directive": f"global_defs/{sub.get('name', '?')} {{...}}",
+                         "line": sub["line"]})
+        elif name == "vrrp_script":
+            script_name, path = _classify_script(entry)
+            result["scripts"][script_name] = {"script": path, "line": entry["line"]}
+        elif name == "vrrp_instance":
+            result["instances"].append(_parse_instance(entry))
+        elif name == "vrrp_sync_group":
+            # A sync group ties instances together so they fail over as a unit. Our render has
+            # no equivalent, and dropping it changes failover semantics — never adopt silently.
+            group = entry["args"][0] if entry["args"] else ""
+            result["sync_groups"].append({"name": group, "line": entry["line"]})
+            result["unsupported"].append(
+                {"directive": f"vrrp_sync_group {group}", "line": entry["line"]})
+        else:
+            # virtual_server (LVS), static_routes, bfd_instance, ...
+            args = " ".join(entry["args"])
+            result["unsupported"].append(
+                {"directive": f"{name} {args} {{...}}".replace("  ", " "), "line": entry["line"]})
+
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Mapping to the VIP model + the adoption gate
+# ---------------------------------------------------------------------------
+# keepalived defaults we are willing to apply when a directive is absent, because the value
+# is unambiguous and re-rendering it changes nothing on the wire.
+_DEFAULT_ADVERT_INT = 1
+_DEFAULT_PRIORITY = 100
+_DEFAULT_STATE = "BACKUP"
+
+# The only track_script our renderer emits (keepalived_config.build_haproxy_check_script).
+OUR_CHECK_SCRIPT_NAME = "chk_haproxy"
+
+
+def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -> Dict[str, Any]:
+    """Map one parsed `vrrp_instance` onto vip_instances / vip_members fields.
+
+    Returns `adoptable` plus `blockers`. A blocker means taking ownership would change what
+    is running — either because our render cannot reproduce something in the file, or because
+    a value we must write is not knowable from the file. Adoption REPLACES the operator's
+    config, so "we could not read it" and "we would change it" are the same hazard, and both
+    have to stop the flow rather than be logged.
+
+    Never log the return value: `vip.auth_pass` is cleartext.
+    """
+    blockers: List[str] = []
+
+    # Directives we would drop. Report the file's own line numbers so the operator can look.
+    dropped = list(parsed.get("unsupported") or []) + list(instance.get("unsupported") or [])
+    for d in dropped:
+        blockers.append(
+            f"line {d['line']}: `{d['directive']}` — OpenManager's renderer cannot reproduce "
+            f"this, so adopting would delete it")
+
+    vips = instance.get("virtual_ips") or []
+    if len(vips) == 0:
+        blockers.append("the instance declares no virtual_ipaddress — nothing to adopt")
+    elif len(vips) > 1:
+        addrs = ", ".join(v["address"] for v in vips)
+        blockers.append(
+            f"the instance carries {len(vips)} addresses ({addrs}); a managed VIP holds exactly "
+            f"one, so adopting would drop all but the first")
+
+    vip_entry = vips[0] if vips else None
+
+    if instance.get("virtual_router_id") is None:
+        blockers.append("no virtual_router_id — it cannot be guessed: a wrong VRID puts the "
+                        "nodes in separate VRRP domains and both would claim the VIP")
+    if not instance.get("interface"):
+        blockers.append("no interface — required to render the instance and the address")
+
+    # An explicit prefix is required. Our renderer ALWAYS writes `<addr>/<prefix>`, the model
+    # column defaults to 24, and keepalived's own default for a bare address is a host route.
+    # Picking either one for the operator would silently change the VIP's netmask, so ask.
+    if vip_entry is not None and vip_entry.get("prefix_length") is None:
+        blockers.append(
+            f"`{vip_entry['address']}` has no explicit prefix length; state it during adoption "
+            f"so the netmask cannot change on takeover")
+
+    # The address must live on the instance's interface — that is the only `dev` we can render.
+    if vip_entry is not None and vip_entry.get("dev") and instance.get("interface") \
+            and vip_entry["dev"] != instance["interface"]:
+        blockers.append(
+            f"the address is bound to `dev {vip_entry['dev']}` but the instance uses "
+            f"`interface {instance['interface']}`; the render always uses the instance interface")
+
+    auth_type = instance.get("auth_type")
+    if auth_type not in (None, "PASS"):
+        blockers.append(f"auth_type {auth_type} is not supported (only PASS is rendered)")
+
+    # A tracked script that is not ours would be replaced by our HAProxy check.
+    tracked = [t for t in (instance.get("track_scripts") or [])]
+    foreign = [t for t in tracked if t != OUR_CHECK_SCRIPT_NAME]
+    if foreign:
+        blockers.append(
+            f"track_script {', '.join(foreign)} would be replaced by OpenManager's HAProxy "
+            f"health check")
+
+    state = instance.get("state") or _DEFAULT_STATE
+    if state not in ("MASTER", "BACKUP"):
+        blockers.append(f"state {state} is not MASTER or BACKUP")
+
+    peers = list(instance.get("unicast_peers") or [])
+    src = instance.get("unicast_src_ip")
+    # Our renderer emits unicast_src_ip and unicast_peer together, or neither.
+    if bool(src) != bool(peers):
+        which = "unicast_src_ip without unicast_peer" if src else "unicast_peer without unicast_src_ip"
+        blockers.append(f"{which} — the render emits both or neither")
+
+    candidate: Dict[str, Any] = {
+        "instance_name": instance.get("instance_name") or "",
+        "adoptable": not blockers,
+        "blockers": blockers,
+        "dropped_directives": dropped,
+        "vip": {
+            "virtual_ip": vip_entry["address"] if vip_entry else None,
+            "prefix_length": vip_entry.get("prefix_length") if vip_entry else None,
+            "virtual_router_id": instance.get("virtual_router_id"),
+            "advert_int": instance.get("advert_int") if instance.get("advert_int") is not None
+            else _DEFAULT_ADVERT_INT,
+            "use_unicast": bool(peers),
+            "track_haproxy": OUR_CHECK_SCRIPT_NAME in tracked,
+            "auth_pass": instance.get("auth_pass"),
+        },
+        "member": {
+            "network_interface": instance.get("interface"),
+            "role": state,
+            "priority": instance.get("priority") if instance.get("priority") is not None
+            else _DEFAULT_PRIORITY,
+        },
+        "peers": peers,
+        "unicast_src_ip": src,
+        # Which values came from a keepalived default rather than the file, so the UI can say so.
+        "defaulted": [
+            k for k, present in (
+                ("advert_int", instance.get("advert_int") is not None),
+                ("priority", instance.get("priority") is not None),
+                ("state", instance.get("state") is not None),
+            ) if not present
+        ],
+    }
+    return candidate
+
+
+# Substrings that identify the two blocker classes an operator is allowed to resolve. They are
+# matched rather than typed because the blocker text is what the UI shows; keeping the marker in
+# the sentence means the message and the rule cannot drift apart.
+_LOSS_MARKER = "would delete it"
+_PREFIX_MARKER = "no explicit prefix length"
+
+
+def remaining_blockers(blockers: List[str], *, prefix_supplied: bool = False,
+                       accept_data_loss: bool = False) -> List[str]:
+    """Blockers that survive what the operator is permitted to resolve.
+
+    Exactly two classes are resolvable, and the distinction is the whole safety argument:
+
+      * a missing prefix length is *unknown*, and the operator can supply it — we refuse to pick
+        a netmask for a live VIP ourselves;
+      * "our renderer cannot reproduce this, so adopting would delete it" is a *loss*, and losing
+        it can be an informed choice.
+
+    Everything else — an unknown virtual_router_id, a fractional advert_int, an unsupported
+    auth_type, an address on a different interface — is neither unknown nor a loss but an
+    impossibility, and no flag may wave it through. This is the single source of truth for that
+    rule; the endpoint and the UI both derive from it.
+    """
+    out: List[str] = []
+    for b in blockers or []:
+        if prefix_supplied and _PREFIX_MARKER in b:
+            continue
+        if accept_data_loss and _LOSS_MARKER in b:
+            continue
+        out.append(b)
+    return out
+
+
+def analyse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse + map in one call: the shape the discovery endpoint stores and the UI renders."""
+    parsed = parse_keepalived_conf(text)
+    return {
+        "instance_count": len(parsed["instances"]),
+        "sync_groups": parsed["sync_groups"],
+        "global_defs": parsed["global_defs"],
+        "candidates": [build_adoption_candidate(parsed, inst) for inst in parsed["instances"]],
+    }

backend/services/mfa_service.py

@@ -0,0 +1,238 @@
+"""MFA (TOTP + backup codes) service layer — Issue #18, v1.6.0.
+
+Owns the cryptographic and persistence-shape concerns of multi-factor auth:
+  - TOTP secret generation / verification with replay protection (RFC 6238)
+  - Backup code generation, hashing (bcrypt) and atomic single-use consumption
+  - Fernet-based encryption of TOTP secrets at rest
+
+Strictly no logging of secrets — only metadata (lengths, counts) is logged.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import secrets as _secrets
+import time
+from typing import List, Optional, Tuple
+from urllib.parse import quote
+
+import bcrypt
+import pyotp
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# RFC 6238 parameters — kept conservative for widest authenticator app compatibility.
+TOTP_DIGITS = 6
+TOTP_PERIOD = 30
+TOTP_DIGEST = "sha1"
+TOTP_VALID_WINDOW_STEPS = 1  # ±1 step (±30s) tolerance
+
+# Backup code spec (Plan section 5).
+# Alphabet drops the confusing pairs: 0/O, 1/I, L. Resulting size is 31, which
+# still yields 31**8 ≈ 8.5×10^11 combinations per half — far beyond brute-force.
+BACKUP_CODE_COUNT = 10
+BACKUP_CODE_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"
+BACKUP_CODE_HALF_LEN = 4  # XXXX-YYYY
+
+# OTP URI defaults.
+DEFAULT_ISSUER = "HAProxy OpenManager"
+ACCOUNT_LABEL_DOMAIN_FALLBACK = "haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Fernet key resolution
+# ---------------------------------------------------------------------------
+
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Resolve the Fernet key, preferring the explicit env var.
+
+    Falls back to HKDF over SECRET_KEY with a versioned info string so a future
+    rotation can be expressed by bumping the version suffix.
+    """
+    explicit = os.getenv("MFA_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:
+            logger.error("MFA_ENCRYPTION_KEY env var present but invalid: %s", exc)
+            # fall through to HKDF derivation rather than crashing the app
+
+    logger.warning(
+        "MFA_ENCRYPTION_KEY env var not set or invalid; deriving from SECRET_KEY (v1). "
+        "Set an explicit MFA_ENCRYPTION_KEY in production to enable key rotation."
+    )
+    hkdf = HKDF(
+        algorithm=hashes.SHA256(),
+        length=32,
+        salt=None,
+        info=b"mfa-totp-secret-v1",
+    )
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution of the Fernet key after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+# ---------------------------------------------------------------------------
+# TOTP secrets
+# ---------------------------------------------------------------------------
+
+
+def generate_totp_secret() -> str:
+    """Return a fresh base32 TOTP secret (32 chars)."""
+    return pyotp.random_base32()
+
+
+def encrypt_secret(secret_plain: str) -> str:
+    """Fernet-encrypt the base32 secret. Returns str for direct DB storage."""
+    token = _get_fernet().encrypt(secret_plain.encode("utf-8"))
+    return token.decode("utf-8")
+
+
+def decrypt_secret(secret_encrypted: str) -> Optional[str]:
+    """Decrypt a previously stored secret. Returns None when the token can't be
+    decrypted (e.g. key rotated without re-enroll). Never raises to the caller.
+    """
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt MFA secret (invalid Fernet token)")
+        return None
+    except Exception as exc:
+        logger.error("Unexpected error decrypting MFA secret: %s", exc)
+        return None
+
+
+def build_otpauth_uri(account_label: str, secret_plain: str, issuer: str = DEFAULT_ISSUER) -> str:
+    """Build an otpauth:// URI that all major authenticator apps accept.
+
+    Format: otpauth://totp/<issuer>:<account>?secret=<b32>&issuer=<issuer>&algorithm=SHA1&digits=6&period=30
+    """
+    issuer_q = quote(issuer, safe="")
+    label = f"{issuer}:{account_label}"
+    label_q = quote(label, safe=":@")
+    return (
+        f"otpauth://totp/{label_q}?secret={secret_plain}"
+        f"&issuer={issuer_q}&algorithm=SHA1&digits={TOTP_DIGITS}&period={TOTP_PERIOD}"
+    )
+
+
+def build_account_label(username: str, hostname_hint: Optional[str] = None) -> str:
+    """Compose the per-user otpauth label, respecting env > hostname > fallback."""
+    domain = (
+        os.getenv("MFA_ACCOUNT_LABEL_DOMAIN", "").strip()
+        or (hostname_hint or "").strip()
+        or ACCOUNT_LABEL_DOMAIN_FALLBACK
+    )
+    return f"{username}@{domain}"
+
+
+def verify_totp_with_replay_guard(
+    secret_plain: str,
+    code: str,
+    last_used_step: Optional[int],
+) -> Tuple[bool, Optional[int]]:
+    """Verify a 6-digit TOTP code with explicit per-step replay protection.
+
+    Returns (success, step_consumed). Caller persists the consumed step on success.
+
+    Implementation notes:
+      - pyotp.TOTP.at(seconds_since_epoch) — to target step N we pass step*PERIOD.
+      - secrets.compare_digest is used for constant-time comparison.
+      - Replay guard rejects codes whose step is <= the previously consumed step.
+    """
+    if not secret_plain or not code:
+        return (False, None)
+    code = code.strip()
+    if len(code) != TOTP_DIGITS or not code.isdigit():
+        return (False, None)
+
+    totp = pyotp.TOTP(secret_plain, digits=TOTP_DIGITS, interval=TOTP_PERIOD, digest=TOTP_DIGEST)
+    now = int(time.time())
+    current_step = now // TOTP_PERIOD
+
+    for offset in (0, -1, 1):
+        step = current_step + offset
+        expected = totp.at(step * TOTP_PERIOD)
+        if len(expected) == len(code) and _secrets.compare_digest(expected, code):
+            if last_used_step is not None and step <= last_used_step:
+                return (False, None)
+            return (True, step)
+    return (False, None)
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+def generate_backup_codes(count: int = BACKUP_CODE_COUNT) -> List[str]:
+    """Return ``count`` plain-text backup codes formatted as ``XXXX-YYYY``."""
+    codes: List[str] = []
+    for _ in range(count):
+        left = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        right = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        codes.append(f"{left}-{right}")
+    return codes
+
+
+def normalize_backup_code(user_input: str) -> str:
+    """Canonical form for comparison: uppercase, strip dashes/spaces."""
+    if not user_input:
+        return ""
+    return user_input.strip().upper().replace("-", "").replace(" ", "")
+
+
+async def _hash_one_backup_code(code_plain: str) -> str:
+    """Bcrypt-hash a single backup code on a worker thread."""
+    normalized = normalize_backup_code(code_plain)
+    hashed = await asyncio.to_thread(bcrypt.hashpw, normalized.encode("utf-8"), bcrypt.gensalt())
+    return hashed.decode("utf-8")
+
+
+async def hash_backup_codes(codes_plain: List[str]) -> List[str]:
+    """Hash backup codes in parallel (each bcrypt op runs in its own thread)."""
+    return await asyncio.gather(*(_hash_one_backup_code(c) for c in codes_plain))
+
+
+async def check_backup_code(user_input: str, code_hash: str) -> bool:
+    """Run a single bcrypt verify on the worker pool."""
+    normalized = normalize_backup_code(user_input)
+    if not normalized:
+        return False
+    return await asyncio.to_thread(
+        bcrypt.checkpw, normalized.encode("utf-8"), code_hash.encode("utf-8")
+    )
+
+
+# ---------------------------------------------------------------------------
+# Misc helpers
+# ---------------------------------------------------------------------------
+
+
+def generate_challenge_token() -> str:
+    """64-char hex challenge token for /api/auth/login → /mfa-verify hand-off."""
+    return _secrets.token_hex(32)

backend/services/ssl_service.py

@@ -40,10 +40,12 @@ flow.
   (callers translate to wizard step-jumpback toasts).
 """
 
+import hashlib
 import json
 import logging
+import time
 from datetime import datetime, timezone
-from typing import Any, Optional
+from typing import Any, List, Optional
 
 from fastapi import HTTPException
 
@@ -106,23 +108,22 @@ def _recompute_status_from_expiry(
         return cert_info_status or "valid", cert_info_days or 0
 
 
-async def create_cert_row(
-    conn,
-    payload: Any,
-    cluster_id: int,
-) -> int:
-    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
-    binding to the given cluster_id. Returns new ssl_certificate_id.
+def _prepare_cert_fields(payload: Any) -> dict:
+    """Parse + validate the PEM material on `payload` and derive every
+    ssl_certificates column value from it (v1.9.0 extraction — shared by
+    `create_cert_row` and the CSR import flow in services/csr_service.py,
+    byte-identical to the former inline body of `create_cert_row`).
 
     payload is expected to expose:
       name, certificate_content, private_key_content, chain_content,
       usage_type (optional, default 'frontend').
 
-    All cert metadata (primary_domain, all_domains, expiry_date,
-    issuer, fingerprint, status, days_until_expiry) is now parsed
-    FROM the PEM content via `parse_ssl_certificate` — operator-
-    supplied values on the payload are accepted as a graceful
-    fallback only when parsing fails (which itself raises 400).
+    Raises HTTPException(400) on any parse/validation failure (invalid PEM,
+    bad private key, cert/key mismatch, bad chain, already-expired cert).
+
+    Returns a dict with keys: cert_content, private_key_content,
+    chain_content, cert_info, primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry, usage_type.
     """
     cert_content = getattr(payload, "certificate_content", None) or ""
     if not cert_content.strip():
@@ -213,6 +214,53 @@ async def create_cert_row(
     )
     usage_type = getattr(payload, "usage_type", "frontend") or "frontend"
 
+    return {
+        "cert_content": cert_content,
+        "private_key_content": private_key_content,
+        "chain_content": chain_content,
+        "cert_info": cert_info,
+        "primary_domain": primary_domain,
+        "all_domains": all_domains,
+        "expiry_date": expiry_date,
+        "issuer": issuer,
+        "fingerprint": fingerprint,
+        "status": status,
+        "days_until_expiry": days_until_expiry,
+        "usage_type": usage_type,
+    }
+
+
+async def create_cert_row(
+    conn,
+    payload: Any,
+    cluster_id: int,
+) -> int:
+    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
+    binding to the given cluster_id. Returns new ssl_certificate_id.
+
+    payload is expected to expose:
+      name, certificate_content, private_key_content, chain_content,
+      usage_type (optional, default 'frontend').
+
+    All cert metadata (primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry) is now parsed
+    FROM the PEM content via `parse_ssl_certificate` — operator-
+    supplied values on the payload are accepted as a graceful
+    fallback only when parsing fails (which itself raises 400).
+    """
+    fields = _prepare_cert_fields(payload)
+    cert_content = fields["cert_content"]
+    private_key_content = fields["private_key_content"]
+    chain_content = fields["chain_content"]
+    expiry_date = fields["expiry_date"]
+    primary_domain = fields["primary_domain"]
+    all_domains = fields["all_domains"]
+    issuer = fields["issuer"]
+    fingerprint = fields["fingerprint"]
+    status = fields["status"]
+    days_until_expiry = fields["days_until_expiry"]
+    usage_type = fields["usage_type"]
+
     existing = await conn.fetchrow(
         """
         SELECT s.id, s.is_active
@@ -408,3 +456,81 @@ async def validate_server_ca_bundle_eligibility(
         cluster_id,
     )
     return row is not None
+
+
+async def stage_ssl_config_versions(
+    conn,
+    cert_id: int,
+    cluster_ids: List[int],
+    action: str = "create",
+    created_by: Optional[int] = None,
+) -> List[dict]:
+    """Stage one PENDING config version per affected cluster after an SSL
+    certificate mutation (v1.9.0 — distilled from the routers/ssl.py POST
+    /certificates staging loop; used by the CSR import flow).
+
+    Uses the EXACT `ssl-{cert_id}-{action}-{timestamp}` version-name scheme of
+    the manual SSL flow so Apply Management, the `has_pending_config`
+    LIKE-filter ('ssl-' || id || '-%'), and the agent delivery predicates
+    treat CSR-imported certificates identically to manually uploaded ones.
+    Agents are NOT notified here — the operator applies manually.
+
+    Per-cluster failures are caught and reported in the returned
+    sync_results list (the DB save has already succeeded — same semantics as
+    the manual flow, where a config-generation failure never rolls back the
+    certificate row).
+    """
+    # Local import: keeps services/haproxy_config free to import ssl helpers
+    # without a module-level cycle.
+    from services.haproxy_config import generate_haproxy_config_for_cluster
+
+    sync_results: List[dict] = []
+    for cluster_id in cluster_ids:
+        try:
+            config_content = await generate_haproxy_config_for_cluster(cluster_id)
+            config_hash = hashlib.sha256(config_content.encode()).hexdigest()
+            version_name = f"ssl-{cert_id}-{action}-{int(time.time())}"
+
+            version_created_by = created_by
+            if version_created_by is None:
+                version_created_by = await conn.fetchval(
+                    "SELECT id FROM users WHERE username = 'admin' LIMIT 1"
+                ) or 1
+
+            await conn.fetchval(
+                """
+                INSERT INTO config_versions
+                (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
+                VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                RETURNING id
+                """,
+                cluster_id,
+                version_name,
+                config_content,
+                config_hash,
+                version_created_by,
+            )
+            logger.info(
+                f"APPLY WORKFLOW: Created PENDING config version {version_name} "
+                f"for cluster {cluster_id} (ssl_service.stage_ssl_config_versions)"
+            )
+            sync_results.append({
+                'node': 'pending',
+                'success': True,
+                'cluster_id': cluster_id,
+                'version': version_name,
+                'status': 'PENDING',
+                'message': 'SSL certificate staged. Click Apply to activate.',
+            })
+        except Exception as e:
+            logger.error(
+                f"Cluster config staging failed for SSL certificate {cert_id} "
+                f"on cluster {cluster_id}: {e}"
+            )
+            sync_results.append({
+                'node': 'cluster',
+                'success': False,
+                'cluster_id': cluster_id,
+                'error': str(e),
+            })
+    return sync_results

backend/tests/test_acl_pattern_file_allow.py

@@ -0,0 +1,190 @@
+"""Issue #38 follow-up — ACL `-f <file>` pattern-file support (v1.8.9).
+
+The Bulgu #12 hard rejects were removed: pattern files are
+operator-managed host files (same policy as the SPOE
+`filter ... config <path>` reference preserved since v1.8.8), bulk
+import always accepted `-f`, and the agent runs `haproxy -c` before
+every reload so a missing file fails safely. These tests pin:
+
+1. ACCEPT — the manual FrontendConfig model and the wizard models
+   accept `-f` in every rule field (string + dict shapes).
+2. GUARDS KEPT — `$(`/backtick shell-substitution rejects and the
+   `X !X` contradiction machinery are unchanged.
+3. WARNINGS — `_pattern_file_warnings` emits exactly one advisory
+   listing the referenced files, and NOTHING for `-f`-free rules
+   (zero-noise: existing users see no new output).
+4. ADVISORY — the bulk-import preview advisory block scans
+   acl/use_backend rules (and only those fields).
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from models.frontend import FrontendConfig  # noqa: E402
+from routers.frontend import _pattern_file_warnings  # noqa: E402
+
+
+ACL_F = "blacklisted src -f /etc/haproxy/blacklist.lst"
+UB_F = "be-secure if { src -f /etc/haproxy/allowlist.lst }"
+REDIR_F = "location /blocked if { src -f /etc/haproxy/blacklist.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 1. ACCEPT — manual FrontendConfig model
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_frontend_config_accepts_acl_file_flag():
+    fe = FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[ACL_F])
+    assert fe.acl_rules == [ACL_F]
+
+
+def test_frontend_config_accepts_use_backend_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", use_backend_rules=[UB_F])
+    assert fe.use_backend_rules == [UB_F]
+
+
+def test_frontend_config_accepts_redirect_string_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[REDIR_F])
+    assert fe.redirect_rules == [REDIR_F]
+
+
+def test_frontend_config_accepts_redirect_dict_file_flag():
+    rule = {"type": "scheme", "scheme": "https",
+            "condition": "if { src -f /etc/haproxy/blacklist.lst }"}
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[rule])
+    assert fe.redirect_rules == [rule]
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 2. GUARDS KEPT — dangerous-content rejects unchanged
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "acl1 path $(rm -rf /)",
+    "acl1 path `id`",
+])
+def test_acl_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[bad_rule])
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "be1 if $(whoami)",
+    "be1 if `id`",
+])
+def test_use_backend_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(
+            name="fe1", bind_port=80, mode="http", use_backend_rules=[bad_rule])
+
+
+def test_contradiction_detection_still_works_on_file_flag_rules():
+    """Interaction guard: a `-f` rule with an `X !X` contradiction is
+    still caught by the handler-level contradiction machinery — the
+    `-f` relaxation must not weaken that gate."""
+    from models.frontend import _frontend_has_acl_contradiction
+    assert _frontend_has_acl_contradiction(
+        "be1 if blacklisted !blacklisted") is True
+    # And a normal -f rule is NOT a contradiction.
+    assert _frontend_has_acl_contradiction(UB_F) is False
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 3. WARNINGS — _pattern_file_warnings (zero-noise contract)
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_pattern_file_warnings_lists_unique_paths():
+    warnings = _pattern_file_warnings(
+        acl_rules=[ACL_F, "other src -f /etc/haproxy/blacklist.lst"],
+        use_backend_rules=[UB_F],
+        redirect_rules=[{"condition": "if { src -f /etc/haproxy/geo.lst }"}],
+    )
+    assert len(warnings) == 1
+    w = warnings[0]
+    assert "/etc/haproxy/blacklist.lst" in w
+    assert "/etc/haproxy/allowlist.lst" in w
+    assert "/etc/haproxy/geo.lst" in w
+    # Duplicate path listed once.
+    assert w.count("/etc/haproxy/blacklist.lst") == 1
+    # Non-blocking framing: mentions fail-safe haproxy -c.
+    assert "haproxy -c" in w
+
+
+def test_pattern_file_warnings_empty_without_file_flag():
+    """Zero-noise: operators who don't use `-f` must see NO warning."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_api path_beg /api", "is_admin src 10.0.0.0/24"],
+        use_backend_rules=["be-api if is_api"],
+        redirect_rules=[{"type": "scheme", "scheme": "https",
+                         "condition": "if !{ ssl_fc }"}],
+    ) == []
+    assert _pattern_file_warnings() == []
+
+
+def test_pattern_file_warnings_ignores_dash_f_substrings():
+    """`-file`/`-foo` substrings must not trigger the advisory."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_self path_beg /self-config-file",
+                   "is_foo path_beg /foo -m beg"],
+    ) == []
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 4. Wizard models accept `-f` (string + dict) — parity
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_wizard_models_accept_file_flag():
+    from models.site_wizard import FrontendStep
+
+    fe = FrontendStep(
+        name="fe1", mode="http", bind_address="*", bind_port=80,
+        acl_rules=[ACL_F],
+        use_backend_rules=["be-x if blacklisted"],
+        redirect_rules=[{"type": "scheme", "target": "https",
+                         "condition": "if { src -f /etc/haproxy/x.lst }"}],
+    )
+    assert fe.acl_rules == [ACL_F]
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/x.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 5. Bulk-import preview advisory — source-level pin
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_parse_bulk_advisory_scans_only_structured_rule_fields():
+    """The preview advisory scans acl_rules/use_backend_rules but NOT
+    request_headers/tcp_request_rules (always-free-form fields —
+    warning there would add new noise for existing users)."""
+    src = Path(__file__).resolve().parents[1] / "routers" / "config.py"
+    text = src.read_text()
+    block_start = text.index("pattern-file advisory")
+    block = text[block_start:block_start + 1200]
+    assert 'acl_rules' in block
+    assert 'use_backend_rules' in block
+    assert 'request_headers' not in block.split("_pattern_paths")[1], (
+        "advisory must not scan request_headers")
+
+
+def test_no_dash_f_reject_left_in_models():
+    """No model file may still hard-reject the `-f` flag."""
+    for rel in ("models/frontend.py", "models/site_wizard.py"):
+        text = (Path(__file__).resolve().parents[1] / rel).read_text()
+        for m in re.finditer(r"-f\(\\s\|\$\)", text):
+            ctx = text[max(0, m.start() - 400):m.start() + 400]
+            assert "raise ValueError" not in ctx, (
+                f"{rel}: a `-f` reject regex still sits next to a raise")

backend/tests/test_acme_backend_url.py

@@ -0,0 +1,291 @@
+"""ACME challenge backend URL validation, resolution and change detection.
+
+These pin the behaviour behind a real incident: a split deployment rendered
+`server _acme_mgmt <mgmt>:8080` against a port with no listener, HTTP-01 failed for
+weeks while DNS-01 kept working, and every existing check reported success. The
+three mechanisms below are what make that impossible to repeat silently.
+"""
+import pytest
+
+from services.haproxy_config import (
+    extract_acme_backend_target,
+    is_config_generation_error,
+    select_acme_backend_source,
+)
+from utils.acme_backend_url import (
+    AcmeBackendUrlError,
+    resolve_acme_backend_target,
+    validate_acme_backend_url,
+)
+
+
+# ---------------------------------------------------------------------------
+# 1. Boundary validation — what an operator may type.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "value,expected",
+    [
+        ("http://10.90.1.4:80", "http://10.90.1.4:80"),
+        ("http://10.90.1.4", "http://10.90.1.4"),
+        ("https://mgmt.internal:8443", "https://mgmt.internal:8443"),
+        # RFC1918 is the NORMAL answer here, unlike utils/ssrf_guard's policy: the
+        # operator is naming their own management host, which on a split deployment
+        # is private by definition.
+        ("http://192.168.1.5:8080", "http://192.168.1.5:8080"),
+        # Empty means "inherit from the next level of the resolution chain".
+        (None, None),
+        ("", None),
+        ("   ", None),
+        # Surrounding whitespace is normalised, not rejected — and the NORMALISED
+        # value is what callers persist, so it can never reach haproxy.cfg.
+        ("  http://10.0.0.5:80  ", "http://10.0.0.5:80"),
+    ],
+)
+def test_accepts_and_normalises_usable_values(value, expected):
+    assert validate_acme_backend_url(value) == expected
+
+
+@pytest.mark.parametrize(
+    "value,code",
+    [
+        # Scheme-less values used to be accepted and then silently became `localhost`
+        # in the renderer — the trap that makes a correct diagnosis un-actionable.
+        ("10.90.1.4:8080", "no_scheme"),
+        ("localhost:8080", "no_scheme"),
+        ("ftp://10.0.0.5", "bad_scheme"),
+        # A newline would inject directives into a file pushed to every node.
+        ("http://10.90.1.4\nbind :9", "whitespace"),
+        ("http://10.90.1.4 x", "whitespace"),
+        # Loopback by number AND by name: `localhost` is what both shipped defaults
+        # contain, so catching only the numeric form would miss the common case.
+        ("http://localhost:8080", "loopback"),
+        ("http://LOCALHOST", "loopback"),
+        ("http://127.0.0.1", "loopback"),
+        ("http://[::1]:80", "loopback"),
+        ("http://0.0.0.0:80", "unspecified"),
+        ("http://169.254.169.254", "link_local"),
+        ("http://u:p@10.0.0.5", "userinfo"),
+        ("http://10.0.0.5/api", "has_path"),
+        ("http://10.0.0.5?x=1", "has_path"),
+        # urlparse defers port parsing to attribute access; unguarded this raises
+        # inside the config generator and destroys the cluster's whole config.
+        ("http://10.0.0.5:99999", "bad_port"),
+        ("http://10.0.0.5:abc", "bad_port"),
+        ("http://-bad-.com", "invalid_host"),
+    ],
+)
+def test_rejects_unusable_values_with_stable_codes(value, code):
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url(value)
+    assert exc.value.code == code
+    assert str(exc.value), "every rejection must carry operator-facing prose"
+
+
+def test_rejects_values_longer_than_the_column():
+    # VARCHAR(500); without this the write fails as an opaque asyncpg 22001 -> 500.
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url("http://" + "a" * 600 + ".com")
+    assert exc.value.code == "too_long"
+
+
+# ---------------------------------------------------------------------------
+# 2. Render-time resolution — never rejects, never raises.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "url,host,port,ssl_flag",
+    [
+        ("http://10.90.1.4:80", "10.90.1.4", 80, ""),
+        # Port-less http stays 8080, NOT the scheme default 80: the bundled compose
+        # publishes nginx on 8080, so installs relying on this have a working path
+        # today and changing it would break them silently in the renewal loop.
+        ("http://10.90.1.4", "10.90.1.4", 8080, ""),
+        ("https://m.io", "m.io", 443, " ssl verify none"),
+        ("http://localhost:8080", "localhost", 8080, ""),
+    ],
+)
+def test_resolution_preserves_existing_rendering(url, host, port, ssl_flag):
+    target = resolve_acme_backend_target(url)
+    assert (target.host, target.port, target.ssl_flag) == (host, port, ssl_flag)
+    assert target.error_code is None
+
+
+@pytest.mark.parametrize(
+    "url",
+    ["", None, "   ", "10.0.0.5:80", "http://h:99999", "http://10.0.0.5\nx", "http://ba d"],
+)
+def test_resolution_reports_instead_of_raising(url):
+    target = resolve_acme_backend_target(url)
+    assert target.error_code, "unusable values must be reported, not raised"
+    assert target.error_message
+
+
+def test_resolution_warns_on_loopback_rather_than_refusing():
+    # Refusing here would make every existing install unappliable: the shipped
+    # defaults ARE loopback, and the failure would block changes unrelated to ACME.
+    target = resolve_acme_backend_target("http://localhost:8080")
+    assert target.error_code is None
+    assert target.warnings and "Loopback" in target.warnings[0]
+
+
+def test_resolution_warns_when_the_port_is_omitted():
+    target = resolve_acme_backend_target("http://10.0.0.5")
+    assert target.port == 8080
+    assert any("port" in w.lower() for w in target.warnings)
+
+
+# ---------------------------------------------------------------------------
+# 3. Change detection — what makes a panel edit actually reach the nodes.
+# ---------------------------------------------------------------------------
+
+
+_CONFIG = """global
+    daemon
+
+frontend fe_http
+    bind 10.90.1.100:80
+    mode http
+    acl is_acme_challenge path_beg /.well-known/acme-challenge/
+    use_backend _acme_challenge_backend if is_acme_challenge
+    default_backend app
+
+backend app
+    server s1 10.0.0.9:8080
+
+# ACME Challenge Backend (auto-managed by HAProxy OpenManager)
+backend _acme_challenge_backend
+    mode http
+    server _acme_mgmt 10.90.1.4:80
+"""
+
+
+def test_extracts_the_challenge_backend_target():
+    assert extract_acme_backend_target(_CONFIG) == "10.90.1.4:80"
+
+
+def test_extracts_target_with_ssl_flag():
+    cfg = _CONFIG.replace("10.90.1.4:80", "m.io:443 ssl verify none")
+    assert extract_acme_backend_target(cfg) == "m.io:443 ssl verify none"
+
+
+def test_ignores_server_lines_in_other_backends():
+    # Comparing the whole config would flag every unrelated pending edit as a change;
+    # this must key on the ACME section alone.
+    cfg = _CONFIG.replace("backend _acme_challenge_backend", "backend something_else")
+    assert extract_acme_backend_target(cfg) is None
+
+
+def test_returns_none_when_the_section_has_no_server_line():
+    cfg = (
+        "backend _acme_challenge_backend\n"
+        "    mode http\n"
+        "    # ACME challenge backend unavailable (loopback)\n"
+    )
+    assert extract_acme_backend_target(cfg) is None
+
+
+@pytest.mark.parametrize("value", ["", None])
+def test_extraction_tolerates_empty_input(value):
+    assert extract_acme_backend_target(value) is None
+
+
+def test_url_change_that_renders_the_same_target_is_not_a_change():
+    # `http://10.90.1.4` and `http://10.90.1.4:8080` are different strings but the
+    # same shipped address; minting a config version for that would put a no-op
+    # pending change in front of the operator.
+    a = resolve_acme_backend_target("http://10.90.1.4")
+    b = resolve_acme_backend_target("http://10.90.1.4:8080")
+    assert (a.host, a.port, a.ssl_flag) == (b.host, b.port, b.ssl_flag)
+
+
+# ---------------------------------------------------------------------------
+# 4. Source selection — an unusable value must not shadow a usable one.
+# ---------------------------------------------------------------------------
+
+
+def test_prefers_the_most_specific_usable_source():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "http://10.0.0.1:80"),
+        ("settings", "http://10.0.0.2:80"),
+        ("env", "http://10.0.0.3:80"),
+    ])
+    assert (source, url, target.host) == ("cluster", "http://10.0.0.1:80", "10.0.0.1")
+    assert skipped == []
+
+
+def test_skips_an_unusable_value_and_uses_the_next_source():
+    # The regression this guards: a scheme-less value left over from the era when the
+    # settings field was free text resolves to nothing. Stopping there would emit a
+    # backend section with no `server` line — `haproxy -c` passes, Apply succeeds, and
+    # every challenge request 503s with no visible cause.
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.90.1.4:8080"),
+        ("settings", ""),
+        ("env", "http://10.90.1.4:8080"),
+    ])
+    assert source == "env"
+    assert target.error_code is None and target.host == "10.90.1.4"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_empty_sources_are_skipped_without_being_reported():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""),
+        ("settings", None),
+        ("env", "http://10.0.0.9:80"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert skipped == []
+
+
+def test_reports_the_last_attempted_value_when_nothing_resolves():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.0.0.1:80"),
+        ("env", "not a url"),
+    ])
+    assert (source, url) == ("env", "not a url")
+    assert target.error_code, "the caller needs an error to render and log"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_all_sources_empty_yields_an_error_not_a_crash():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""), ("settings", ""), ("env", ""),
+    ])
+    assert source == "cluster" and target.error_code == "empty" and skipped == []
+
+
+def test_loopback_is_usable_enough_to_render():
+    # Warned about, never skipped: the shipped defaults are loopback, so treating it as
+    # unusable would make the fallback chain fall off its own end on a stock install.
+    source, _url, target, _skipped = select_acme_backend_source([
+        ("env", "http://localhost:8080"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert target.warnings
+
+
+# ---------------------------------------------------------------------------
+# 5. The generator's failure sentinel must never be mistaken for a config.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "content",
+    [
+        "# Error generating configuration: boom",
+        "# Error: Cluster not found",
+        "",
+        None,
+    ],
+)
+def test_detects_generation_failure_sentinels(content):
+    assert is_config_generation_error(content) is True
+
+
+@pytest.mark.parametrize("content", [_CONFIG, "global\n    daemon\n"])
+def test_real_configs_are_not_flagged(content):
+    assert is_config_generation_error(content) is False

backend/tests/test_acme_diagnostics.py

@@ -104,13 +104,30 @@ async def test_check_dns_empty_ips_marks_failure(monkeypatch):
 
 
 # ----------------------------------------------------------------------------
-# Port-80 check (HEAD probe)
+# Port-80 check (GET probe)
+#
+# The probe is a GET, not a HEAD: a reverse proxy that has lost its
+# /.well-known/acme-challenge/ location falls through to its catch-all and serves
+# an SPA with HTTP 200, which a status-code-only check accepts as healthy while
+# every real validation fails. The fakes below therefore carry a body.
 # ----------------------------------------------------------------------------
 
 
-class _FakeHEADResp:
-    def __init__(self, status):
+class _FakeContent:
+    def __init__(self, body):
+        self._body = body
+
+    async def read(self, n=-1):
+        if self._body is None:
+            raise ConnectionResetError("reset mid-body")
+        return self._body if n is None or n < 0 else self._body[:n]
+
+
+class _FakeGETResp:
+    def __init__(self, status, body=b"", content_type="text/plain"):
         self.status = status
+        self.headers = {"content-type": content_type}
+        self.content = _FakeContent(body)
 
     async def __aenter__(self):
         return self
@@ -120,10 +137,13 @@ class _FakeHEADResp:
 
 
 class _FakeSession:
-    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False):
+    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False,
+                 bodies=None, content_types=None):
         self._statuses = list(statuses or [])
         self._raise_timeout = raise_timeout
         self._raise_client_error = raise_client_error
+        self._bodies = list(bodies or [])
+        self._content_types = list(content_types or [])
 
     async def __aenter__(self):
         return self
@@ -131,14 +151,16 @@ class _FakeSession:
     async def __aexit__(self, *args):
         return False
 
-    def head(self, url, allow_redirects=False):
+    def get(self, url, allow_redirects=False):
         if self._raise_timeout:
             raise asyncio.TimeoutError()
         if self._raise_client_error:
             import aiohttp
             raise aiohttp.ClientError("connection refused")
         status = self._statuses.pop(0) if self._statuses else 200
-        return _FakeHEADResp(status)
+        body = self._bodies.pop(0) if self._bodies else b""
+        ctype = self._content_types.pop(0) if self._content_types else "text/plain"
+        return _FakeGETResp(status, body, ctype)
 
 
 def _mock_public_dns(monkeypatch, ip="93.184.216.34"):
@@ -178,6 +200,65 @@ async def test_check_port80_ok_on_404(monkeypatch):
     assert out["status"] == "ok"
 
 
+@pytest.mark.asyncio
+async def test_check_port80_warns_when_200_carries_a_web_page(monkeypatch):
+    """The failure that motivated the GET probe.
+
+    A reverse proxy whose /.well-known/acme-challenge/ location has drifted away
+    falls through to its catch-all and serves the SPA. The status is 200, so the old
+    `status in (200, 404)` rule called the install healthy while every validation
+    failed. Only the body distinguishes them.
+    """
+    _mock_public_dns(monkeypatch)
+    spa = b'<!doctype html><html><head><title>HAProxy OpenManager</title></head>'
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200], bodies=[spa], content_types=["text/html"])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    target = out["details"]["targets"][0]
+    assert target["body_class"] == "html"
+    assert not target.get("ok")
+    assert "HTML" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_port80_falls_back_to_status_when_the_body_cannot_be_read(monkeypatch):
+    """A body that cannot be read is missing evidence, not a verdict.
+
+    Turning a connection reset mid-response into a hard failure would make a healthy
+    404 fail intermittently, so the check keeps its original status-only semantics
+    whenever there is nothing to judge.
+    """
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[404], bodies=[None])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "ok"
+    assert out["details"]["targets"][0]["body_class"] == "unread"
+
+
+@pytest.mark.asyncio
+async def test_check_port80_warns_on_a_redirect(monkeypatch):
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[301])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    assert out["details"]["targets"][0]["diagnosis"]
+
+
 @pytest.mark.asyncio
 async def test_check_port80_warn_on_egress_timeout(monkeypatch):
     """Corporate egress blocks port 80 outbound — warn, don't fail."""
@@ -337,16 +418,92 @@ async def test_check_routing_fail_when_no_port80_frontend():
     assert "No HTTP frontend" in out["message"]
 
 
+def _routing_row(**over):
+    row = {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
+           "mode": "http", "default_backend": "be", "cluster_id": 1, "acme_enabled": True}
+    row.update(over)
+    return row
+
+
+def _applied(has_route=True, server_line="server _acme_mgmt 10.90.1.4:80"):
+    return {"has_route": has_route, "server_line": server_line}
+
+
 @pytest.mark.asyncio
-async def test_check_routing_ok_when_port80_frontend_present():
+async def test_check_routing_ok_when_challenge_route_is_in_the_applied_config():
+    # A port-80 frontend row alone is NOT enough. It describes what the database
+    # wants; the nodes run whatever was last applied. During the incident this
+    # function reported "ok" from the row count while the live config had no usable
+    # challenge route at all.
     conn = AsyncMock()
-    conn.fetch.return_value = [
-        {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
-         "mode": "http", "default_backend": "be"},
-    ]
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied()
     out = await check_routing(conn, ["a.example.com"], [1])
     assert out["status"] == "ok"
     assert len(out["details"]["frontends"]) == 1
+    assert out["details"]["challenge_backends"] == {1: "10.90.1.4:80"}
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_acme_is_disabled_on_the_cluster():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(acme_enabled=False)]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "disabled" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_route_is_not_in_the_applied_config():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(has_route=False)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "apply the cluster" in out["message"].lower()
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_has_no_server_line():
+    # `haproxy -c` passes because the section exists, so nothing else catches this;
+    # every challenge request 503s from an empty backend.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line=None)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "503" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_is_loopback():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line="server _acme_mgmt 127.0.0.1:8080")
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "HAProxy node" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_rather_than_fails_on_a_tcp_only_port80_cluster():
+    # The `fail` branch must stay reachable only when NO port-80 frontend exists at
+    # all: SiteWizard blocks submit on any failing check, so turning this into a
+    # failure would lock tcp-only installs the day it ships.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode="tcp")]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "tcp mode" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_treats_null_mode_as_http_like_the_renderer():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode=None)]
+    conn.fetchrow.return_value = _applied()
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "ok"
 
 
 # ----------------------------------------------------------------------------
@@ -549,3 +706,180 @@ async def test_run_checks_unknown_only_returns_empty():
         account_id=None, only=["bogus"],
     )
     assert out == []
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #94 / #95 (Round-25 audit) — diagnostic-runner robustness
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_swallows_single_check_crash(monkeypatch):
+    """Bulgu #94 — a single check exception must NOT collapse the suite.
+
+    Pre-fix, an asyncpg UndefinedColumnError from check_agents (e.g. the
+    Bulgu #84 ``a.last_heartbeat`` typo on an older deploy) propagated
+    up to the FastAPI router which had no `except`, so the operator saw
+    HTTP 500 with no body. The diagnostic panel is precisely the place
+    that should SURFACE this — never opaque-500 it. We now wrap each
+    check; the failing one becomes a structured `fail` row and the
+    other four still render.
+    """
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+
+    # check_routing + check_agents both call conn.fetch; explode on the
+    # FIRST call (which is check_routing) and return rows on the second.
+    call_count = {"n": 0}
+
+    async def _fetch(*args, **kwargs):
+        call_count["n"] += 1
+        if call_count["n"] == 1:
+            raise RuntimeError("simulated: column a.last_heartbeat does not exist")
+        return []
+
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # All five checks must still appear in the response.
+    ids = [c["id"] for c in out]
+    assert ids == ["dns", "port80", "routing", "account", "agents"]
+    routing = next(c for c in out if c["id"] == "routing")
+    assert routing["status"] == "fail"
+    assert "Diagnostic check crashed" in routing["message"]
+    assert routing["details"]["exception_type"] == "RuntimeError"
+    assert "last_heartbeat" in routing["details"]["exception_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_coerces_string_cluster_ids(monkeypatch):
+    """Bulgu #94 — cluster_ids stored as JSONB strings (legacy paths)
+    must not crash check_routing / check_agents with
+    ``invalid input syntax for type integer: "1"``."""
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    captured_args = []
+
+    async def _fetch(*args, **kwargs):
+        captured_args.append(args)
+        return []
+
+    conn = AsyncMock()
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=["1", "2", "garbage", None, 3],
+        account_id=None,
+    )
+    # The list passed to asyncpg should already be a pure-int list.
+    # check_routing is the first call that uses cluster_ids.
+    routing_call_args = [a for a in captured_args if "frontends" in a[0]]
+    assert routing_call_args, "check_routing should have queried frontends"
+    cluster_ids_arg = routing_call_args[0][1]
+    assert cluster_ids_arg == [1, 2, 3], (
+        f"cluster_ids must be coerced to ints before being passed to "
+        f"asyncpg's ::int[] cast; got: {cluster_ids_arg!r}"
+    )
+    assert all(c["status"] != "fail" or c["id"] != "routing"
+               for c in out
+               if c["id"] == "routing" and "Diagnostic check crashed" in (c.get("message") or "")
+              ), "routing should not have crashed on coerced cluster_ids"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_does_not_swallow_cancellation(monkeypatch):
+    """`_safe_check` must catch `Exception` but NOT `BaseException`.
+
+    asyncio.CancelledError is a BaseException (Python 3.8+) so it must
+    propagate out of `_safe_check` — otherwise a request that the
+    client cancelled mid-flight would silently keep running diagnostic
+    checks instead of unwinding cleanly. We hand `_safe_check` a coro
+    that raises CancelledError and assert it bubbles up.
+    """
+    from services.acme_diagnostics import _safe_check
+
+    async def _cancelled_coro():
+        raise asyncio.CancelledError()
+
+    with pytest.raises(asyncio.CancelledError):
+        await _safe_check("dns", "DNS resolution", _cancelled_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_handles_keyboardinterrupt(monkeypatch):
+    """`_safe_check` must also not swallow `KeyboardInterrupt`."""
+    from services.acme_diagnostics import _safe_check
+
+    async def _interrupt_coro():
+        raise KeyboardInterrupt()
+
+    with pytest.raises(KeyboardInterrupt):
+        await _safe_check("dns", "DNS resolution", _interrupt_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_coerce_cluster_ids_handles_none():
+    """Coercion must handle None input without raising."""
+    from services.acme_diagnostics import _coerce_cluster_ids
+    assert _coerce_cluster_ids(None) == []
+    assert _coerce_cluster_ids([]) == []
+    assert _coerce_cluster_ids([1, 2, 3]) == [1, 2, 3]
+    assert _coerce_cluster_ids(["1", "2"]) == [1, 2]
+    assert _coerce_cluster_ids([1.5]) == [1]  # int() truncates floats
+    assert _coerce_cluster_ids(["abc", None, "5"]) == [5]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_filters_invalid_domains(monkeypatch):
+    """Non-string entries in `domains` must not reach the DNS resolver."""
+    seen_domains = []
+
+    def fake_gethostbyname_ex(domain):
+        seen_domains.append(domain)
+        return (domain, [], ["10.0.0.1"])
+
+    monkeypatch.setattr(socket, "gethostbyname_ex", fake_gethostbyname_ex)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+    conn.fetch.return_value = []
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com", None, "", 42, "b.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # Both check_dns and check_port80 resolve DNS, so each valid domain
+    # may appear multiple times — but invalid entries (None, "", 42)
+    # must never reach the resolver.
+    assert set(seen_domains) == {"a.example.com", "b.example.com"}
+    assert None not in seen_domains
+    assert "" not in seen_domains
+    assert 42 not in seen_domains
+    dns_check = next(c for c in out if c["id"] == "dns")
+    assert dns_check["status"] == "ok"

backend/tests/test_acme_diagnostics_router_round25.py

@@ -0,0 +1,554 @@
+"""Router-level tests for the ACME diagnostic endpoints (Round-25 audit).
+
+These tests pin the contract introduced by Bulgu #94 / #95:
+
+* ``POST /api/letsencrypt/orders/{order_id}/diagnostics`` must NEVER return
+  HTTP 500 for an in-suite failure. Authentication / authorisation /
+  rate-limit / not-found errors still raise the appropriate 4xx, but any
+  unexpected exception during check execution is converted to HTTP 200
+  with a structured failure envelope so the UI can render the cause.
+
+* ``GET /api/letsencrypt/orders/{order_id}/events`` must NEVER return
+  HTTP 500 because of schema drift in ``user_activity_logs`` (the
+  original 500 cause: SELECTing a non-existent ``status`` column). A
+  partial failure is reported via ``meta.errors[]``.
+
+The tests use AsyncMock-based fake connections rather than spinning up a
+real Postgres so they run hermetically inside CI.
+"""
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from routers import acme_diagnostics as router_mod
+
+
+# ----------------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------------
+
+
+class _FakeUser(dict):
+    pass
+
+
+def _patch_auth_and_db(monkeypatch, conn, user_id=1):
+    """Patch the auth / db helpers used by both endpoints so the tests
+    don't have to construct a real FastAPI request stack."""
+    async def _fake_user(_auth):
+        return _FakeUser(id=user_id, username="t", email="t@x")
+
+    async def _fake_perm(_uid, *_a, **_k):
+        return True
+
+    async def _fake_get_conn():
+        return conn
+
+    async def _fake_close_conn(_c):
+        return None
+
+    async def _fake_rate_limit(*_a, **_kw):
+        return None
+
+    monkeypatch.setattr(router_mod, "get_current_user_from_token", _fake_user)
+    monkeypatch.setattr(router_mod, "check_user_permission", _fake_perm)
+    monkeypatch.setattr(router_mod, "get_database_connection", _fake_get_conn)
+    monkeypatch.setattr(router_mod, "close_database_connection", _fake_close_conn)
+    monkeypatch.setattr(router_mod, "_enforce_rate_limit", _fake_rate_limit)
+
+
+# ----------------------------------------------------------------------------
+# /diagnostics endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_load_order_crash_returns_envelope_not_500(monkeypatch):
+    """A DB crash during load_order must surface a 200 envelope, not 500.
+
+    Pre-fix, any RuntimeError between auth and run_checks bubbled out of
+    the bare ``try/finally`` block and FastAPI returned an opaque HTTP 500.
+    The Round-25 fix wraps load_order so the operator sees the cause
+    inside the diagnostic panel.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB connectivity loss")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.run_diagnostics(order_id=42, authorization="Bearer x")
+
+    assert out["order_id"] == 42
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["checks"][0]["status"] == "fail"
+    assert out["checks"][0]["id"] == "diagnostics_runner"
+    assert "simulated DB connectivity loss" in out["checks"][0]["message"]
+    assert out["meta"]["correlation_id"]
+    assert out["meta"]["error_stage"] == "load_order"
+    assert "humanized_error" in out
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_run_checks_crash_returns_envelope(monkeypatch):
+    """A crash inside run_checks (after order is loaded) must also stay 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated check orchestrator crash")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["meta"]["error_stage"] == "run_checks"
+    assert out["meta"]["error_type"] == "RuntimeError"
+    assert "simulated check orchestrator crash" in out["meta"]["error_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_returns_meta_summary_on_success(monkeypatch):
+    """Successful diagnostics responses carry a meta summary the UI uses
+    to surface 'N checks failed, M warnings' without recomputing."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _fake_checks(*_a, **_kw):
+        return [
+            {"id": "dns", "label": "DNS", "status": "ok", "severity": "info", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "routing", "label": "Routing", "status": "fail", "severity": "error", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "port80", "label": "Port 80", "status": "warn", "severity": "warn", "message": "", "details": {}, "duration_ms": 1},
+        ]
+
+    monkeypatch.setattr(router_mod, "run_checks", _fake_checks)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["meta"]["checks_total"] == 3
+    assert out["meta"]["checks_failed"] == 1
+    assert out["meta"]["checks_warn"] == 1
+    assert out["meta"]["correlation_id"]
+
+
+# ----------------------------------------------------------------------------
+# /events endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_missing_status_column_returns_partial_envelope(monkeypatch):
+    """Bulgu #95 — user_activity_logs lacks a `status` column.
+
+    Pre-fix, the SELECT pulled `status` directly and the endpoint
+    returned HTTP 500 for every order that had any correlated
+    user-activity rows. The Round-25 fix introspects the schema; here
+    we simulate a deployment with no `status` column AND a JOIN /
+    query that would otherwise crash — the endpoint must stay 200,
+    return whatever it could collect from acme_order_events, and
+    record the user_activity_logs section as degraded but recoverable.
+    """
+    conn = AsyncMock()
+
+    # _load_order
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    fetchval_calls = {"n": 0}
+
+    async def _fetchval(sql, *args):
+        fetchval_calls["n"] += 1
+        # First call: existence check for acme_order_events table
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    columns_no_status = [
+        {"column_name": "id"},
+        {"column_name": "user_id"},
+        {"column_name": "action"},
+        {"column_name": "resource_type"},
+        {"column_name": "resource_id"},
+        {"column_name": "details"},
+        {"column_name": "created_at"},
+        # NOTE: no "status" column — this is the canonical schema.
+    ]
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql and "user_activity_logs" in sql:
+            return columns_no_status
+        if "FROM acme_order_events" in sql:
+            return []  # empty timeline is fine for this test
+        if "FROM user_activity_logs" in sql:
+            # If the projection includes `status` we will fail loudly.
+            assert "status" not in sql, (
+                "SELECT must not include `status` when the column is absent; "
+                f"SQL was: {sql!r}"
+            )
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["count"] == 0
+    assert out["meta"]["correlation_id"]
+    # No section errors expected — schema-aware projection silently
+    # adapted, the panel just got an empty event list.
+    assert out["meta"]["errors"] == []
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_acme_order_events_query_crash_returns_partial(monkeypatch):
+    """A crash in the acme_order_events sub-query must NOT kill the
+    whole endpoint — the user_activity_logs section should still run
+    and the failure must appear in meta.errors."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+            ]
+        if "FROM acme_order_events" in sql:
+            raise RuntimeError("simulated acme_order_events index corruption")
+        if "FROM user_activity_logs" in sql:
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["count"] == 0
+    error_sections = [e["section"] for e in out["meta"]["errors"]]
+    assert "acme_order_events" in error_sections
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_load_order_404_still_raises(monkeypatch):
+    """The 404 HTTPException raised by `_load_order` for an unknown order
+    must remain a 404 — the Round-25 envelope is only for *unexpected*
+    failures, not for client-supplied invalid order IDs."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    conn.fetchrow.return_value = None  # no order found
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.get_order_events(order_id=9999, authorization="Bearer x")
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_setup_crash_returns_check_envelope(monkeypatch):
+    """The single-check re-run path must also envelope, never 500.
+
+    Pre-fix the rerun handler used the same bare ``try/finally`` shape
+    as the suite POST. If `_load_order` / `_enforce_rate_limit` raised,
+    the operator clicking the row's "Re-run" button got an opaque
+    toast and the row never updated. Now the rerun handler returns a
+    `fail` check shaped the same way the table already renders, so
+    the row updates in place with the cause + correlation_id.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB drop during rerun")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="dns", authorization="Bearer x",
+    )
+    assert out["order_id"] == 5
+    assert out["check"]["status"] == "fail"
+    assert out["check"]["id"] == "dns"
+    assert "simulated DB drop during rerun" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "setup"
+    assert out["meta"]["correlation_id"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_run_checks_crash_returns_check_envelope(monkeypatch):
+    """A crash inside run_checks during a re-run also stays 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated run_checks failure")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="agents", authorization="Bearer x",
+    )
+    assert out["check"]["id"] == "agents"
+    assert out["check"]["status"] == "fail"
+    assert "simulated run_checks failure" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "run_checks"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_invalid_check_id_still_400(monkeypatch):
+    """An unknown check_id must remain a 400, not an envelope. The
+    envelope is only for *unexpected* server-side failures, not for
+    client-supplied invalid identifiers."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5, check_id="bogus", authorization="Bearer x",
+        )
+    assert exc_info.value.status_code == 400
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_status_column_is_used_when_present(monkeypatch):
+    """If a deployment DID add a `status` column (e.g. via a private
+    schema extension), the projection picks it up and the resulting
+    severity reflects it."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    captured_sql = {"ua": None}
+    from datetime import datetime, timezone
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+                {"column_name": "status"},
+            ]
+        if "FROM acme_order_events" in sql:
+            return []
+        if "FROM user_activity_logs" in sql:
+            captured_sql["ua"] = sql
+            return [{
+                "id": 100,
+                "action": "letsencrypt.order.create",
+                "resource_type": "letsencrypt_order",
+                "resource_id": "5",
+                "details": '{"foo":"bar"}',
+                "created_at": datetime(2026, 5, 13, 19, 0, 0, tzinfo=timezone.utc),
+                "user_id": 7,
+                "status": "failure",
+            }]
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert "status" in captured_sql["ua"]
+    assert out["count"] == 1
+    ev = out["events"][0]
+    assert ev["source"] == "user_activity_log"
+    assert ev["severity"] == "warn"  # status="failure" → warn
+    assert ev["details"] == {"foo": "bar"}
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #96 (prod-canary follow-up): int4 overflow on order_id must NOT
+# leak the asyncpg DataError message ("invalid input for query argument
+# $1: ... value out of int32 range") into the operator-facing response
+# body. Same shape as the "row not found" path: clean HTTPException(404).
+# ----------------------------------------------------------------------------
+
+
+import asyncpg as _asyncpg  # noqa: E402 — late import so the test module
+# can still be collected even if asyncpg has changed its exception module.
+
+
+def _data_error(msg: str) -> _asyncpg.exceptions.DataError:
+    """Construct an asyncpg DataError that mirrors what Postgres returns
+    when a path-param order_id overflows int4. We can't easily build the
+    real instance from the binary protocol, so we synthesize one with the
+    same class so the router's `except asyncpg.exceptions.DataError`
+    branch is exercised."""
+    return _asyncpg.exceptions.DataError(msg)
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_diagnostics_int4_overflow_returns_clean_404(monkeypatch):
+    """``order_id`` outside the int4 range must surface as a clean 404,
+    NOT as a `diagnostics_unavailable` envelope leaking the asyncpg
+    DataError message ("query argument $1", "int32 range").
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 2147483648 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(
+            order_id=2_147_483_648,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    # The operator must see the canonical "not found" detail, NOT the
+    # raw asyncpg error message.
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+    assert "query argument" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_events_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the events endpoint — out-of-range order_id is a
+    clean 404, not a `meta.errors[]` envelope leaking SQL detail."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 9999999999 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.get_order_events(
+            order_id=9_999_999_999,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_rerun_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the per-check rerun endpoint — out-of-range
+    order_id is a clean 404, not a structured ``check.fail`` envelope
+    leaking the asyncpg DataError message."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 5000000000 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5_000_000_000,
+            check_id="dns",
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_load_order_dataerror_does_not_leak_correlation_envelope(monkeypatch):
+    """Belt-and-braces: even when the DataError carries other kinds of
+    invalid-input strings (e.g. type coercion failure on an int4
+    column), `_load_order` must still answer with the canonical 404
+    envelope and NOT route it through `_diagnostic_failure_envelope`
+    (which would surface the raw SQL detail to the UI)."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error("invalid integer literal: 'NaN'")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(order_id=123, authorization="Bearer x")
+
+    assert excinfo.value.status_code == 404
+    # No SQL detail leak
+    assert "invalid integer literal" not in str(excinfo.value.detail).lower()
+    assert "NaN" not in str(excinfo.value.detail)

backend/tests/test_acme_humanizer.py

@@ -81,6 +81,7 @@ def test_legacy_plain_string_with_brace_but_invalid_json_falls_back():
     ("urn:ietf:params:acme:error:rejectedIdentifier", "blacklisted",    "rejected"),
     ("urn:ietf:params:acme:error:serverInternal", "internal err",       "ACME server"),
     ("urn:ietf:params:acme:error:userActionRequired", "agree to ToS",   "User action"),
+    ("urn:ietf:params:acme:error:externalAccountRequired", "EAB required", "External Account Binding"),
 ])
 def test_known_problem_types_are_humanized(problem_type, detail_text, expected_title_contains):
     payload = json.dumps({"type": problem_type, "detail": detail_text, "status": 400})

backend/tests/test_acme_pydantic_validation.py

@@ -8,7 +8,31 @@ from pydantic import ValidationError
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from routers.letsencrypt import CertificateRequest
+from routers.letsencrypt import CertificateRequest, AccountCreate
+
+
+class TestAccountCreateEAB:
+    """Issue #35 follow-up: EAB HMAC key must be valid base64url; empty/None passes through
+    (falls back to global Settings) so non-EAB accounts (HTTP-01 / LE / Cloudflare) are unaffected."""
+
+    def test_no_eab_is_allowed(self):
+        acc = AccountCreate(email="a@b.com")
+        assert acc.eab_hmac_key is None and acc.eab_kid is None
+
+    def test_valid_base64url_hmac_accepted(self):
+        # urlsafe base64, unpadded and padded — both accepted.
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZGVmZ2g")
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZA==")
+
+    def test_invalid_base64_hmac_rejected(self):
+        # 5 base64 chars (count ≡ 1 mod 4) is undecodable — the exact shape that would otherwise
+        # make register_account's _b64url_decode raise a cryptic binascii error.
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="AAAAA")
+
+    def test_oversized_hmac_rejected(self):
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="A" * 600)
 
 
 class TestCertificateRequestDomains:

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_csr_import.py

@@ -0,0 +1,469 @@
+"""
+v1.9.0 CSR creation — unit tests for the signed-certificate import flow and
+config-version staging (pattern: test_ssl_service_extraction.py, AsyncMock conn).
+
+Pins the security-relevant invariants:
+- key match is a HARD gate: match=False → 400 before any INSERT, and
+  match=None (unverifiable) → 500, never a lenient pass (we generated the
+  key ourselves — deliberate divergence from create_cert_row's fallback).
+- the new cert row is cluster_id=NULL / last_config_status='PENDING' /
+  source='csr' (PENDING keeps it invisible to agents until Apply).
+- completing the CSR NULLs the private key copy.
+- staging reuses the exact `ssl-{id}-create-{ts}` version-name scheme.
+"""
+import json
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from fastapi import HTTPException
+
+from models.csr import SSLCSRImport
+from services.csr_service import (
+    assert_csr_name_available,
+    import_signed_certificate,
+    insert_csr_row,
+)
+from services.ssl_service import stage_ssl_config_versions
+
+
+_VALID_PARSE = {
+    "primary_domain": "www.example.com",
+    "all_domains": ["www.example.com"],
+    "expiry_date": datetime(2099, 1, 1, tzinfo=timezone.utc),
+    "issuer": "CN=Test CA",
+    "fingerprint": "AA:BB:CC",
+    "status": "valid",
+    "days_until_expiry": 365,
+}
+
+_FAKE_CERT = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+_FAKE_KEY = "-----BEGIN PRIVATE KEY-----\nY\n-----END PRIVATE KEY-----"
+
+
+def _csr_row(**overrides):
+    row = {
+        "id": 5,
+        "name": "csr-www",
+        "common_name": "www.example.com",
+        "subject": "{}",
+        "sans": json.dumps(["www.example.com"]),
+        "key_algorithm": "rsa-2048",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nZ\n-----END CERTIFICATE REQUEST-----",
+        "private_key_pem": _FAKE_KEY,
+        "status": "pending",
+        "ssl_certificate_id": None,
+    }
+    row.update(overrides)
+    return row
+
+
+def _mk_conn():
+    conn = AsyncMock()
+    # asyncpg's conn.transaction() is a SYNC call returning an async CM.
+    conn.transaction = MagicMock()
+    return conn
+
+
+def _import_payload(**overrides):
+    base = dict(
+        certificate_content=_FAKE_CERT,
+        chain_content=None,
+        usage_type="frontend",
+        is_global=False,
+        cluster_ids=[1, 2],
+        name=None,
+    )
+    base.update(overrides)
+    return SSLCSRImport(**base)
+
+
+@contextmanager
+def _patched(match=None, parse=None):
+    """Patch every parser touchpoint of the import path: the function-local
+    imports in csr_service (utils.ssl_parser.*) and the module-level imports
+    in ssl_service._prepare_cert_fields (services.ssl_service.*)."""
+    match_result = match if match is not None else {"match": True}
+    parse_result = dict(parse or _VALID_PARSE)
+    with patch("utils.ssl_parser.verify_certificate_key_match", return_value=match_result), \
+         patch("utils.ssl_parser.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.validate_private_key", return_value=True), \
+         patch("services.ssl_service.validate_certificate_chain", return_value=True):
+        yield
+
+
+# ----------------------------------------------------------------------------
+# import_signed_certificate
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_import_happy_path_inserts_pending_csr_sourced_cert():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]  # FOR UPDATE row, no name clash
+    conn.fetchval.return_value = 42                 # INSERT ... RETURNING id
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert result["reactivated"] is False
+
+    # Concurrency invariants: everything runs inside a transaction and the
+    # CSR row is locked FOR UPDATE (serialises double-import and delete-races).
+    assert conn.transaction.call_count == 1
+    lock_sql = conn.fetchrow.call_args_list[0].args[0]
+    assert "FOR UPDATE" in lock_sql
+
+    insert_sql, *insert_args = conn.fetchval.call_args.args
+    assert "INSERT INTO ssl_certificates" in insert_sql
+    assert "NULL, 'PENDING'" in insert_sql, "cert must stay invisible to agents until Apply"
+    assert "'csr'" in insert_sql, "source column must record the CSR origin"
+    # The stored CSR key — not any request-supplied key — must be persisted.
+    assert _FAKE_KEY in insert_args
+
+    # One junction row per requested cluster.
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert len(junction_calls) == 2
+    assert {c.args[2] for c in junction_calls} == {1, 2}
+
+    # CSR completion must destroy the key copy.
+    completion_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_csrs" in c.args[0]
+    ]
+    assert len(completion_calls) == 1
+    assert "private_key_pem = NULL" in completion_calls[0].args[0]
+    assert "status = 'completed'" in completion_calls[0].args[0]
+    assert completion_calls[0].args[1] == 5   # csr_id
+    assert completion_calls[0].args[2] == 42  # cert_id
+
+
+@pytest.mark.asyncio
+async def test_import_global_creates_zero_junction_rows():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        await import_signed_certificate(
+            conn, 5, _import_payload(is_global=True, cluster_ids=None), user_id=7
+        )
+
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert junction_calls == [], "global cert = zero junction rows (existing convention)"
+
+
+@pytest.mark.asyncio
+async def test_import_key_mismatch_rejected_400_before_any_write():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": False, "reason": "public key mismatch"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "does not match" in exc_info.value.detail
+    assert not conn.fetchval.await_count, "nothing must be inserted on mismatch"
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_unverifiable_key_match_is_hard_error_not_lenient():
+    """match=None means OUR stored key is unreadable — integrity failure,
+    never the lenient pass create_cert_row historically allows."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": None, "reason": "key could not be parsed"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 500
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_expired_certificate_rejected_400():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    expired = dict(_VALID_PARSE)
+    expired["status"] = "expired"
+    expired["days_until_expiry"] = -10
+    with _patched(parse=expired):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "expired" in exc_info.value.detail.lower()
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_malformed_certificate_rejected_400_not_500():
+    """A cert with PEM markers but unparseable content (truncated CA response)
+    is OPERATOR INPUT — it must get the manual flow's 400, not the 500 that
+    the strict key-match branch reserves for a corrupt STORED key."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(parse={"error": "Could not parse certificate"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "Invalid SSL certificate" in exc_info.value.detail
+    assert not conn.fetchval.await_count
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_completed_csr_conflicts_409():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(status="completed", ssl_certificate_id=42)]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 409
+    assert "already completed" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_import_missing_csr_404():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [None]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 999, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_import_active_name_collision_rejected_with_hint():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 9, "is_active": True}]
+
+    with _patched():
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "already exists" in exc_info.value.detail
+    assert "name" in exc_info.value.detail  # points at the override escape hatch
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_name_override_is_used_for_the_cert_row():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        result = await import_signed_certificate(
+            conn, 5, _import_payload(name="renamed-cert"), user_id=7
+        )
+
+    assert result["certificate_name"] == "renamed-cert"
+    _, *insert_args = conn.fetchval.call_args.args
+    assert "renamed-cert" in insert_args
+    # And the collision check must have run against the override, not csr.name.
+    name_lookup = conn.fetchrow.call_args_list[1]
+    assert name_lookup.args[1] == "renamed-cert"
+
+
+@pytest.mark.asyncio
+async def test_import_reactivates_soft_deleted_name_and_warns():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 77, "is_active": False}]
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 77
+    assert result["reactivated"] is True
+    assert any("reactivated" in w for w in result["warnings"])
+    assert not conn.fetchval.await_count, "reactivation must UPDATE, not INSERT"
+    update_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_certificates" in c.args[0]
+    ]
+    assert len(update_calls) == 1
+    update_sql = update_calls[0].args[0]
+    assert "source = 'csr'" in update_sql
+    # The reactivated row must come back to life invisible to agents until
+    # Apply, with the row itself active again.
+    assert "last_config_status = 'PENDING'" in update_sql
+    assert "is_active = TRUE" in update_sql
+    # Old cluster bindings must be wiped before re-binding to the new scope.
+    junction_deletes = [
+        c for c in conn.execute.call_args_list
+        if c.args and "DELETE FROM ssl_certificate_clusters" in c.args[0]
+    ]
+    assert len(junction_deletes) == 1
+    assert junction_deletes[0].args[1] == 77
+    # …and the importer's requested clusters re-bound via the junction.
+    junction_inserts = [
+        c for c in conn.execute.call_args_list
+        if c.args and "INSERT INTO ssl_certificate_clusters" in c.args[0]
+    ]
+    assert {c.args[2] for c in junction_inserts} == {1, 2}
+
+
+@pytest.mark.asyncio
+async def test_import_san_drift_warns_but_succeeds():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [
+        _csr_row(sans=json.dumps(["www.example.com", "api.example.com"])),
+        None,
+    ]
+    conn.fetchval.return_value = 42
+
+    drifted = dict(_VALID_PARSE)
+    drifted["all_domains"] = ["www.example.com", "cdn.example.com"]
+    with _patched(parse=drifted):
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert any("added" in w and "cdn.example.com" in w for w in result["warnings"])
+    assert any("dropped" in w and "api.example.com" in w for w in result["warnings"])
+
+
+# ----------------------------------------------------------------------------
+# insert_csr_row / assert_csr_name_available
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_active_cert_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [11]  # active cert with the name exists
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "certificate" in exc_info.value.detail.lower()
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_pending_csr_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [None, 12]  # no cert, but a pending CSR
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "pending CSR" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_insert_csr_row_translates_unique_violation_to_400():
+    """The uq_ssl_csrs_name_pending partial index closes the create/create
+    race — the loser must get a clean 400, not a 500."""
+    import asyncpg as _asyncpg
+
+    conn = _mk_conn()
+    # availability checks pass, INSERT hits the unique index
+    conn.fetchval.side_effect = [
+        None, None, _asyncpg.exceptions.UniqueViolationError("dup"),
+    ]
+    payload = SimpleNamespace(
+        name="raced", common_name="www.example.com", key_algorithm="rsa-2048"
+    )
+    bundle = {"subject": {}, "sans": ["www.example.com"], "csr_pem": "PEM", "private_key_pem": "KEY"}
+
+    with pytest.raises(HTTPException) as exc_info:
+        await insert_csr_row(conn, payload, bundle, user_id=1)
+    assert exc_info.value.status_code == 400
+    assert "concurrent" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# router-level guards
+# ----------------------------------------------------------------------------
+
+
+def test_cluster_id_int32_guard_rejects_out_of_range_with_404():
+    """Body-supplied cluster ids must never reach asyncpg out of int4 range
+    (DataError → raw 500) — same Bulgu #96 hygiene as the csr_id path param."""
+    from routers.csr import _assert_valid_cluster_id
+
+    _assert_valid_cluster_id(1)
+    _assert_valid_cluster_id(2_147_483_647)
+    for bad in (0, -1, 2_147_483_648, 99_999_999_999):
+        with pytest.raises(HTTPException) as exc_info:
+            _assert_valid_cluster_id(bad)
+        assert exc_info.value.status_code == 404
+        assert "Cluster not found" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# stage_ssl_config_versions
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_stage_creates_one_pending_version_per_cluster_with_ssl_naming():
+    import re
+
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001  # config_versions INSERT RETURNING id
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(return_value="# cfg"),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert all(r["success"] for r in results)
+    assert [r["cluster_id"] for r in results] == [1, 2]
+
+    insert_calls = [
+        c for c in conn.fetchval.call_args_list
+        if c.args and "INSERT INTO config_versions" in c.args[0]
+    ]
+    assert len(insert_calls) == 2
+    for call in insert_calls:
+        sql = call.args[0]
+        assert "FALSE, 'PENDING'" in sql, "staged versions must be inactive + PENDING"
+        version_name = call.args[2]
+        # EXACT manual-flow scheme: Apply Management + has_pending_config
+        # LIKE-filters key off 'ssl-{id}-...'.
+        assert re.match(r"^ssl-42-create-\d+$", version_name), version_name
+        assert call.args[5] == 7  # created_by honours the importing user
+
+
+@pytest.mark.asyncio
+async def test_stage_reports_per_cluster_failure_without_raising():
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001
+
+    async def _gen(cluster_id):
+        if cluster_id == 2:
+            raise RuntimeError("config generation exploded")
+        return "# cfg"
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(side_effect=_gen),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert results[0]["success"] is True
+    assert results[1]["success"] is False
+    assert "exploded" in results[1]["error"]

backend/tests/test_csr_key_encryption.py

@@ -0,0 +1,129 @@
+"""Issue #53 (v1.10.1) — at-rest encryption for the pending CSR private key.
+
+Pure logic: no DB, no network. Covers the round-trip, the backward-compatible read of rows
+written before this release, the unrecoverable-key path after a key rotation, and a static
+assertion that the write path can no longer store a raw PEM.
+"""
+import os
+import re
+from pathlib import Path
+
+import pytest
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-csr-encryption-unit-tests")
+
+from cryptography.fernet import Fernet
+
+from utils.csr_key_crypto import (
+    decrypt_csr_private_key,
+    encrypt_csr_private_key,
+    is_encrypted,
+    reset_fernet_for_tests,
+)
+
+_SAMPLE_PEM = (
+    "-----BEGIN PRIVATE KEY-----\n"
+    "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj\n"
+    "-----END PRIVATE KEY-----\n"
+)
+
+
+def test_roundtrip_and_ciphertext_does_not_contain_the_key():
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+    # The stored form must not be the PEM, and must not leak any recognisable fragment of it.
+    assert token != _SAMPLE_PEM
+    assert "-----BEGIN" not in token
+    assert "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj" not in token
+    assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+
+
+def test_is_encrypted_discriminates_token_from_legacy_pem():
+    reset_fernet_for_tests()
+    assert is_encrypted(encrypt_csr_private_key(_SAMPLE_PEM)) is True
+    assert is_encrypted(_SAMPLE_PEM) is False
+    assert is_encrypted("") is False
+    assert is_encrypted(None) is False
+
+
+def test_legacy_plaintext_row_is_read_unchanged():
+    # Rows written before v1.10.1 hold a raw PEM. They must keep working with NO data migration,
+    # otherwise upgrading would strand every CSR that is out for signature.
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(_SAMPLE_PEM) == _SAMPLE_PEM
+
+
+def test_empty_or_missing_value_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(None) is None
+    assert decrypt_csr_private_key("") is None
+
+
+def test_key_rotation_makes_the_stored_key_unrecoverable_rather_than_wrong():
+    """After a rotation the caller must get None, never a silently wrong key."""
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+
+    # Rotate: an explicit, different CSR_ENCRYPTION_KEY takes precedence over the derived one.
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    os.environ["CSR_ENCRYPTION_KEY"] = Fernet.generate_key().decode()
+    try:
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) is None
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_explicit_env_key_is_used_and_survives_reset():
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    key = Fernet.generate_key().decode()
+    os.environ["CSR_ENCRYPTION_KEY"] = key
+    try:
+        reset_fernet_for_tests()
+        token = encrypt_csr_private_key(_SAMPLE_PEM)
+        # Decryptable with the same explicit key from a fresh instance...
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+        # ...and independently verifiable with the raw Fernet key.
+        assert Fernet(key.encode()).decrypt(token.encode()).decode() == _SAMPLE_PEM
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_derivation_uses_its_own_hkdf_info_string():
+    """Each secret class derives an independent key, so rotating one never affects another."""
+    src = (Path(__file__).resolve().parent.parent / "utils" / "csr_key_crypto.py").read_text()
+    assert b"csr-private-key-v1".decode() in src
+    # Must NOT reuse another class's info string.
+    for foreign in ("dns-provider-creds-v1", "vip-vrrp-secret-v1", "mfa-totp-secret-v1"):
+        assert foreign not in src, f"CSR key derivation must not reuse the {foreign} info string"
+
+
+def test_write_path_stores_the_encrypted_form_not_the_pem():
+    """Static pin: insert_csr_row must encrypt before the INSERT.
+
+    A future refactor that passed bundle['private_key_pem'] straight through would silently
+    reintroduce plaintext storage, and no unit test with a mocked connection would notice.
+    """
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    insert_fn = src[src.index("async def insert_csr_row("):]
+    insert_fn = insert_fn[: insert_fn.index("\nasync def ")]
+    assert "encrypt_csr_private_key(bundle['private_key_pem'])" in insert_fn
+    # The raw PEM must not be a bind parameter of the INSERT itself.
+    assert not re.search(r"^\s*bundle\['private_key_pem'\],\s*$", insert_fn, re.M)
+
+
+def test_import_path_decrypts_and_fails_closed_on_unrecoverable_key():
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    fn = src[src.index("async def import_signed_certificate("):]
+    assert "decrypt_csr_private_key(row['private_key_pem'])" in fn
+    # A None decrypt must raise rather than fall through to the key-match comparison.
+    assert "cannot be decrypted" in fn

backend/tests/test_csr_migration.py

@@ -0,0 +1,104 @@
+"""
+v1.9.0 CSR creation — static source assertions (pattern: test_vip_purge.py).
+
+Guards the migration wiring that a unit test cannot exercise without a real
+database: the SCHEMA_VERSION bump (without it, deployed installs skip the
+whole migration run and the ssl_csrs table never appears), the migration
+registration, the security-relevant DDL, and the router registration.
+"""
+import os
+import re
+
+_BACKEND_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel_path: str) -> str:
+    with open(os.path.join(_BACKEND_DIR, rel_path), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_schema_version_bumped_to_10():
+    src = _read(os.path.join("database", "migrations.py"))
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert m, "SCHEMA_VERSION constant not found in migrations.py"
+    assert int(m.group(1)) >= 10, (
+        "SCHEMA_VERSION must be >= 10 for the v1.9.0 ssl_csrs table — "
+        "without the bump, existing installs (version >= 9) skip the whole "
+        "migration run and never gain the table."
+    )
+
+
+def test_ssl_csrs_migration_defined_and_registered():
+    src = _read(os.path.join("database", "migrations.py"))
+    assert "async def ensure_ssl_csrs_table" in src
+
+    inner = src.split("async def _run_all_migrations_inner", 1)[1]
+    inner = inner.split("\nasync def ", 1)[0]  # body of the runner only
+    assert "await ensure_ssl_csrs_table()" in inner, (
+        "ensure_ssl_csrs_table must be invoked from _run_all_migrations_inner"
+    )
+
+
+def test_ssl_csrs_ddl_essentials():
+    src = _read(os.path.join("database", "migrations.py"))
+    ddl_start = src.index("CREATE TABLE IF NOT EXISTS ssl_csrs")
+    ddl = src[ddl_start:ddl_start + 2500]
+
+    assert "private_key_pem TEXT" in ddl
+    assert "name VARCHAR(100) NOT NULL" in ddl, (
+        "ssl_csrs.name must align with ssl_certificates.name VARCHAR(100)"
+    )
+    assert "ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL" in ddl, (
+        "deleting the imported cert must not cascade into CSR history"
+    )
+    # Partial unique index: only PENDING CSRs reserve their target cert name.
+    assert "uq_ssl_csrs_name_pending" in src
+    assert re.search(
+        r"uq_ssl_csrs_name_pending\s+ON\s+ssl_csrs\(name\)\s+WHERE\s+status\s*=\s*'pending'",
+        src,
+    ), "name uniqueness must be scoped to pending CSRs (partial index)"
+
+
+def test_csr_router_registered_in_main():
+    src = _read("main.py")
+    assert "from routers.csr import router as csr_router" in src
+    assert "app.include_router(csr_router)" in src
+
+
+def test_csr_endpoint_permission_mapping():
+    """Pin which ssl.<action> permission each endpoint enforces: a regression
+    that dropped or weakened a _require() call would otherwise pass the
+    auth-rejection tests (they only assert 401/403 for unauthenticated calls)."""
+    src = _read(os.path.join("routers", "csr.py"))
+
+    def _handler_body(decorator):
+        start = src.index(decorator)
+        nxt = src.find("@router.", start + 1)
+        return src[start:nxt if nxt != -1 else len(src)]
+
+    expectations = [
+        ('@router.post("")', '"create"'),
+        ('@router.get("")', '"read"'),
+        ('@router.get("/{csr_id}")', '"read"'),
+        ('@router.post("/{csr_id}/import")', '"create"'),
+        ('@router.delete("/{csr_id}")', '"delete"'),
+    ]
+    for decorator, action in expectations:
+        body = _handler_body(decorator)
+        assert f"_require(authorization, {action})" in body, (
+            f"endpoint {decorator} must enforce ssl.{action.strip(chr(34))}"
+        )
+
+
+def test_csr_router_never_selects_private_key():
+    """The CSR endpoints must use the explicit column list — a bare
+    `SELECT *` into an API response is how the key would leak. The one place
+    SELECT * is allowed is the service-layer FOR UPDATE row (it needs the key
+    to pair with the cert); the router itself must not touch the column."""
+    src = _read(os.path.join("routers", "csr.py"))
+    code_only = re.sub(r"#.*", "", src)  # strip comments; the column name may
+    # legitimately appear there as documentation
+    assert "private_key_pem" not in code_only, (
+        "routers/csr.py must never reference private_key_pem in code"
+    )
+    assert "SELECT *" not in code_only, "routers/csr.py must use explicit column lists"

backend/tests/test_csr_models.py

@@ -0,0 +1,172 @@
+"""
+v1.9.0 CSR creation — Pydantic model validation tests (models/csr.py).
+
+The CSR name shares the SSL certificate name's path-traversal contract
+(Bulgu #21) with one deliberate tightening: max 100 chars, matching the
+ssl_certificates.name VARCHAR(100) column.
+"""
+import pytest
+from pydantic import ValidationError
+
+from models.csr import SSLCSRCreate, SSLCSRImport
+
+_CERT_PEM = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+
+
+def _create(**overrides):
+    base = dict(name="my-csr", common_name="www.example.com")
+    base.update(overrides)
+    return SSLCSRCreate(**base)
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRCreate
+# ----------------------------------------------------------------------------
+
+
+def test_minimal_valid_create():
+    m = _create()
+    assert m.name == "my-csr"
+    assert m.common_name == "www.example.com"
+    assert m.key_algorithm == "rsa-2048"
+    assert m.sans == []
+
+
+@pytest.mark.parametrize("bad_name", [
+    "../../etc/cron.d/evil",   # path traversal
+    "a..b",                    # embedded ..
+    ".hidden",                 # hidden filename
+    "-flag",                   # CLI flag confusion
+    "has space",
+    "wild*card",
+    "",
+    "x" * 101,                 # VARCHAR(100) alignment — 200 is NOT allowed here
+])
+def test_name_rejects_unsafe_values(bad_name):
+    with pytest.raises(ValidationError):
+        _create(name=bad_name)
+
+
+def test_name_accepts_100_chars():
+    assert _create(name="x" * 100).name == "x" * 100
+
+
+def test_common_name_wildcard_accepted_and_lowercased():
+    m = _create(common_name="*.Example.COM")
+    assert m.common_name == "*.example.com"
+
+
+@pytest.mark.parametrize("bad_cn", [
+    "",
+    "under_score.example.com",      # _ is not LDH
+    "*.*.example.com",              # wildcard only as leftmost single label
+    "-leading.example.com",
+    "a" * 70 + ".example.com",      # label > 63
+    "cn-longer-than-64-chars-" + "x" * 45 + ".example.com",  # CN > 64 total
+])
+def test_common_name_rejects_invalid(bad_cn):
+    with pytest.raises(ValidationError):
+        _create(common_name=bad_cn)
+
+
+def test_sans_normalised_deduped_and_capped():
+    m = _create(sans=["API.example.com", "api.example.com", "cdn.example.com"])
+    assert m.sans == ["api.example.com", "cdn.example.com"]
+
+    with pytest.raises(ValidationError):
+        _create(sans=[f"h{i}.example.com" for i in range(101)])
+
+
+def test_country_normalised_or_rejected():
+    assert _create(country="tr").country == "TR"
+    assert _create(country=None).country is None
+    for bad in ("TUR", "T", "1A"):
+        with pytest.raises(ValidationError):
+            _create(country=bad)
+
+
+def test_subject_fields_reject_control_characters():
+    with pytest.raises(ValidationError):
+        _create(organization="Evil\x00Corp")
+    with pytest.raises(ValidationError):
+        _create(locality="line\nbreak")
+
+
+def test_subject_fields_reject_overlength():
+    with pytest.raises(ValidationError):
+        _create(organization="x" * 65)
+
+
+def test_key_algorithm_strict_enum():
+    for good in ("rsa-2048", "rsa-4096", "ecdsa-p256", "ecdsa-p384"):
+        assert _create(key_algorithm=good).key_algorithm == good
+    for bad in ("rsa-1024", "rsa-8192", "ed25519", "2048", ""):
+        with pytest.raises(ValidationError):
+            _create(key_algorithm=bad)
+
+
+def test_email_basic_validation():
+    assert _create(email="ops@example.com").email == "ops@example.com"
+    with pytest.raises(ValidationError):
+        _create(email="not-an-email")
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRImport
+# ----------------------------------------------------------------------------
+
+
+def test_import_minimal_global():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True)
+    assert m.usage_type == "frontend"
+    assert m.name is None
+
+
+def test_import_requires_clusters_when_not_global():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[])
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[1])
+    assert m.cluster_ids == [1]
+
+
+def test_import_certificate_must_be_pem():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="not a pem", is_global=True)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="", is_global=True)
+
+
+def test_import_certificate_size_capped():
+    huge = _CERT_PEM + "A" * (64 * 1024 + 1)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=huge, is_global=True)
+
+
+def test_import_chain_optional_but_validated():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, chain_content="  ")
+    assert m.chain_content is None
+    with pytest.raises(ValidationError):
+        SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, chain_content="garbage"
+        )
+
+
+def test_import_name_override_shares_the_name_contract():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="renamed")
+    assert m.name == "renamed"
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="../evil")
+    # Empty override collapses to None (falls back to the CSR's own name).
+    m2 = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="  ")
+    assert m2.name is None
+
+
+def test_import_usage_type_enum():
+    for good in ("frontend", "server"):
+        assert SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, usage_type=good
+        ).usage_type == good
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, usage_type="both")

backend/tests/test_csr_router_auth.py

@@ -0,0 +1,66 @@
+"""
+v1.9.0 CSR creation — behavioral auth tests for /api/ssl/csrs endpoints
+(pattern: test_ssl_list_endpoint_auth.py).
+
+Every CSR endpoint must refuse unauthenticated / garbage-token requests.
+The CSR detail route additionally must never 200 without auth because it
+returns the CSR PEM; no endpoint ever returns the private key, but auth is
+the first line regardless.
+"""
+import pytest
+
+_VALID_CREATE_BODY = {
+    "name": "auth-test-csr",
+    "common_name": "www.example.com",
+}
+
+_VALID_IMPORT_BODY = {
+    "certificate_content": (
+        "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+    ),
+    "is_global": True,
+}
+
+_ENDPOINTS = [
+    ("get", "/api/ssl/csrs", None),
+    ("get", "/api/ssl/csrs/1", None),
+    ("post", "/api/ssl/csrs", _VALID_CREATE_BODY),
+    ("post", "/api/ssl/csrs/1/import", _VALID_IMPORT_BODY),
+    ("delete", "/api/ssl/csrs/1", None),
+]
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_unauthenticated_rejected(client, method, path, body):
+    """No Authorization header → endpoint must refuse the request."""
+    res = getattr(client, method)(path, json=body) if body is not None else getattr(client, method)(path)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} without Authorization returned "
+        f"{res.status_code} — anonymous access to CSR data must not be "
+        f"possible. Body: {res.text[:200]}"
+    )
+    if res.status_code == 200:  # defensive, mirrors the R18 test style
+        data = res.json()
+        assert not data, "CSR endpoint returned data without auth"
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_invalid_token_rejected(client, method, path, body):
+    """Garbage token → endpoint must refuse the request."""
+    headers = {"Authorization": "Bearer not-a-valid-jwt"}
+    if body is not None:
+        res = getattr(client, method)(path, json=body, headers=headers)
+    else:
+        res = getattr(client, method)(path, headers=headers)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} with an invalid token returned {res.status_code}"
+    )
+
+
+def test_csr_routes_are_registered(client):
+    """The router must actually be mounted — a 404 would make the auth tests
+    above pass vacuously."""
+    res = client.get("/api/ssl/csrs")
+    assert res.status_code != 404, (
+        "GET /api/ssl/csrs returned 404 — csr_router is not registered in main.py"
+    )

backend/tests/test_csr_service_crypto.py

@@ -0,0 +1,171 @@
+"""
+v1.9.0 CSR creation — pure-crypto tests for services/csr_service.py.
+
+No mocks: every algorithm's output must parse with `cryptography` and the
+CSR's public key must match the generated private key (the property the
+whole import flow depends on).
+"""
+from types import SimpleNamespace
+
+import pytest
+
+from cryptography import x509
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import ExtensionOID, NameOID
+
+from services.csr_service import csr_row_to_dict, diff_domains, generate_csr_bundle
+
+
+def _payload(**overrides):
+    base = dict(
+        name="test-csr",
+        common_name="www.example.com",
+        organization=None,
+        organizational_unit=None,
+        locality=None,
+        state=None,
+        country=None,
+        email=None,
+        sans=[],
+        key_algorithm="rsa-2048",
+    )
+    base.update(overrides)
+    return SimpleNamespace(**base)
+
+
+def _spki(key):
+    return key.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+
+
+@pytest.mark.parametrize(
+    "algo,key_cls,key_check",
+    [
+        ("rsa-2048", rsa.RSAPrivateKey, lambda k: k.key_size == 2048),
+        ("rsa-4096", rsa.RSAPrivateKey, lambda k: k.key_size == 4096),
+        ("ecdsa-p256", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp256r1"),
+        ("ecdsa-p384", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp384r1"),
+    ],
+)
+def test_generate_bundle_all_algorithms(algo, key_cls, key_check):
+    bundle = generate_csr_bundle(_payload(key_algorithm=algo))
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    key = serialization.load_pem_private_key(
+        bundle["private_key_pem"].encode(), password=None
+    )
+
+    assert isinstance(key, key_cls)
+    assert key_check(key)
+    # The CSR must be signed by exactly this key.
+    csr_spki = csr.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+    assert csr_spki == _spki(key)
+    assert csr.is_signature_valid
+    # PKCS8, unencrypted — the agent concatenates cert+key into one PEM and
+    # HAProxy cannot read passphrase-protected keys.
+    assert bundle["private_key_pem"].startswith("-----BEGIN PRIVATE KEY-----")
+
+
+def test_subject_contains_all_provided_fields():
+    bundle = generate_csr_bundle(_payload(
+        organization="Example Corp",
+        organizational_unit="IT",
+        locality="Istanbul",
+        state="Marmara",
+        country="TR",
+        email="ops@example.com",
+    ))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+
+    def _one(oid):
+        attrs = csr.subject.get_attributes_for_oid(oid)
+        return attrs[0].value if attrs else None
+
+    assert _one(NameOID.COMMON_NAME) == "www.example.com"
+    assert _one(NameOID.ORGANIZATION_NAME) == "Example Corp"
+    assert _one(NameOID.ORGANIZATIONAL_UNIT_NAME) == "IT"
+    assert _one(NameOID.LOCALITY_NAME) == "Istanbul"
+    assert _one(NameOID.STATE_OR_PROVINCE_NAME) == "Marmara"
+    assert _one(NameOID.COUNTRY_NAME) == "TR"
+    assert _one(NameOID.EMAIL_ADDRESS) == "ops@example.com"
+    assert bundle["subject"] == {
+        "O": "Example Corp", "OU": "IT", "L": "Istanbul",
+        "ST": "Marmara", "C": "TR", "emailAddress": "ops@example.com",
+    }
+
+
+def test_subject_omits_empty_fields():
+    bundle = generate_csr_bundle(_payload())
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    assert not csr.subject.get_attributes_for_oid(NameOID.ORGANIZATION_NAME)
+    assert bundle["subject"] == {}
+
+
+def test_sans_cn_first_and_deduped():
+    bundle = generate_csr_bundle(_payload(
+        common_name="www.example.com",
+        sans=["api.example.com", "www.example.com", "api.example.com", "cdn.example.com"],
+    ))
+    assert bundle["sans"] == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    dns_names = san_ext.value.get_values_for_type(x509.DNSName)
+    assert dns_names == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+
+def test_wildcard_common_name_flows_into_san():
+    bundle = generate_csr_bundle(_payload(common_name="*.example.com"))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    assert san_ext.value.get_values_for_type(x509.DNSName) == ["*.example.com"]
+
+
+def test_diff_domains_reports_added_and_dropped():
+    warnings = diff_domains(
+        ["www.example.com", "api.example.com"],
+        ["WWW.example.com", "cdn.example.com"],
+    )
+    assert len(warnings) == 2
+    added = next(w for w in warnings if "added" in w)
+    dropped = next(w for w in warnings if "dropped" in w)
+    assert "cdn.example.com" in added
+    assert "api.example.com" in dropped
+    # Case-insensitive: www must NOT be reported in either direction.
+    assert "www.example.com" not in added
+    assert "www.example.com" not in dropped
+
+
+def test_diff_domains_identical_sets_yield_no_warnings():
+    assert diff_domains(["a.example.com"], ["A.EXAMPLE.COM"]) == []
+    assert diff_domains([], []) == []
+
+
+def test_csr_row_to_dict_never_exposes_private_key():
+    row = {
+        "id": 1,
+        "name": "x",
+        "private_key_pem": "-----BEGIN PRIVATE KEY-----\nSECRET\n-----END PRIVATE KEY-----",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nX\n-----END CERTIFICATE REQUEST-----",
+        "subject": '{"O": "Example"}',
+        "sans": '["a.example.com"]',
+    }
+    out = csr_row_to_dict(row)
+    assert "private_key_pem" not in out
+    assert "csr_pem" not in out          # lists exclude the PEM
+    assert out["subject"] == {"O": "Example"}
+    assert out["sans"] == ["a.example.com"]
+
+    detail = csr_row_to_dict(row, include_pem=True)
+    assert "private_key_pem" not in detail  # NEVER, even on detail
+    assert detail["csr_pem"].startswith("-----BEGIN CERTIFICATE REQUEST-----")

backend/tests/test_dns01.py

@@ -0,0 +1,615 @@
+"""Issue #35 — ACME DNS-01: focused unit tests for the pure logic (no DB/network).
+
+Covers the TXT-value math (RFC 8555 §8.4 — raw SHA-256 digest, base64url, NOT hex),
+the _acme-challenge record-name derivation (wildcard stripping), credential encryption
+round-trip + tamper handling, the DNS provider registry/allow-list, and (v1.10.0) the
+GoDaddy provider's zone-relative name derivation and additive RRset merge math.
+"""
+import base64
+import hashlib
+import os
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-dns01-unit-tests")
+
+from services.acme_service import ACMEService
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import (
+    encrypt_dns_credentials, decrypt_dns_credentials, reset_fernet_for_tests,
+)
+
+
+def _b64url(b: bytes) -> str:
+    return base64.urlsafe_b64encode(b).rstrip(b"=").decode("ascii")
+
+
+def test_dns_txt_value_is_raw_sha256_base64url():
+    key_auth = "token123.thumbprintABC"
+    expected = _b64url(hashlib.sha256(key_auth.encode("utf-8")).digest())
+    assert ACMEService._dns_txt_value(key_auth) == expected
+    # Must NOT be the (classic-mistake) base64url of the HEX digest.
+    hex_based = _b64url(hashlib.sha256(key_auth.encode("utf-8")).hexdigest().encode("utf-8"))
+    assert ACMEService._dns_txt_value(key_auth) != hex_based
+
+
+def test_challenge_dns_name_derivation():
+    assert ACMEService._challenge_dns_name("example.com") == "_acme-challenge.example.com"
+    # Wildcard: the '*.' is stripped, so apex + wildcard share the SAME record name.
+    assert ACMEService._challenge_dns_name("*.example.com") == "_acme-challenge.example.com"
+    assert ACMEService._challenge_dns_name("foo.bar.example.com") == "_acme-challenge.foo.bar.example.com"
+
+
+def test_credential_encryption_roundtrip():
+    reset_fernet_for_tests()
+    creds = {"api_token": "super-secret-token-value"}
+    token = encrypt_dns_credentials(creds)
+    assert token != "super-secret-token-value"
+    assert "super-secret-token-value" not in token  # ciphertext, not plaintext
+    assert decrypt_dns_credentials(token) == creds
+
+
+def test_decrypt_invalid_token_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_dns_credentials("not-a-valid-fernet-token") is None
+
+
+def test_provider_registry_and_allow_list():
+    names = {p["name"] for p in list_providers()}
+    assert {"manual", "cloudflare", "godaddy"} <= names
+    assert is_supported("manual") and is_supported("cloudflare") and is_supported("godaddy")
+    assert not is_supported("route53")  # not in the allow-list
+
+    assert get_provider("manual").automated is False
+    cf = get_provider("cloudflare", {"api_token": "x"})
+    assert cf.automated is True
+    assert any(f["key"] == "api_token" for f in cf.credential_fields)
+
+    raised = False
+    try:
+        get_provider("definitely-not-a-provider")
+    except ValueError:
+        raised = True
+    assert raised
+
+
+def test_cloudflare_token_sanitize():
+    # Issue #35 follow-up: a pasted token with quotes/spaces/control/unicode chars produced an
+    # invalid Authorization header (CF 6003 "Invalid request headers"). The sanitizer strips them.
+    from services.dns_providers.cloudflare import _sanitize_token, CloudflareDNSProvider
+
+    # Surrounding double quotes stripped.
+    assert _sanitize_token('"abc123-_def"') == 'abc123-_def'
+    # Interior spaces / tabs / newlines removed.
+    assert _sanitize_token('abc 123\tdef\n') == 'abc123def'
+    # A clean token68 string is unchanged (cannot corrupt a valid Cloudflare token).
+    clean = 'A1b2-_C3.d4~e5+f6/g7=='
+    assert _sanitize_token(clean) == clean
+    # Single quotes and a zero-width char removed.
+    assert _sanitize_token("'tok" + chr(0x200b) + "en'") == 'token'
+
+    # The provider constructor sanitizes into _token and keeps the raw input for diagnostics.
+    p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
+    assert p._token == 'my-token_123'
+    assert p._raw_token == '"my-token_123"'
+
+
+# --- v1.10.0: GoDaddy provider (pure logic only — no network, no DB) ---
+
+
+def test_godaddy_credential_fields_schema():
+    # Re-assert DnsCredentialsUpsert's validator rules directly against the declared schema, so the
+    # UI can never render a field whose submission the API would reject with a 422.
+    import re
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    fields = GoDaddyDNSProvider.credential_fields
+    assert [f["key"] for f in fields] == ["api_key", "api_secret"]
+    for f in fields:
+        assert re.match(r"^[a-zA-Z0-9_]{1,50}$", f["key"])         # DnsCredentialsUpsert key regex
+        assert f["type"] == "password"                              # renders Input.Password, not Input
+        assert isinstance(f["max_length"], int) and 0 < f["max_length"] <= 4000  # validator value cap
+        assert f["help"] and isinstance(f["help"], str)             # shown in the Form.Item `extra` slot
+    # api_secret is optional on purpose: leaving it blank is how a Personal Access Token is used
+    # (Bearer), which is the migration path off the sso-key scheme GoDaddy is retiring.
+    assert fields[0]["required"] is True and fields[1]["required"] is False
+    # Must not reuse Cloudflare's field name: the register modal's credential Form.Items are named
+    # cred_<key> in a SHARED form and are not cleared when the provider dropdown changes.
+    assert "api_token" not in {f["key"] for f in fields}
+
+
+def test_godaddy_provider_is_automated():
+    p = get_provider("godaddy", {"api_key": "k", "api_secret": "s"})
+    assert p.automated is True      # else the orchestrator takes the manual-confirm branch
+    assert p.name == "godaddy" and 1 <= len(p.name) <= 50  # dns_provider Field(min_length=1, max_length=50)
+    assert p.label == "GoDaddy"
+
+
+def test_godaddy_missing_credentials_returns_not_ok():
+    # verify_credentials must RETURN {"ok": False}, never raise: the router turns any non-
+    # DnsProviderError into the information-free generic 422 and the user never sees the reason.
+    import asyncio
+
+    for creds in ({}, {"api_secret": "s"}):  # blank UI fields arrive as MISSING keys, not ""
+        r = asyncio.run(get_provider("godaddy", creds).verify_credentials())
+        assert r["ok"] is False and r["detail"]
+    # Short-circuits before any request, so this touches no network.
+
+
+def test_godaddy_auth_header_formats_and_secret_never_leaks():
+    from services.dns_providers.godaddy import GoDaddyDNSProvider, _scrub
+
+    sentinel = "SENTINEL-SECRET-DO-NOT-LEAK"
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": sentinel})
+    # Literal prefix, one space, a single colon — no base64, no quoting.
+    assert p._auth_header() == f"sso-key KEY123:{sentinel}"
+    # No secret -> Personal Access Token. This one branch is the whole sso-key-sunset migration.
+    assert GoDaddyDNSProvider({"api_key": "PAT"})._auth_header() == "Bearer PAT"
+    # _scrub removes credential substrings from anything bound for a log or an order event.
+    assert sentinel not in _scrub(f"boom {sentinel} boom", "KEY123", sentinel)
+    assert "KEY123" not in _scrub("boom KEY123", "KEY123", sentinel)
+    assert _scrub("x" * 500, "KEY123") == "x" * 300  # bounded, so a huge body can't flood an event
+
+    # The channel that actually persists text: _http_error composes the message an order event and
+    # letsencrypt_orders.error_detail will carry, so it must scrub its own inputs — a caller that
+    # forgets to pre-scrub must not be able to leak. (Regression guard: scrubbing used to live at
+    # the single call site in _request instead of here.)
+    exc = p._http_error(403, f"DENIED_{sentinel}", f"token {sentinel} rejected", None)
+    assert sentinel not in str(exc) and "***" in str(exc)
+
+    # This module must not log at all — logging is the one channel _scrub cannot reach, since the
+    # arguments would be formatted by the logging framework rather than passed through it.
+    import inspect
+    import re as _re
+    from services.dns_providers import godaddy as gd_mod
+
+    assert not _re.search(r"\blogger\.\w+\(", inspect.getsource(gd_mod)), \
+        "godaddy.py must not log; surface everything through DnsProviderError so it is scrubbed"
+
+
+def test_godaddy_relative_record_name():
+    # GoDaddy names are RELATIVE to the zone with no trailing dot; the apex is the literal "@".
+    from services.dns_providers.godaddy import _relative_name
+
+    assert _relative_name("_acme-challenge.example.com", "example.com") == "_acme-challenge"
+    assert _relative_name("_acme-challenge.foo.bar.example.com", "example.com") == "_acme-challenge.foo.bar"
+    assert _relative_name("example.com", "example.com") == "@"          # never "" — see _rrset_path
+    assert _relative_name("_acme-challenge.example.com.", "example.com") == "_acme-challenge"
+    assert _relative_name("_ACME-Challenge.Example.COM", "example.com") == "_acme-challenge"
+    # Apex and wildcard produce the SAME relative name — which is exactly why the merge below
+    # has to be additive.
+    apex = ACMEService._challenge_dns_name("example.com")
+    wild = ACMEService._challenge_dns_name("*.example.com")
+    assert _relative_name(apex, "example.com") == _relative_name(wild, "example.com") == "_acme-challenge"
+
+
+def test_godaddy_rrset_merge_is_additive():
+    # THE critical test: GoDaddy's PUT REPLACES an entire RRset, so the merge math is the only thing
+    # keeping a wildcard+apex certificate's two coexisting TXT values alive.
+    from services.dns_providers.godaddy import _live_values, _merge_add, _merge_remove
+
+    def vals(body):
+        return sorted(r["data"] for r in body)
+
+    assert vals(_merge_add([{"data": "valueA", "ttl": 600}], "valueB")) == ["valueA", "valueB"]
+    assert _merge_add([{"data": "valueA"}], "valueA") is None      # idempotent; ACME retries land here
+    # Total, not an all()-over-a-computed-list (which passes vacuously on an empty result): the
+    # first publish at a fresh name must emit exactly one element, carrying the 600s TTL floor.
+    assert _merge_add([], "v") == [{"data": "v", "ttl": 600}]      # below 600 GoDaddy answers 422
+    # Tombstone rows ({"data": ""}) must never be echoed back — GoDaddy answers 422 INVALID_BODY.
+    assert _live_values([{"data": ""}, {"data": "x"}, {}]) == ["x"]
+    assert vals(_merge_add([{"data": ""}, {"data": "valueA"}], "valueB")) == ["valueA", "valueB"]
+
+    assert vals(_merge_remove([{"data": "valueA"}, {"data": "valueB"}], "valueB")) == ["valueA"]
+    assert _merge_remove([{"data": "valueA"}], "valueZ") is None    # already gone — tolerate
+    assert _merge_remove([], "valueZ") is None
+    # [] means "use DELETE": PUT with an empty array is rejected (422, "Records must be specified").
+    assert _merge_remove([{"data": "valueA"}], "valueA") == []
+    assert _merge_remove([{"data": ""}, {"data": "valueA"}], "valueA") == []
+
+
+def test_godaddy_never_builds_a_zone_wide_txt_path():
+    # A 3-segment path (.../records/TXT) is the endpoint that wipes EVERY TXT in the zone — SPF,
+    # DKIM, DMARC, domain verifications. An empty relative name must never be able to produce it.
+    from services.dns_providers.godaddy import _rrset_path
+
+    p = _rrset_path("example.com", "_acme-challenge")
+    assert p == "/domains/example.com/records/TXT/_acme-challenge"
+    assert p.count("/") == 5 and not p.endswith("/TXT")
+    assert _rrset_path("example.com", "@").endswith("/%40")  # apex percent-encoded for proxy safety
+    # "." and ".." survive quote() and are then normalized away by yarl when the URL is built, so
+    # ".../records/TXT/.." would resolve to the whole-zone endpoint. They must be refused too.
+    for bad in [("example.com", ""), ("", "_acme-challenge"), ("example.com", "."),
+                ("example.com", ".."), ("example.com", "...")]:
+        raised = False
+        try:
+            _rrset_path(*bad)
+        except DnsProviderError:
+            raised = True
+        assert raised, f"_rrset_path{bad} must refuse to build a zone-wide TXT path"
+    # And the only way to reach those inputs — a malformed domain — really does produce them.
+    from services.dns_providers.godaddy import _relative_name as _rel
+    assert _rel("..example.com", "example.com") == "."
+
+
+def test_godaddy_credential_encryption_roundtrip():
+    # The two-field credential dict rides the same Fernet blob as Cloudflare's single token.
+    reset_fernet_for_tests()
+    creds = {"api_key": "gd-key-plaintext", "api_secret": "gd-secret-plaintext"}
+    token = encrypt_dns_credentials(creds)
+    assert "gd-key-plaintext" not in token and "gd-secret-plaintext" not in token  # ciphertext
+    assert decrypt_dns_credentials(token) == creds
+    # This sorted key list is exactly what GET /dns-credentials exposes as credential_fields_present
+    # — names only, never values.
+    assert sorted(decrypt_dns_credentials(token).keys()) == ["api_key", "api_secret"]
+
+
+_GD_NS = "/domains/example.com/records/NS"
+_GD_TXT = "/domains/example.com/records/TXT/_acme-challenge"
+
+
+def _gd_provider(responses):
+    """A GoDaddy provider whose _request is replaced by a recorder.
+
+    The pure-merge tests above prove the MATH; this proves the WRITE PATH actually uses it. Without
+    it, replacing the merge with a single-value PUT — the mutation that silently destroys the
+    sibling value of every wildcard+apex certificate — leaves the whole suite green.
+
+    `responses` maps (method, path) -> value to return, or an Exception to raise. Unmapped calls
+    return None, which is how the zone suffix-walk's failed probes are modelled.
+    """
+    import types
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    calls = []
+
+    async def _fake_request(self, session, method, path, **kwargs):
+        calls.append((method, path, kwargs.get("json")))
+        result = responses.get((method, path))
+        if isinstance(result, Exception):
+            raise result
+        return result
+
+    p = GoDaddyDNSProvider({"api_key": "k", "api_secret": "s"})
+    p._request = types.MethodType(_fake_request, p)
+    return p, calls
+
+
+def _assert_never_zone_wide(calls):
+    # A write to .../records or .../records/TXT replaces every TXT (or every record) in the zone.
+    for method, path, _json in calls:
+        if method in ("PUT", "DELETE"):
+            assert not path.endswith("/records"), f"zone-wide write: {method} {path}"
+            assert not path.endswith("/records/TXT"), f"type-wide write: {method} {path}"
+
+
+def test_godaddy_add_write_path_merges_siblings():
+    import asyncio
+
+    # An existing sibling value at the same name — the apex half of an apex+wildcard certificate.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+
+    writes = [c for c in calls if c[0] in ("PUT", "PATCH", "DELETE")]
+    assert len(writes) == 1 and writes[0][0] == "PUT" and writes[0][1] == _GD_TXT
+    # BOTH values must be in the body: GoDaddy's PUT replaces the whole RRset.
+    assert sorted(r["data"] for r in writes[0][2]) == ["valueA", "valueB"]
+    _assert_never_zone_wide(calls)
+
+
+def test_godaddy_add_write_path_is_idempotent_and_fails_closed():
+    import asyncio
+
+    # Already published -> no write at all (this is where an ACME retry cycle lands).
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueB", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    assert [c for c in calls if c[0] != "GET"] == []
+
+    # Unreadable RRset read (2xx whose body did not parse as a list) must FAIL, never be treated as
+    # an empty RRset — the PUT that follows would replace the sibling values with only ours.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): None,
+    })
+    raised = False
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    except DnsProviderError:
+        raised = True
+    assert raised, "an unreadable RRset read must not be coerced into an empty RRset"
+    assert [c for c in calls if c[0] != "GET"] == []
+
+
+def test_godaddy_remove_write_path_uses_delete_for_the_last_value():
+    import asyncio
+
+    # Two values -> PUT back the survivor only.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}, {"data": "valueB"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueB"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0][0] == "PUT"
+    assert [r["data"] for r in writes[0][2]] == ["valueA"]
+
+    # Last value -> DELETE. `PUT []` is rejected by GoDaddy (422 INVALID_BODY), so an empty PUT
+    # body would make every cleanup fail forever.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueA"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0] == ("DELETE", _GD_TXT, None)
+    assert not any(c[0] == "PUT" and c[2] == [] for c in calls)
+
+    # Value already gone -> no write, no error.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueZ"))
+    assert [c for c in calls if c[0] != "GET"] == []
+    _assert_never_zone_wide(calls)
+
+
+class _FakeGDResponse:
+    """Minimal stand-in for aiohttp's ClientResponse: status, headers, and json()."""
+
+    _NO_BODY = object()
+
+    def __init__(self, status, body=_NO_BODY, headers=None):
+        self.status = status
+        self._body = body
+        self.headers = headers or {}
+
+    async def json(self, content_type=None):
+        if self._body is _FakeGDResponse._NO_BODY:
+            raise ValueError("no body to decode")  # what an empty 204 does
+        return self._body
+
+
+class _FakeGDSession:
+    def __init__(self, response):
+        self._response = response
+        self.calls = []
+
+    def request(self, method, url, **kwargs):
+        self.calls.append((method, url, kwargs))
+        response = self._response
+
+        class _Ctx:
+            async def __aenter__(self_inner):
+                return response
+
+            async def __aexit__(self_inner, *exc):
+                return False
+
+        return _Ctx()
+
+
+def test_godaddy_request_status_handling():
+    import asyncio
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": "SEC456"})
+
+    def call(response):
+        session = _FakeGDSession(response)
+        try:
+            return asyncio.run(p._request(session, "PUT", "/domains/example.com/records/TXT/x",
+                                          json=[{"data": "v", "ttl": 600}])), None, session
+        except DnsProviderError as exc:
+            return None, str(exc), session
+
+    # 204 with an EMPTY body is the normal answer to every GoDaddy write — it must not raise.
+    body, err, session = call(_FakeGDResponse(204))
+    assert body is None and err is None
+    # Redirects are deliberately not followed (aiohttp would forward the Authorization header), so
+    # a 3xx is a FAILED call. Treating it as success would report a redirected write as a no-op.
+    _kw = session.calls[0][2]
+    assert _kw["allow_redirects"] is False
+    assert _kw["headers"]["Authorization"] == "sso-key KEY123:SEC456"
+    assert _kw["headers"]["Accept"] == "application/json"
+    for status in (301, 302, 307):
+        body, err, _ = call(_FakeGDResponse(status))
+        assert body is None and err and str(status) in err, f"HTTP {status} must not read as success"
+
+    # 200 with a list is passed through verbatim.
+    body, err, _ = call(_FakeGDResponse(200, [{"data": "v"}]))
+    assert err is None and body == [{"data": "v"}]
+
+    # Error mapping: each message must name what the operator has to fix.
+    _, err, _ = call(_FakeGDResponse(401, {"code": "UNABLE_TO_AUTHENTICATE", "message": "nope"}))
+    assert "PRODUCTION" in err and "UNABLE_TO_AUTHENTICATE" in err
+    _, err, _ = call(_FakeGDResponse(403, {"code": "ACCESS_DENIED", "message": "not allowed"}))
+    assert "domains.dns:update" in err
+    # 429: Retry-After wins; the legacy body field is the fallback; absent both -> 60s default.
+    _, err, _ = call(_FakeGDResponse(429, None, {"Retry-After": "17"}))
+    assert "~17s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"retryAfterSec": 42}))
+    assert "~42s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"Retry-After": "not-a-number"}))
+    assert "~60s" in err
+    # A non-dict error body must not crash the error mapper.
+    _, err, _ = call(_FakeGDResponse(500, "<html>gateway</html>"))
+    assert "500" in err
+
+    # A transport failure MID-READ must not be mistaken for "empty body". Only a decode error may
+    # be swallowed: a caller reading an RRset would otherwise see None and could take it for an
+    # empty record set, and the full-RRset PUT that follows would destroy the sibling values.
+    import aiohttp
+
+    class _TruncatedResponse(_FakeGDResponse):
+        async def json(self, content_type=None):
+            raise aiohttp.ClientPayloadError("connection closed mid-body")
+
+    body, err, _ = call(_TruncatedResponse(200))
+    assert body is None and err and "GoDaddy" in err
+
+
+def test_godaddy_zone_resolution_walks_suffixes_and_caches():
+    import asyncio
+    from services.dns_providers.godaddy import _GoDaddyHTTPError
+
+    # The deepest candidate is not a zone (404 = "not this zone"); the walk must continue to the
+    # registrable domain and then reuse it, so the second challenge at the same name costs no probe.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("nope", status=404, code="UNKNOWN_DOMAIN"),
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueA"))
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    probes = [c for c in calls if c[1].endswith("/records/NS")]
+    assert len(probes) == 2, "the resolved zone must be cached for the life of the provider"
+    # Relative name derived from the RESOLVED zone, never from the deepest candidate.
+    assert all(c[1] == _GD_TXT for c in calls if "/records/TXT/" in c[1])
+
+    # A credential/eligibility failure during the walk must surface, not be swallowed as
+    # "no managed domain" — otherwise the operator chases a DNS problem that is really a bad key.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("denied", status=403, code="ACCESS_DENIED"),
+    })
+    raised = ""
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "v"))
+    except DnsProviderError as exc:
+        raised = str(exc)
+    assert "denied" in raised and "No managed GoDaddy domain" not in raised
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/tests/test_haproxy_validator_bulgu12.py

@@ -206,41 +206,38 @@ def test_module_level_validate_haproxy_config_forwards_partial_fragment():
 
 
 # ──────────────────────────────────────────────────────────────────────
-# Wizard Pydantic gate: ACL `-f` flag must be REJECTED at submit.
+# Issue #38 follow-up: ACL `-f <file>` pattern-file references are
+# ACCEPTED (the Bulgu #12 hard reject was removed — pattern files are
+# operator-managed host files, the agent's pre-reload `haproxy -c`
+# makes a missing file fail safely, and bulk import always accepted
+# `-f`). These tests pin the ACCEPT behaviour.
 # ──────────────────────────────────────────────────────────────────────
 
 
-def test_wizard_pydantic_rejects_acl_with_file_flag():
-    """Pre-fix the wizard's ACL string validator passed
-    `acl name path -i -m reg -f /path` straight through. Apply-time
-    HAProxy `-c` then failed with "failed to open pattern file".
-    Pin that the validator now rejects `-f` at submit.
+def test_wizard_pydantic_accepts_acl_with_file_flag():
+    """Issue #38 follow-up — the wizard's ACL string validator must
+    ACCEPT `-f <file>` pattern-file references (Bulgu #12 reject
+    removed). Operators with large host-managed IP blacklists rely
+    on this in production.
     """
     from models.site_wizard import FrontendStep
 
-    # Minimal valid wizard frontend kwargs — only the offending
-    # acl_rules entry should trigger the failure.
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=["acl1 path -i -m reg -f /path"],
     )
-    from pydantic import ValidationError
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(**fe_kwargs)
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Bulgu #12 regression: ACL -f flag must be rejected with a "
-        f"clear pattern-file error. Got: {msg}"
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"], (
+        "ACL `-f` rule must round-trip verbatim through the wizard model"
     )
 
 
 @pytest.mark.parametrize(
     "rule",
     [
-        # Various spacing / position variants the regex must catch.
+        # Various spacing / position variants must all be accepted.
         "acl1 path -f /etc/haproxy/list",
         "acl1 path -i -f /tmp/x.lst",
         "acl1 src -f /etc/haproxy/admins.lst",
@@ -249,23 +246,19 @@ def test_wizard_pydantic_rejects_acl_with_file_flag():
         "acl1 path -f",
     ],
 )
-def test_wizard_pydantic_rejects_acl_with_file_flag_variants(rule):
-    """Every spacing / position variant the operator might type must
-    be rejected. Pinned defensively so the regex never accidentally
-    relaxes to "only matches trailing -f".
-    """
+def test_wizard_pydantic_accepts_acl_with_file_flag_variants(rule):
+    """Every spacing / position variant must be accepted verbatim
+    (Issue #38 follow-up — no `-f` shape may be rejected)."""
     from models.site_wizard import FrontendStep
-    from pydantic import ValidationError
 
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=[rule],
     )
-    with pytest.raises(ValidationError):
-        FrontendStep(**fe_kwargs)
+    assert fe.acl_rules == [rule]
 
 
 def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
@@ -291,42 +284,29 @@ def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
     assert len(fe.acl_rules) == 3
 
 
-def test_manual_frontend_validator_rejects_acl_with_file_flag():
-    """Parity check: the manual Frontend API
-    (`models/frontend.py::validate_acl_rules`) must apply the same
-    `-f` rejection. Operators see consistent behaviour from both the
-    wizard and the per-entity frontend page.
+def test_manual_frontend_validator_accepts_acl_with_file_flag():
+    """Parity check (Issue #38 follow-up): the manual Frontend API
+    (`models/frontend.py::validate_acl_rules`) must ACCEPT `-f`
+    pattern-file references, same as the wizard and bulk import.
     """
     from models.frontend import FrontendConfig
-    from pydantic import ValidationError
 
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendConfig(
-            name="fe1",
-            bind_port=80,
-            mode="http",
-            acl_rules=["acl1 path -i -m reg -f /path"],
-        )
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Manual frontend API parity regression: ACL -f flag must be "
-        f"rejected. Got: {msg}"
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        acl_rules=["acl1 path -i -m reg -f /path"],
     )
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"]
 
 
-def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
-    """Round-3 audit extension — structured redirect dicts (the
-    alternative shape that `models/site_wizard.py::_validate_redirect_rules`
-    accepts alongside legacy strings) also flow through to
-    `services/haproxy_config.py::_format_redirect_rule` and emit
-    their `condition` / `target` verbatim into the rendered HAProxy
-    directive. Without the dict-aware reject the visual builder's
-    `-f` block could be bypassed by hand-crafting a dict payload
-    against the API — recreating the same `failed to open pattern
-    file` failure at apply time.
+def test_wizard_pydantic_accepts_structured_redirect_dict_with_file_flag():
+    """Issue #38 follow-up — structured redirect dicts carrying `-f`
+    pattern-file references in `condition`/`target` are ACCEPTED
+    (the Bulgu #12 dict-aware reject was removed together with the
+    string-rule reject).
     """
-    from models.site_wizard import FrontendStep, BackendStep
-    from pydantic import ValidationError
+    from models.site_wizard import FrontendStep
 
     fe_kwargs = dict(
         name="fe1",
@@ -334,37 +314,32 @@ def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
         mode="http",
     )
 
-    # `condition` carrying `-f` must be rejected.
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "scheme",
-                    "target": "https",
-                    "condition": "if { src -f /etc/haproxy/admins.lst }",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `condition` carrying `-f` is accepted.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "scheme",
+                "target": "https",
+                "condition": "if { src -f /etc/haproxy/admins.lst }",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/admins.lst }"
 
-    # `target` carrying `-f` must also be rejected (defence-in-depth
-    # for hand-crafted payloads).
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "location",
-                    "target": "/foo -f /tmp/x.lst",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `target` carrying `-f` is accepted too.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "location",
+                "target": "/foo -f /tmp/x.lst",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["target"] == "/foo -f /tmp/x.lst"
 
-    # Clean structured dict still passes — no false positive.
+    # Clean structured dict still passes.
     FrontendStep(
         **fe_kwargs,
         redirect_rules=[
@@ -667,8 +642,8 @@ def test_user_reported_wizard_config_emits_no_false_warnings():
     zero WARNINGs from the directives we expanded.
     """
     # Distilled from the user's bulk-site-create snapshot, minus the
-    # `-f` ACL (which the new Pydantic gate rejects before this
-    # validator ever runs).
+    # `-f` ACL (accepted since the Issue #38 follow-up, but irrelevant
+    # to the directive-expansion warnings this test pins).
     config = """# ─── Wizard candidate fragment (dry-run preview) ───
 frontend fe-site1
     bind *:80
@@ -6363,9 +6338,16 @@ def test_bulgu83_static_marker_in_fe_warning_toast():
     calls these rules "legacy" and now lists each offending rule
     body. Static-source check so a refactor that re-introduces
     the misleading wording or drops the rule snippets is caught.
+
+    Skipped automatically when the test runs inside the backend
+    Dockerfile build context (which only copies `backend/` and
+    therefore has no `frontend/` tree next to it). This mirrors
+    the guard already used by every other front-end static-source
+    pin in this file (e.g. lines 583-592, 795-810, 833-845, etc.)
+    — Bulgu #83's pin was missing it, which broke the corporate
+    CI's `RUN python -m pytest` step in the backend Docker build
+    immediately after Round-23 went live.
     """
-    # The frontend tree lives next to backend/ at the workspace
-    # root, so walk up one extra level from _BACKEND_DIR.
     fm_path = (
         _BACKEND_DIR.parent
         / "frontend"
@@ -6373,6 +6355,11 @@ def test_bulgu83_static_marker_in_fe_warning_toast():
         / "components"
         / "FrontendManagement.js"
     )
+    if not fm_path.exists():
+        pytest.skip(
+            f"frontend not present at {fm_path}; running in backend-only "
+            "container is expected — skip JS source pin"
+        )
     fm_src = fm_path.read_text()
     assert "Bulgu #83 (round-23 audit)" in fm_src
     # No more `legacy routing/redirect rule(s)` wording.

backend/tests/test_keepalive_fault_state.py

@@ -0,0 +1,21 @@
+"""v1.7.6 — guard: the agent must surface keepalived FAULT state.
+
+A VIP whose interface has no usable IPv4 (or whose track-script fails) puts keepalived into
+FAULT — the virtual IP is NOT held. Previously get_keepalive_state only grepped (MASTER|BACKUP),
+so a FAULT'd VIP reported as BACKUP — misleading (looks healthy-ish). It must report FAULT so the
+UI shows it red. Both get_keepalive_state copies (installer + SKIP_TO_DAEMON daemon) must include
+FAULT. Pure source guard."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def test_keepalive_state_detects_fault_in_both_copies():
+    with open(os.path.join(ROOT, "utils", "agent_scripts", "linux_install.sh"), encoding="utf-8") as f:
+        s = f.read()
+    # FAULT added to the state grep in both copies (installer + daemon), both detection methods.
+    assert s.count("MASTER|BACKUP|FAULT") >= 2
+    # and the misleading MASTER|BACKUP-only grep is gone.
+    assert 'grep -oE "(MASTER|BACKUP)"' not in s

backend/tests/test_keepalived_config.py

@@ -0,0 +1,129 @@
+"""Issue #27 (v1.7.0) — unit tests for the keepalived config generator + secret crypto.
+
+Pure-function tests; no DB. Validates the rendered keepalived.conf for MASTER/BACKUP,
+unicast peers, the failover weight arithmetic, the script-security requirements, the
+ownership marker, and Fernet round-trip.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from services import keepalived_config as kc  # noqa: E402
+
+
+VIP = {
+    "id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+    "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True,
+}
+MEMBERS = [
+    {"role": "MASTER", "priority": 150, "network_interface": "eth0", "agent_id": 1, "ip_address": "10.0.0.11"},
+    {"role": "BACKUP", "priority": 100, "network_interface": "eth0", "agent_id": 2, "ip_address": "10.0.0.12"},
+]
+
+
+def _render(this_idx, auth="s3cr3t"):
+    this_agent = MEMBERS[this_idx]
+    peers = [m["ip_address"] for m in MEMBERS if m["agent_id"] != this_agent["agent_id"]]
+    return kc.render_keepalived_conf(vip=VIP, members=MEMBERS, this_agent=this_agent,
+                                     peer_ips=peers, auth_pass_plain=auth)
+
+
+class TestRender:
+    def test_master_state_priority_iface_vrid(self):
+        conf = _render(0)
+        assert "state MASTER" in conf
+        assert "priority 150" in conf
+        assert "interface eth0" in conf
+        assert "virtual_router_id 51" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+    def test_backup_state(self):
+        conf = _render(1)
+        assert "state BACKUP" in conf
+        assert "priority 100" in conf
+
+    def test_unicast_peers(self):
+        # MASTER's config lists the BACKUP as its unicast peer (and its own src ip).
+        conf = _render(0)
+        assert "unicast_src_ip 10.0.0.11" in conf
+        assert "unicast_peer" in conf
+        assert "10.0.0.12" in conf
+
+    def test_script_security_block(self):
+        conf = _render(0)
+        assert "enable_script_security" in conf
+        assert "script_user root" in conf
+
+    def test_ownership_marker(self):
+        assert kc.OWNERSHIP_MARKER in _render(0)
+
+    def test_weight_makes_failed_master_lose(self):
+        # On HAProxy failure the master's effective priority must drop below the backup.
+        conf = _render(0)
+        weight_line = [l for l in conf.splitlines() if l.strip().startswith("weight ")][0]
+        weight = int(weight_line.strip().split()[1])
+        assert 150 + weight < 100, "failed master must fall below every backup"
+        assert "track_script" in conf and "chk_haproxy" in conf
+
+    def test_track_disabled_omits_script(self):
+        vip = {**VIP, "track_haproxy": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain=None)
+        assert "vrrp_script" not in conf
+        assert "track_script" not in conf
+
+    def test_no_auth_when_secret_absent(self):
+        conf = _render(0, auth=None)
+        assert "auth_pass" not in conf
+
+    def test_multicast_omits_unicast(self):
+        vip = {**VIP, "use_unicast": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain="x")
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+
+    def test_single_node_omits_unicast_block(self):
+        # Single-node VIP (no peers): even with use_unicast=True we must NOT emit a bare
+        # `unicast_src_ip`/`unicast_peer` — keepalived treats a unicast keyword with no peers
+        # as deprecated, warns, and falls back to multicast (and `keepalived -t` flags it).
+        # Omitting the block yields a clean multicast config that holds the VIP solo.
+        only = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                 "agent_id": 1, "ip_address": "10.0.0.11"}]
+        conf = kc.render_keepalived_conf(vip=VIP, members=only, this_agent=only[0],
+                                         peer_ips=[], auth_pass_plain=None)
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+        assert "state MASTER" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+
+class TestCheckScript:
+    def test_default_process_name(self):
+        s = kc.build_haproxy_check_script()
+        assert "pidof haproxy" in s
+        assert kc.OWNERSHIP_MARKER in s
+
+    def test_process_name_from_bin_path(self):
+        s = kc.build_haproxy_check_script(bin_path="/opt/hap/sbin/haproxy-ent")
+        assert "pidof haproxy-ent" in s
+
+    def test_malicious_bin_path_falls_back(self):
+        s = kc.build_haproxy_check_script(bin_path="/x/haproxy; rm -rf /")
+        assert "rm -rf" not in s
+        assert "pidof haproxy" in s
+
+
+class TestSecretCrypto:
+    def test_roundtrip(self):
+        kc.reset_fernet_for_tests()
+        token = kc.encrypt_vrrp_secret("s3cr3t")
+        assert token != "s3cr3t"
+        assert kc.decrypt_vrrp_secret(token) == "s3cr3t"
+
+    def test_decrypt_garbage_returns_none(self):
+        kc.reset_fernet_for_tests()
+        assert kc.decrypt_vrrp_secret("not-a-fernet-token") is None

backend/tests/test_keepalived_parser.py

@@ -0,0 +1,341 @@
+"""Issue #27 follow-up (v1.10.4) — unit tests for parsing an EXISTING keepalived.conf so a
+hand-maintained VIP can be adopted.
+
+Pure-function tests; no DB, no network. The parser exists because the heartbeat carries only
+the VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+fields — and because adoption REPLACES the operator's file, so anything our renderer cannot
+reproduce has to be reported as a blocker rather than silently dropped.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-keepalived-parser-tests")
+
+from services import keepalived_config as kc  # noqa: E402
+from services.keepalived_parser import (  # noqa: E402
+    KeepalivedParseError, analyse_keepalived_conf, build_adoption_candidate,
+    parse_keepalived_conf,
+)
+
+
+# A realistic hand-maintained config: two nodes, unicast VRRP, password auth, HAProxy check.
+HANDWRITTEN = """\
+! Configuration File for keepalived
+global_defs {
+    enable_script_security
+    script_user root
+}
+
+vrrp_script chk_haproxy {
+    script "/etc/keepalived/check_haproxy.sh"
+    interval 2
+    weight -21
+}
+
+vrrp_instance VI_1 {
+    state MASTER
+    interface eth0        # public leg
+    virtual_router_id 51
+    priority 150
+    advert_int 1
+    authentication {
+        auth_type PASS
+        auth_pass s3cr3t
+    }
+    unicast_src_ip 10.0.0.11
+    unicast_peer {
+        10.0.0.12
+    }
+    virtual_ipaddress {
+        10.0.0.100/24 dev eth0
+    }
+    track_script {
+        chk_haproxy
+    }
+}
+"""
+
+
+def _only_candidate(text):
+    parsed = parse_keepalived_conf(text)
+    assert len(parsed["instances"]) == 1
+    return build_adoption_candidate(parsed, parsed["instances"][0])
+
+
+def test_parses_a_handwritten_config_into_model_fields():
+    cand = _only_candidate(HANDWRITTEN)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["blockers"] == []
+    assert cand["vip"] == {
+        "virtual_ip": "10.0.0.100",
+        "prefix_length": 24,
+        "virtual_router_id": 51,
+        "advert_int": 1,
+        "use_unicast": True,
+        "track_haproxy": True,
+        "auth_pass": "s3cr3t",
+    }
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["peers"] == ["10.0.0.12"] and cand["unicast_src_ip"] == "10.0.0.11"
+    assert cand["defaulted"] == []          # every value came from the file, nothing assumed
+
+
+def test_comment_and_layout_variants():
+    # `!` and `#` both start comments; a block may open and close on one line; a quoted
+    # script path keeps its spaces. None of this may change the parse.
+    text = """\
+#!/not/a/shebang — this whole line is a comment
+vrrp_script chk { script "/opt/my scripts/chk.sh" }
+vrrp_instance VI_1 { state BACKUP
+    interface eth1   ! trailing bang comment
+    virtual_router_id 7
+    priority 90
+    virtual_ipaddress { 192.168.5.9/32 dev eth1 }
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert parsed["scripts"]["chk"]["script"] == "/opt/my scripts/chk.sh"
+    inst = parsed["instances"][0]
+    assert inst["state"] == "BACKUP" and inst["interface"] == "eth1"
+    assert inst["virtual_router_id"] == 7 and inst["priority"] == 90
+    assert inst["virtual_ips"] == [
+        {"address": "192.168.5.9", "prefix_length": 32, "dev": "eth1", "extra": []}
+    ]
+
+
+def test_documented_defaults_are_applied_and_flagged():
+    # keepalived's own defaults for absent directives. Applying them re-renders the same
+    # behaviour, so they are allowed — but the UI must be able to say they were assumed.
+    text = """\
+vrrp_instance VI_1 {
+    interface eth0
+    virtual_router_id 12
+    virtual_ipaddress { 10.1.1.5/24 dev eth0 }
+}
+"""
+    cand = _only_candidate(text)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["member"]["role"] == "BACKUP" and cand["member"]["priority"] == 100
+    assert cand["vip"]["advert_int"] == 1
+    assert sorted(cand["defaulted"]) == ["advert_int", "priority", "state"]
+    # No authentication block and no track_script — both legal, both faithfully represented.
+    assert cand["vip"]["auth_pass"] is None and cand["vip"]["track_haproxy"] is False
+    assert cand["vip"]["use_unicast"] is False
+
+
+def _blockers_for(text):
+    return " | ".join(_only_candidate(text)["blockers"])
+
+
+def test_directives_we_cannot_render_block_adoption():
+    # THE central safety property: adoption overwrites the file, so a failover hook we do not
+    # render would be destroyed. It must stop the flow, not warn.
+    text = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _blockers_for(text)
+    assert "notify_master" in blockers and "would delete it" in blockers
+    assert _only_candidate(text)["adoptable"] is False
+
+
+def test_multiple_addresses_in_one_instance_block_adoption():
+    text = HANDWRITTEN.replace("        10.0.0.100/24 dev eth0",
+                               "        10.0.0.100/24 dev eth0\n        10.0.0.101/24 dev eth0")
+    blockers = _blockers_for(text)
+    assert "2 addresses" in blockers and "10.0.0.101" in blockers
+
+
+def test_missing_vrid_blocks_adoption_with_the_split_brain_reason():
+    text = HANDWRITTEN.replace("    virtual_router_id 51\n", "")
+    blockers = _blockers_for(text)
+    assert "no virtual_router_id" in blockers and "separate VRRP domains" in blockers
+
+
+def test_missing_prefix_blocks_adoption():
+    # Our renderer always writes an explicit prefix; guessing one would change the netmask of a
+    # live VIP, so the operator has to state it.
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _blockers_for(text)
+    assert "no explicit prefix length" in blockers
+
+
+def test_address_on_a_different_dev_blocks_adoption():
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100/24 dev eth1")
+    blockers = _blockers_for(text)
+    assert "dev eth1" in blockers and "interface eth0" in blockers
+
+
+def test_foreign_track_script_blocks_adoption():
+    text = HANDWRITTEN.replace("        chk_haproxy", "        chk_custom")
+    blockers = _blockers_for(text)
+    assert "chk_custom" in blockers and "replaced by OpenManager" in blockers
+
+
+def test_unsupported_auth_type_blocks_adoption():
+    text = HANDWRITTEN.replace("auth_type PASS", "auth_type AH")
+    assert "auth_type AH" in _blockers_for(text)
+
+
+def test_fractional_advert_int_blocks_adoption():
+    # Rounding 0.5s to 1s changes VRRP timing, so adopt-and-alter is not acceptable.
+    text = HANDWRITTEN.replace("advert_int 1", "advert_int 0.5")
+    blockers = _blockers_for(text)
+    assert "advert_int 0.5" in blockers and "fractional" in blockers
+
+
+def test_half_configured_unicast_blocks_adoption():
+    text = HANDWRITTEN.replace("    unicast_peer {\n        10.0.0.12\n    }\n", "")
+    assert "unicast_src_ip without unicast_peer" in _blockers_for(text)
+
+
+def test_sync_group_and_lvs_sections_block_adoption():
+    text = HANDWRITTEN + """
+vrrp_sync_group VG1 {
+    group {
+        VI_1
+    }
+}
+virtual_server 10.0.0.100 80 {
+    lb_algo rr
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert [g["name"] for g in parsed["sync_groups"]] == ["VG1"]
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "vrrp_sync_group VG1" in directives and "virtual_server" in directives
+    # Both are top-level, so EVERY candidate in the file is blocked — a sync group changes
+    # failover semantics for the instances it groups.
+    cand = build_adoption_candidate(parsed, parsed["instances"][0])
+    assert cand["adoptable"] is False
+
+
+def test_extra_global_defs_are_reported_as_losses():
+    text = HANDWRITTEN.replace("    script_user root",
+                               "    script_user root\n    router_id LVS_DEVEL")
+    parsed = parse_keepalived_conf(text)
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "global_defs/router_id LVS_DEVEL" in directives
+    assert parsed["global_defs"]["router_id"] == "LVS_DEVEL"
+
+
+def test_multiple_instances_yield_one_candidate_each():
+    text = HANDWRITTEN + """
+vrrp_instance VI_2 {
+    state BACKUP
+    interface eth0
+    virtual_router_id 52
+    priority 100
+    advert_int 1
+    virtual_ipaddress { 10.0.0.200/24 dev eth0 }
+}
+"""
+    analysed = analyse_keepalived_conf(text)
+    assert analysed["instance_count"] == 2
+    names = [c["instance_name"] for c in analysed["candidates"]]
+    assert names == ["VI_1", "VI_2"]
+    assert [c["vip"]["virtual_ip"] for c in analysed["candidates"]] == ["10.0.0.100", "10.0.0.200"]
+    assert all(c["adoptable"] for c in analysed["candidates"])
+
+
+def test_unbalanced_braces_raise():
+    for bad in ("vrrp_instance VI_1 {\n    state MASTER\n", "}\n"):
+        raised = False
+        try:
+            parse_keepalived_conf(bad)
+        except KeepalivedParseError:
+            raised = True
+        assert raised, f"should have raised for {bad!r}"
+
+
+def test_our_own_render_round_trips_with_zero_blockers():
+    """The invariant that keeps the parser honest: a config WE generated must parse back into
+    the same model with nothing unsupported. If a future change to render_keepalived_conf emits
+    a directive the parser does not know, this fails — instead of adoption silently reporting
+    that OpenManager's own output is unadoptable."""
+    vip = {"id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+           "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True}
+    members = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                "agent_id": 1, "ip_address": "10.0.0.11"},
+               {"role": "BACKUP", "priority": 100, "network_interface": "eth0",
+                "agent_id": 2, "ip_address": "10.0.0.12"}]
+    rendered = kc.render_keepalived_conf(
+        vip=vip, members=members, this_agent=members[0],
+        peer_ips=["10.0.0.12"], auth_pass_plain="s3cr3t")
+
+    cand = _only_candidate(rendered)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["vip"]["virtual_ip"] == vip["virtual_ip"]
+    assert cand["vip"]["prefix_length"] == vip["prefix_length"]
+    assert cand["vip"]["virtual_router_id"] == vip["virtual_router_id"]
+    assert cand["vip"]["track_haproxy"] is True and cand["vip"]["use_unicast"] is True
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["vip"]["auth_pass"] == "s3cr3t"
+
+    # And the same for the no-auth / multicast / untracked shape, which renders fewer blocks.
+    plain = kc.render_keepalived_conf(
+        vip={**vip, "use_unicast": False, "track_haproxy": False},
+        members=members, this_agent=members[1], peer_ips=[], auth_pass_plain=None)
+    cand2 = _only_candidate(plain)
+    assert cand2["adoptable"] is True, cand2["blockers"]
+    assert cand2["vip"]["use_unicast"] is False and cand2["vip"]["track_haproxy"] is False
+    assert cand2["vip"]["auth_pass"] is None
+
+
+# --- v1.10.4 adoption gate: which blockers an operator may resolve --------------------------
+
+
+def test_only_prefix_and_data_loss_are_waivable():
+    from services.keepalived_parser import remaining_blockers
+
+    loss = "line 9: `notify_master \"/x.sh\"` — OpenManager's renderer cannot reproduce this, so adopting would delete it"
+    prefix = "`10.0.0.5` has no explicit prefix length; state it during adoption so the netmask cannot change on takeover"
+    hard_vrid = "no virtual_router_id — it cannot be guessed: a wrong VRID puts the nodes in separate VRRP domains"
+    hard_auth = "auth_type AH is not supported (only PASS is rendered)"
+    all_four = [loss, prefix, hard_vrid, hard_auth]
+
+    # Nothing waived: everything survives.
+    assert remaining_blockers(all_four) == all_four
+    # A supplied prefix resolves ONLY the prefix blocker.
+    assert remaining_blockers(all_four, prefix_supplied=True) == [loss, hard_vrid, hard_auth]
+    # Accepting data loss resolves ONLY the loss blocker.
+    assert remaining_blockers(all_four, accept_data_loss=True) == [prefix, hard_vrid, hard_auth]
+    # Both together still cannot wave through an impossibility — this is the property that stops
+    # a UI flag from destroying a VIP whose VRID or auth_type we could not reproduce.
+    assert remaining_blockers(all_four, prefix_supplied=True, accept_data_loss=True) == \
+        [hard_vrid, hard_auth]
+    # And an adoptable candidate stays adoptable.
+    assert remaining_blockers([]) == []
+
+
+def test_waiver_markers_match_the_messages_the_parser_actually_emits():
+    # The gate matches on substrings of the blocker prose, so a reworded message would silently
+    # stop being waivable. Pin both directions against real parser output.
+    from services.keepalived_parser import remaining_blockers
+
+    no_prefix = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _only_candidate(no_prefix)["blockers"]
+    assert blockers, "expected a prefix blocker"
+    assert remaining_blockers(blockers, prefix_supplied=True) == []
+
+    with_hook = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _only_candidate(with_hook)["blockers"]
+    assert blockers, "expected a data-loss blocker"
+    assert remaining_blockers(blockers, accept_data_loss=True) == []
+
+
+def test_auth_pass_masking_leaves_no_trace_of_the_secret():
+    # The discovered config is stored and served to the UI, so the ingest endpoint masks the VRRP
+    # password. Reuse the router's own regex so the test breaks if it is loosened.
+    from routers.agent import _AUTH_PASS_MASK_RE
+
+    secret = "s3cr3t with spaces"
+    text = HANDWRITTEN.replace("auth_pass s3cr3t", f"auth_pass {secret}")
+    masked = _AUTH_PASS_MASK_RE.sub(r"\1********", text)
+    assert secret not in masked and "s3cr3t" not in masked
+    assert "auth_pass ********" in masked
+    # Everything else survives, so the preview is still useful.
+    assert "virtual_router_id 51" in masked and "10.0.0.100/24 dev eth0" in masked

backend/tests/test_mfa_backwards_compat.py

@@ -0,0 +1,150 @@
+"""Backwards-compatibility regression tests for the MFA rollout (Issue #18).
+
+These tests don't hit a real database — they exercise the authoritative
+contract surfaces (login response shape, auth_middleware behaviour) using
+mocks where needed so the suite stays fast and deterministic.
+"""
+from __future__ import annotations
+
+import os
+import sys
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+class TestAuthMiddlewareUnchanged:
+    """auth_middleware MUST NOT look for MFA claims — Madde 2 of the plan."""
+
+    def test_decoder_imports_without_mfa_dependencies(self):
+        import auth_middleware
+        # The middleware's verification function exists and is callable.
+        assert callable(getattr(auth_middleware, "get_current_user_from_token", None))
+
+    def test_middleware_source_has_no_mfa_claim_check(self):
+        """The middleware source must not reference ``mfa`` claims directly."""
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "auth_middleware.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        # Allow incidental occurrences (e.g. comments); but never a claim lookup.
+        assert "payload.get('mfa'" not in source
+        assert 'payload.get("mfa"' not in source
+        assert "claims['mfa'" not in source
+        assert 'claims["mfa"' not in source
+
+
+class TestMfaModelsCoexistWithUserModels:
+    def test_models_user_module_unchanged_pydantic_shape(self):
+        from models import user as user_mod
+        # Ensure that User / UserUpdate / LoginRequest still load and still
+        # don't expose mfa-related fields (kept in models.mfa).
+        for cls in (user_mod.User, user_mod.UserUpdate, user_mod.LoginRequest):
+            fields = set(cls.model_fields.keys())
+            assert not {"mfa_enabled", "mfa_required", "mfa_token"} & fields, (
+                f"{cls.__name__} unexpectedly exposes MFA field; should stay byte-identical."
+            )
+
+    def test_models_mfa_module_exposes_expected_models(self):
+        from models import mfa as mfa_mod
+        for name in (
+            "MfaVerifyRequest",
+            "MfaEnrollStartResponse",
+            "MfaEnrollConfirmRequest",
+            "MfaEnrollConfirmResponse",
+            "MfaDisableRequest",
+            "MfaRegenerateBackupRequest",
+            "MfaRegenerateBackupResponse",
+            "MfaAdminResetRequest",
+            "MfaAdminResetAllRequest",
+            "MfaStatusResponse",
+        ):
+            assert hasattr(mfa_mod, name), f"Missing model: {name}"
+
+
+class TestRouterIncluded:
+    def test_main_includes_mfa_router(self):
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "main.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        assert "from routers.mfa import router as mfa_router" in source
+        assert "app.include_router(mfa_router)" in source
+
+
+class TestLoginResponseShapeForNonMfaUser:
+    """When MFA columns are missing or mfa_enabled=FALSE, /login returns the
+    pre-MFA response shape — no ``mfa_required`` / ``mfa_token`` keys leak through.
+    """
+
+    def test_login_without_mfa_returns_legacy_shape(self):
+        from fastapi.testclient import TestClient
+        from main import app
+
+        client = TestClient(app)
+
+        async def _fetch_mfa_state_none(conn, user_id):
+            return None
+
+        async def _no_log(*args, **kwargs):
+            return None
+
+        fake_user = {
+            "id": 1,
+            "username": "admin",
+            "email": "admin@example.com",
+            "password_hash": "$2b$12$placeholder",
+            "is_active": True,
+            "role": "admin",
+            "created_at": None,
+            "updated_at": None,
+            "last_login_at": None,
+        }
+
+        mock_conn = MagicMock()
+        mock_conn.fetchrow = AsyncMock(return_value=fake_user)
+        mock_conn.fetch = AsyncMock(return_value=[])
+        mock_conn.execute = AsyncMock(return_value=None)
+
+        async def _get_conn():
+            return mock_conn
+
+        async def _close(conn):
+            return None
+
+        with patch(
+            "routers.auth.get_database_connection", _get_conn
+        ), patch("routers.auth.close_database_connection", _close), patch(
+            "routers.auth._fetch_mfa_state", _fetch_mfa_state_none
+        ), patch("routers.auth.log_user_activity", _no_log), patch(
+            "bcrypt.checkpw", return_value=True
+        ):
+            resp = client.post(
+                "/api/auth/login",
+                json={"username": "admin", "password": "anything"},
+            )
+
+        assert resp.status_code == 200, resp.text
+        body = resp.json()
+        assert "access_token" in body
+        assert "token_type" in body
+        assert "expires_in" in body
+        assert "user" in body
+        assert "roles" in body
+        assert "permissions" in body
+        # CRITICAL — pre-MFA contract must not be polluted with MFA fields.
+        assert "mfa_required" not in body
+        assert "mfa_token" not in body
+        assert "methods" not in body

backend/tests/test_mfa_rate_limit_key.py

@@ -0,0 +1,200 @@
+"""Tests for middleware.mfa_rate_limit_key — user-aware + ingress-aware key."""
+import importlib
+from datetime import datetime, timedelta
+from typing import Dict, Optional
+
+import pytest
+from fastapi import Request
+from jose import jwt
+
+
+def _make_request(
+    headers: Optional[Dict[str, str]] = None,
+    peer: str = "127.0.0.1",
+) -> Request:
+    """Tiny ASGI scope shim — enough for the key_func surface."""
+    raw_headers = []
+    if headers:
+        raw_headers = [
+            (k.encode("latin-1"), v.encode("latin-1")) for k, v in headers.items()
+        ]
+    scope = {
+        "type": "http",
+        "headers": raw_headers,
+        "client": (peer, 12345),
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    return Request(scope)
+
+
+@pytest.fixture
+def reload_key(monkeypatch):
+    """Reload the key module so MFA_TRUSTED_PROXY_CIDRS is re-parsed."""
+
+    def _reload(**env):
+        for var in ("MFA_TRUSTED_PROXY_CIDRS",):
+            monkeypatch.delenv(var, raising=False)
+        for k, v in env.items():
+            monkeypatch.setenv(k, v)
+        from middleware import mfa_rate_limit_key as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+# ---------------------------------------------------------------------------
+# User-aware key extraction
+# ---------------------------------------------------------------------------
+
+
+def _mint_jwt(user_id, claim: str = "user_id") -> str:
+    """Mint a test JWT. Note: RFC 7519 says ``sub`` is a StringOrURI,
+    and python-jose validates that type when decoding, so callers that use
+    ``claim='sub'`` must pass a string user_id (matches production behavior
+    where auth_middleware also accepts string ``sub``)."""
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        claim: user_id,
+        "exp": datetime.utcnow() + timedelta(minutes=10),
+    }
+    return jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+
+def test_user_aware_via_user_id_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt(42, claim="user_id")
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:42"
+
+
+def test_user_aware_via_sub_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt("7", claim="sub")  # JWT spec: sub is a string
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:7"
+
+
+def test_no_auth_header_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_missing_bearer_prefix_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(headers={"authorization": "abc.def.ghi"}, peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_bearer_null_or_undefined_falls_back_to_ip(reload_key):
+    m = reload_key()
+    for bogus in ("null", "undefined", "", "   "):
+        req = _make_request(
+            headers={"authorization": f"Bearer {bogus}"}, peer="198.51.100.9"
+        )
+        assert m.mfa_rate_limit_key(req) == "ip:198.51.100.9"
+
+
+def test_tampered_jwt_falls_back_to_ip(reload_key):
+    """A token with a forged signature must NOT be honored — fallback to IP."""
+    m = reload_key()
+    bad = "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjogMTIzfQ.NOT_A_VALID_SIGNATURE"
+    req = _make_request(headers={"authorization": f"Bearer {bad}"}, peer="10.1.2.3")
+    assert m.mfa_rate_limit_key(req) == "ip:10.1.2.3"
+
+
+def test_expired_jwt_falls_back_to_ip(reload_key):
+    m = reload_key()
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        "user_id": 9,
+        "exp": datetime.utcnow() - timedelta(minutes=5),
+    }
+    expired = jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+    req = _make_request(headers={"authorization": f"Bearer {expired}"}, peer="10.0.0.7")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.7"
+
+
+# ---------------------------------------------------------------------------
+# Trusted-proxy X-Forwarded-For handling
+# ---------------------------------------------------------------------------
+
+
+def test_untrusted_peer_xff_is_ignored(reload_key):
+    """X-Forwarded-For from an untrusted client cannot move buckets."""
+    m = reload_key()  # no trusted CIDRs
+    req = _make_request(
+        headers={"x-forwarded-for": "1.2.3.4"},
+        peer="203.0.113.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_trusted_peer_xff_is_honored(reload_key):
+    """Peer in trusted CIDR → first XFF hop becomes the bucket."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(
+        headers={"x-forwarded-for": "203.0.113.42, 10.0.0.99"},
+        peer="10.0.0.99",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.42"
+
+
+def test_trusted_cidr_multiple_ranges(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8, 172.16.0.0/12")
+    req = _make_request(
+        headers={"x-forwarded-for": "198.51.100.4"},
+        peer="172.16.5.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:198.51.100.4"
+
+
+def test_trusted_peer_no_xff_falls_back_to_peer(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(peer="10.0.0.99")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.99"
+
+
+def test_invalid_cidr_in_env_is_logged_and_ignored(reload_key, caplog):
+    import logging
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limit_key"):
+        m = reload_key(MFA_TRUSTED_PROXY_CIDRS="not-a-cidr, 10.0.0.0/8")
+    assert any("ignoring invalid CIDR" in r.message for r in caplog.records)
+    # The valid one is still effective.
+    req = _make_request(
+        headers={"x-forwarded-for": "9.9.9.9"},
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:9.9.9.9"
+
+
+def test_user_bucket_wins_over_xff(reload_key):
+    """Auth always wins, even from a trusted proxy."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    token = _mint_jwt(99)  # integer user_id claim
+    req = _make_request(
+        headers={
+            "authorization": f"Bearer {token}",
+            "x-forwarded-for": "1.1.1.1",
+        },
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "user:99"
+
+
+def test_no_client_in_scope_does_not_crash(reload_key):
+    m = reload_key()
+    scope = {
+        "type": "http",
+        "headers": [],
+        "client": None,
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    req = Request(scope)
+    # Whatever it returns, it must be deterministic and not raise.
+    out = m.mfa_rate_limit_key(req)
+    assert out.startswith("ip:")

backend/tests/test_mfa_rate_limits.py

@@ -0,0 +1,93 @@
+"""Tests for middleware.mfa_rate_limits — env-driven MFA rate-limit config."""
+import importlib
+import logging
+
+import pytest
+
+
+@pytest.fixture
+def reload_module(monkeypatch):
+    """Helper: reload the module after env mutation so dataclass defaults
+    pick up the new values."""
+
+    def _reload(**env):
+        for key in list(globals().get('_OVERRIDDEN_ENVS', set())):
+            monkeypatch.delenv(key, raising=False)
+        for key, value in env.items():
+            monkeypatch.setenv(key, value)
+        from middleware import mfa_rate_limits as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+def test_defaults_when_no_env(reload_module, monkeypatch):
+    """No env var set → secure defaults applied (user-aware key assumption)."""
+    for key in (
+        "MFA_RATE_LIMIT_ENROLL_START",
+        "MFA_RATE_LIMIT_ENROLL_CONFIRM",
+        "MFA_RATE_LIMIT_DISABLE",
+        "MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES",
+        "MFA_RATE_LIMIT_ADMIN_RESET",
+        "MFA_RATE_LIMIT_ADMIN_RESET_ALL",
+    ):
+        monkeypatch.delenv(key, raising=False)
+    m = reload_module()
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert m.MFA_LIMITS.enroll_confirm == "10/minute"
+    assert m.MFA_LIMITS.disable == "10/minute"
+    assert m.MFA_LIMITS.regenerate_backup_codes == "5/hour"
+    assert m.MFA_LIMITS.admin_reset == "60/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "1/day"
+
+
+def test_env_override_per_endpoint(reload_module):
+    m = reload_module(
+        MFA_RATE_LIMIT_ENROLL_START="100/hour",
+        MFA_RATE_LIMIT_ADMIN_RESET_ALL="3/day",
+    )
+    assert m.MFA_LIMITS.enroll_start == "100/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "3/day"
+    # Untouched values still default.
+    assert m.MFA_LIMITS.disable == "10/minute"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "totally-bogus",
+        "5/lightyear",
+        "abc/minute",
+        "5",
+        "/minute",
+        "5//minute",
+        "",
+    ],
+)
+def test_invalid_format_falls_back_to_default(reload_module, caplog, bad):
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limits"):
+        m = reload_module(MFA_RATE_LIMIT_ENROLL_START=bad)
+    # Falls back to the secure default for enroll_start.
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert any("not a valid slowapi limit string" in r.message for r in caplog.records)
+
+
+def test_whitespace_around_value_is_tolerated(reload_module):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE="  30/minute  ")
+    assert m.MFA_LIMITS.disable == "30/minute"
+
+
+@pytest.mark.parametrize(
+    "valid",
+    ["1/second", "100/minute", "1000/hour", "10/day"],
+)
+def test_all_valid_periods_accepted(reload_module, valid):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE=valid)
+    assert m.MFA_LIMITS.disable == valid
+
+
+def test_dataclass_is_frozen(reload_module):
+    """Frozen dataclass guards against accidental mutation after import."""
+    m = reload_module()
+    with pytest.raises((AttributeError, Exception)):
+        m.MFA_LIMITS.disable = "999/second"  # type: ignore[misc]

backend/tests/test_mfa_service.py

@@ -0,0 +1,223 @@
+"""Unit tests for the MFA service layer (Issue #18, v1.6.0).
+
+These tests cover the pure-Python side of MFA — no DB, no FastAPI.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+import re
+import sys
+import time
+
+import pytest
+
+# Repo path setup (mirrors other tests in this folder).
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import pyotp  # noqa: E402
+
+from services import mfa_service  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# TOTP
+# ---------------------------------------------------------------------------
+
+
+class TestTotpSecret:
+    def test_secret_is_base32(self):
+        secret = mfa_service.generate_totp_secret()
+        # pyotp.random_base32() returns 32-character base32 strings.
+        assert len(secret) == 32
+        assert re.fullmatch(r"[A-Z2-7]+", secret), "secret must be valid base32"
+
+    def test_secrets_are_unique(self):
+        secrets = {mfa_service.generate_totp_secret() for _ in range(50)}
+        assert len(secrets) == 50
+
+
+class TestVerifyTotp:
+    def setup_method(self):
+        self.secret = mfa_service.generate_totp_secret()
+        self.totp = pyotp.TOTP(self.secret, digits=6, interval=30, digest="sha1")
+
+    def test_happy_path(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        assert step == int(time.time()) // 30
+
+    def test_invalid_code_format_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "abc", None)
+        assert ok is False and step is None
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "12345", None)
+        assert ok is False and step is None
+
+    def test_tolerance_minus_30s(self):
+        now = int(time.time())
+        previous_step = (now // 30) - 1
+        prev_code = self.totp.at(previous_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, prev_code, None)
+        assert ok is True
+        assert step == previous_step
+
+    def test_tolerance_plus_30s(self):
+        now = int(time.time())
+        next_step = (now // 30) + 1
+        next_code = self.totp.at(next_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, next_code, None)
+        assert ok is True
+        assert step == next_step
+
+    def test_replay_rejected(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        # Submit again with the previously-consumed step: must be rejected.
+        ok2, step2 = mfa_service.verify_totp_with_replay_guard(self.secret, code, step)
+        assert ok2 is False
+        assert step2 is None
+
+    def test_wrong_code_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "000000", None)
+        assert ok is False
+        assert step is None
+
+
+# ---------------------------------------------------------------------------
+# Fernet + key resolution
+# ---------------------------------------------------------------------------
+
+
+class TestFernet:
+    def setup_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def teardown_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def test_encrypt_decrypt_roundtrip_with_env_key(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        key = Fernet.generate_key().decode()
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", key)
+        mfa_service.reset_fernet_for_tests()
+
+        secret = "JBSWY3DPEHPK3PXP" * 2
+        token = mfa_service.encrypt_secret(secret)
+        assert token and token != secret
+        recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+
+    def test_decrypt_invalid_token_returns_none(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", Fernet.generate_key().decode())
+        mfa_service.reset_fernet_for_tests()
+
+        assert mfa_service.decrypt_secret("not-a-valid-fernet-token") is None
+
+    def test_hkdf_fallback_when_env_unset(self, monkeypatch, caplog):
+        monkeypatch.delenv("MFA_ENCRYPTION_KEY", raising=False)
+        mfa_service.reset_fernet_for_tests()
+
+        with caplog.at_level("WARNING"):
+            secret = "JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP"
+            token = mfa_service.encrypt_secret(secret)
+            recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+        assert any("MFA_ENCRYPTION_KEY" in r.message for r in caplog.records), (
+            "expected a WARN log when falling back to SECRET_KEY derivation"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+class TestBackupCodes:
+    def test_generate_count_and_format(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(codes) == 10
+        # 31-char alphabet: A-H J K M N P-Z 2-9 (excludes I, L, O, 0, 1).
+        for code in codes:
+            assert re.fullmatch(r"[A-HJKM-NP-Z2-9]{4}-[A-HJKM-NP-Z2-9]{4}", code), code
+
+    def test_alphabet_excludes_confusing_characters(self):
+        # Generate enough codes to virtually guarantee any forbidden char would surface.
+        for _ in range(20):
+            codes = mfa_service.generate_backup_codes()
+            for code in codes:
+                for ch in code.replace("-", ""):
+                    assert ch not in "0O1IL", f"forbidden char {ch!r} in {code!r}"
+
+    def test_codes_are_unique(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(set(codes)) == len(codes)
+
+    def test_normalize_strips_case_dash_space(self):
+        assert mfa_service.normalize_backup_code("abcd-efgh") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code(" ab cd-ef gh ") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code("") == ""
+        assert mfa_service.normalize_backup_code(None) == ""  # type: ignore[arg-type]
+
+    def test_hash_and_check_async(self):
+        async def _run():
+            plain = mfa_service.generate_backup_codes()[:1]
+            hashes = await mfa_service.hash_backup_codes(plain)
+            assert len(hashes) == 1
+            assert await mfa_service.check_backup_code(plain[0], hashes[0]) is True
+            assert await mfa_service.check_backup_code("WRONG-CODE", hashes[0]) is False
+            # Case + dash normalization
+            assert await mfa_service.check_backup_code(plain[0].lower(), hashes[0]) is True
+            assert (
+                await mfa_service.check_backup_code(plain[0].replace("-", ""), hashes[0])
+                is True
+            )
+
+        asyncio.run(_run())
+
+
+# ---------------------------------------------------------------------------
+# otpauth URI
+# ---------------------------------------------------------------------------
+
+
+class TestOtpAuthUri:
+    def test_uri_shape(self):
+        uri = mfa_service.build_otpauth_uri("alice@example.com", "JBSWY3DPEHPK3PXP")
+        assert uri.startswith("otpauth://totp/")
+        assert "secret=JBSWY3DPEHPK3PXP" in uri
+        assert "issuer=" in uri
+        assert "algorithm=SHA1" in uri
+        assert "digits=6" in uri
+        assert "period=30" in uri
+
+    def test_account_label_env_override(self, monkeypatch):
+        monkeypatch.setenv("MFA_ACCOUNT_LABEL_DOMAIN", "ops.example.com")
+        label = mfa_service.build_account_label("alice", hostname_hint="ignored.com")
+        assert label == "alice@ops.example.com"
+
+    def test_account_label_hostname_hint(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint="api.local")
+        assert label == "alice@api.local"
+
+    def test_account_label_fallback(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint=None)
+        assert label == "alice@haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Challenge token
+# ---------------------------------------------------------------------------
+
+
+class TestChallengeToken:
+    def test_length_and_uniqueness(self):
+        tokens = {mfa_service.generate_challenge_token() for _ in range(50)}
+        assert len(tokens) == 50
+        for t in tokens:
+            assert len(t) == 64
+            assert re.fullmatch(r"[0-9a-f]{64}", t)

backend/tests/test_outbound_instrumentation.py

@@ -0,0 +1,327 @@
+"""v1.11.0: every outbound HTTP call is recorded, and instrumentation can never
+become the failure.
+
+Two independent risks:
+
+**Secrets.** The outbound calls carry the most sensitive material in the
+system: the ACME JWS (a replayable signed capability for the lifetime of its
+nonce) and the DNS provider API credentials. Those call sites must opt out of
+request-body capture and out of verbatim error text — the tests below assert
+that at the call site, not just in the helper.
+
+**Availability.** Both DNS provider funnels end in
+`except Exception: raise DnsProviderError("Unexpected ... failure")`, and in
+GoDaddy's publish path that reverts `dns_record_published` and stalls the ACME
+order. So an exception escaping `outbound_span` would be reported to the
+operator as a provider outage. It must never raise — and it must never swallow.
+"""
+import asyncio
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+from unittest.mock import patch  # noqa: E402
+
+from utils import http_instrumentation  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.http_instrumentation import (  # noqa: E402
+    TARGET_ACME,
+    TARGET_DNS_CLOUDFLARE,
+    TARGET_DNS_GODADDY,
+    outbound_span,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(*parts):
+    with open(os.path.join(_BACKEND, *parts), encoding="utf-8") as f:
+        return f.read()
+
+
+def _function_body(src, signature):
+    start = src.index(signature)
+    rest = src[start:]
+    # Next def at the same or lower indentation ends the body.
+    end = rest.find("\n    async def ", 1)
+    alt = rest.find("\n    def ", 1)
+    if alt != -1 and (end == -1 or alt < end):
+        end = alt
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    rows = []
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", rows.append)
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(http_instrumentation, "get_config", lambda: request_log_settings._CACHE)
+    return rows
+
+
+# --------------------------------------------------------------------------
+# outbound_span behaviour
+# --------------------------------------------------------------------------
+
+def test_records_a_successful_call(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="POST",
+                                 url="https://acme-v02.api.letsencrypt.org/acme/new-order") as span:
+            span.set_response(201, {"content-type": "application/json"}, {"status": "pending"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.direction == "outbound"
+    assert row.target == TARGET_ACME
+    assert row.method == "POST"
+    assert row.status_code == 201
+    assert row.status_class == 2
+    assert row.response_body_value == {"status": "pending"}
+
+
+def test_exception_is_recorded_and_reraised_unchanged(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com/x"):
+            raise ValueError("connection reset")
+
+    with pytest.raises(ValueError, match="connection reset"):
+        asyncio.run(run())
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a call that never got a response must be status_class 0 — the sentinel the "
+        "error-retention window keys off"
+    )
+    assert row.error.startswith("ValueError")
+
+
+def test_safe_error_only_records_the_type_not_the_message(captured):
+    async def run():
+        async with outbound_span(target=TARGET_DNS_GODADDY, method="PUT",
+                                 url="https://api.godaddy.com/v1/domains/example.com/records/TXT/_acme-challenge",
+                                 safe_error_only=True):
+            raise RuntimeError("failed talking to https://api.godaddy.com/v1/domains/secret-zone")
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(run())
+
+    assert captured[0].error == "RuntimeError"
+    assert "secret-zone" not in (captured[0].error or "")
+
+
+def test_instrumentation_failure_never_becomes_a_provider_failure(captured, monkeypatch):
+    """A bug in row construction must not surface to the operator as
+    'Unexpected GoDaddy API failure' and stall an ACME order."""
+    def explode(row):
+        raise RuntimeError("sink is broken")
+
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", explode)
+
+    async def run():
+        async with outbound_span(target=TARGET_DNS_CLOUDFLARE, method="GET",
+                                 url="https://api.cloudflare.com/client/v4/zones") as span:
+            span.set_response(200, {}, {"success": True})
+            return "provider-result"
+
+    assert asyncio.run(run()) == "provider-result", (
+        "a broken sink propagated out of outbound_span; both DNS funnels would convert "
+        "that into DnsProviderError('Unexpected ... failure'), and in GoDaddy's publish "
+        "path that reverts dns_record_published and stalls the ACME order"
+    )
+
+
+def test_block_exception_still_propagates_when_the_sink_is_broken(monkeypatch):
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer",
+                        lambda row: (_ for _ in ()).throw(RuntimeError("sink is broken")))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com"):
+            raise KeyError("original")
+
+    with pytest.raises(KeyError, match="original"):
+        asyncio.run(run())
+
+
+def test_capture_body_false_stores_the_summary_not_the_payload(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_ACME, method="POST", url="https://acme/new-order",
+            request_body={"jws": True, "kid_present": True, "payload_empty": False},
+            capture_body=False,
+        ) as span:
+            span.set_response(200, {}, {"status": "valid"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.request_body_value == {"jws": True, "kid_present": True, "payload_empty": False}
+    assert row.request_body_raw is None
+    # The CA's RESPONSE is still captured — that is the half operators need.
+    assert row.response_body_value == {"status": "valid"}
+
+
+def test_urls_are_scrubbed_before_storage(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_DNS_CLOUDFLARE, method="GET",
+            url="https://user:hunter2@api.cloudflare.com/client/v4/zones?api_key=abc&page=1",
+        ) as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+
+    url = captured[0].url
+    assert "hunter2" not in url
+    assert "abc" not in url
+    assert "page=1" in url
+
+
+def test_outbound_rows_inherit_the_inbound_request_id(captured):
+    from utils.request_log_sink import request_id_context
+
+    async def run():
+        token = request_id_context.set("abc123def456")
+        try:
+            async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+                span.set_response(200, {}, {})
+        finally:
+            request_id_context.reset(token)
+
+    asyncio.run(run())
+
+    assert captured[0].request_id == "abc123def456", (
+        "an outbound call must carry the inbound request's id, otherwise the detail view "
+        "cannot show which API call triggered which CA/DNS call"
+    )
+
+
+def test_background_calls_get_a_task_scoped_id(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured[0].request_id.startswith("bg:")
+
+
+def test_disabled_outbound_capture_produces_no_row(captured, monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE",
+                        replace(DEFAULT_CONFIG, capture_outbound=False))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            # The call site keeps working — set_response must still be callable.
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured == []
+
+
+def test_set_response_tolerates_a_response_without_headers(captured):
+    """Some call sites are driven in tests by minimal fakes exposing only
+    `.status`."""
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="HEAD", url="https://acme/nonce") as span:
+            span.set_response(200, None)
+
+    asyncio.run(run())
+    assert captured[0].status_code == 200
+
+
+# --------------------------------------------------------------------------
+# Call-site coverage
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path,target", [
+    (("services", "acme_service.py"), "TARGET_ACME"),
+    (("services", "acme_diagnostics.py"), "TARGET_ACME_DIAG"),
+    (("services", "dns_providers", "cloudflare.py"), "TARGET_DNS_CLOUDFLARE"),
+    (("services", "dns_providers", "godaddy.py"), "TARGET_DNS_GODADDY"),
+    (("routers", "letsencrypt.py"), "TARGET_LETSENCRYPT_CA"),
+    (("routers", "settings.py"), "TARGET_SETTINGS_PROBE"),
+    (("haproxy_client.py",), "TARGET_HAPROXY_STATS"),
+    (("agent_notifications.py",), "TARGET_AGENT"),
+])
+def test_every_outbound_module_is_instrumented(path, target):
+    src = _read(*path)
+    assert "outbound_span(" in src, f"{'/'.join(path)} makes HTTP calls but records nothing"
+    assert target in src, f"{'/'.join(path)} does not tag its rows with {target}"
+
+
+def test_acme_signed_request_never_captures_the_jws_body():
+    """The JWS body is {protected, payload, signature}: `protected` carries the
+    nonce and account kid, `signature` is made with the account private key. A
+    stored (protected, signature) pair is a replayable ACME credential."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    assert "capture_body=False" in body, (
+        "the ACME JWS request body would be written to request_logs verbatim — that is a "
+        "replayable signed credential sitting in an audit table"
+    )
+    assert '"jws": True' in body, "no synthetic summary replaces the suppressed JWS body"
+
+
+def test_acme_span_is_inside_the_retry_loop():
+    """The session is built outside `for attempt in range(3)`; the span must be
+    inside it, so a badNonce retry is its own row rather than being folded into
+    the successful attempt."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    loop_at = body.index("for attempt in range(3):")
+    span_at = body.index("async with outbound_span(")
+    assert loop_at < span_at, (
+        "outbound_span wraps the retry loop instead of sitting inside it, so three "
+        "attempts collapse into one log row and a nonce retry becomes invisible"
+    )
+
+
+@pytest.mark.parametrize("path", [
+    ("services", "dns_providers", "cloudflare.py"),
+    ("services", "dns_providers", "godaddy.py"),
+])
+def test_dns_providers_record_error_types_only(path):
+    src = _read(*path)
+    body = _function_body(src, "    async def _request(")
+    assert "safe_error_only=True" in body, (
+        f"{'/'.join(path)} would record the full exception text, which can carry the "
+        f"request URL and through it the tenant/zone identifier"
+    )
+
+
+def test_godaddy_narrow_value_error_handling_is_preserved():
+    """R-round hardening: only a JSON decode failure may be swallowed. Widening
+    it would make a mid-read transport failure look like an empty RRset, and the
+    follow-up full-RRset PUT would then destroy coexisting TXT values."""
+    src = _read("services", "dns_providers", "godaddy.py")
+    body = _function_body(src, "    async def _request(")
+    assert "except ValueError:" in body
+    assert "except Exception:\n                        body = None" not in body
+
+
+def test_acme_diagnostics_keeps_its_ipv4_pinned_connector():
+    """Duplicates an existing assertion on purpose: instrumenting this module
+    must not have refactored the SSRF-guard connector away."""
+    src = _read("services", "acme_diagnostics.py")
+    assert "TCPConnector(family=socket.AF_INET" in src, (
+        "the IPv4 pin was removed from the port-80 probe — that reopens the dual-stack "
+        "AAAA bypass the SSRF guard closes"
+    )
+
+
+def test_haproxy_stats_never_logs_basic_auth_or_the_csv():
+    """aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+    password, and a full stats CSV has no audit value."""
+    src = _read("haproxy_client.py")
+    body = _function_body(src, "    async def _get_stats_via_http(")
+    assert "capture_body=False" in body
+    assert "capture_response_body=False" in body
+    assert "auth=auth" in body and "request_body=auth" not in body

backend/tests/test_request_log_fleet_scale.py

@@ -0,0 +1,327 @@
+"""v1.11.0: the log's cost must follow operator activity, not fleet size.
+
+Every property here was a real defect measured on the feature branch, and each
+one only shows up at scale or at the edge of a setting's documented range, which
+is why none of them were caught by the rule-level tests.
+
+  * one row per API call becomes millions per day once the fleet is a few
+    hundred nodes, and the row cap then evicts the forensic history the feature
+    exists for;
+  * the operator role could not see the rows its grant was written for;
+  * every background call ever made shared one correlation id;
+  * queue memory was a function of an operator-editable setting, not a limit.
+"""
+import asyncio
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils.http_instrumentation import _correlation_id, begin_background_trace  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    get_config,
+    set_config,
+)
+from utils.request_log_sink import (  # noqa: E402
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    RequestLogSink,
+    request_id_context,
+)
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MIDDLEWARE = os.path.join(_BACKEND, "middleware", "request_logger.py")
+
+
+@pytest.fixture(autouse=True)
+def _restore_config():
+    """These tests mutate the module-global snapshot; put it back."""
+    before = get_config()
+    yield
+    set_config(before)
+
+
+def _row(**kw):
+    kw.setdefault("request_id", "a" * 32)
+    kw.setdefault("direction", "inbound")
+    kw.setdefault("method", "GET")
+    kw.setdefault("url", "/api/agents/prod-lb-1/config")
+    kw.setdefault("status_code", 200)
+    return RequestLogRow(**kw)
+
+
+class _CountingSink(RequestLogSink):
+    """Counts what survives `offer()` without needing an event loop."""
+
+    def __init__(self, **kw):
+        super().__init__(kw.pop("maxsize", 10000), 100, 500, **kw)
+        self.accepted = []
+
+    def _ensure_queue(self):
+        sink = self
+
+        class _Q:
+            def put_nowait(self, row):
+                sink.accepted.append(row)
+
+            def qsize(self):
+                return len(sink.accepted)
+
+        return _Q()
+
+
+# --------------------------------------------------------------------------
+# Volume: successful agent polls are not rows
+# --------------------------------------------------------------------------
+
+def test_successful_agent_polls_are_dropped_by_default():
+    """~9 800 rows/day PER AGENT, all of them 200s meaning "nothing changed".
+
+    At 200 nodes that is ~2M rows/day and the 500 000 row cap is reached in
+    about six hours, so the configured "7 days of successes, 30 days of
+    failures" silently becomes about six hours of each — for everything in the
+    table, not just for the agent rows.
+    """
+    assert DEFAULT_CONFIG.capture_agent_success is False, (
+        "the default must be off; on, the table's size is a function of node "
+        "count rather than of anything anyone did"
+    )
+    sink = _CountingSink()
+    for _ in range(100):
+        sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert sink.accepted == []
+
+
+@pytest.mark.parametrize("status", [401, 422, 500, None])
+def test_failed_agent_calls_are_always_kept(status):
+    """The half an operator actually needs, and rare enough to be free.
+
+    `None` is a transport error with no HTTP response at all, which
+    status_class reports as 0.
+    """
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=status))
+    assert len(sink.accepted) == 1, f"a {status} agent call must be recorded"
+
+
+def test_operator_traffic_is_unaffected_by_the_agent_gate():
+    sink = _CountingSink()
+    sink.offer(_row(target=None, status_code=200, user_id=7))
+    assert len(sink.accepted) == 1
+
+
+def test_the_gate_can_be_turned_on_for_debugging():
+    set_config(replace(get_config(), capture_agent_success=True))
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert len(sink.accepted) == 1
+
+
+def test_agent_traffic_is_identified_by_headers_not_by_a_database_lookup():
+    """The hot path runs on every request; a lookup per call is not affordable.
+
+    The installed agent sends `X-API-Key` and never `Authorization`; the UI
+    sends a JWT and never an agent key.
+    """
+    from middleware.request_logger import _is_agent_call
+
+    def scope(headers):
+        return {"type": "http", "headers": [(k.encode(), v.encode()) for k, v in headers.items()]}
+
+    assert _is_agent_call(scope({"x-api-key": "agt_x"})) is True
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z"})) is False
+    # generate-install-script accepts either; self-upgrade sends only the key.
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z", "x-api-key": "agt_x"})) is False
+    assert _is_agent_call(scope({})) is False
+
+
+def test_agent_gate_does_not_reach_for_a_connection():
+    """`offer()` is called from the request coroutine and must stay pure."""
+    src = open(os.path.join(_BACKEND, "utils", "request_log_sink.py"), encoding="utf-8").read()
+    body = src.split("def offer(", 1)[1].split("\n    # -- consumer", 1)[0]
+    for forbidden in ("await ", "get_database_connection", "fetch"):
+        assert forbidden not in body, f"offer() must not {forbidden.strip()!r} — it runs on the hot path"
+
+
+# --------------------------------------------------------------------------
+# Visibility: the operator grant has to mean something
+# --------------------------------------------------------------------------
+
+def test_read_only_scoping_admits_agent_rows_but_not_other_users():
+    """`operator` holds requestlog.read to "debug failing applies" — but an
+    apply fails on the NODE, and the node reports over its own API key, so that
+    row has user_id NULL and own-rows-only scoping hid it.
+
+    Keyed on `target`, NOT on `user_id IS NULL`: anonymous traffic (failed
+    logins and their usernames, unauthenticated probes) is not agent traffic
+    and must stay admin-only.
+    """
+    src = open(_ROUTER, encoding="utf-8").read()
+    clause = re.search(r"if not can_manage:(.*?)where_sql =", src, re.S)
+    assert clause, "the self-scoping block moved; re-check this test"
+    # Code only: the comment above the clause explains what it deliberately
+    # does NOT do, and would otherwise match the negative assertion below.
+    body = "\n".join(
+        line for line in clause.group(1).splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "TARGET_INBOUND_AGENT" in body, "agent rows are still hidden from requestlog.read"
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL would also expose anonymous traffic, including failed "
+        "logins and the usernames they carry"
+    )
+
+
+def test_detail_endpoint_uses_the_same_scoping_rule_as_the_list():
+    src = open(_ROUTER, encoding="utf-8").read()
+    detail = src.split('@router.get("/{log_id}")', 1)[1]
+    assert "TARGET_INBOUND_AGENT" in detail, (
+        "the detail endpoint would 404 on the very rows the list now shows"
+    )
+
+
+@pytest.mark.parametrize("decorator", [
+    '@router.get("/settings")', '@router.put("/settings")',
+    '@router.get("/stats")', '@router.post("/purge")',
+    '@router.get("")', '@router.get("/{log_id}")',
+])
+def test_permission_is_enforced_before_the_try_block(decorator):
+    """The repo's GHSA-3p5c pattern: a permission check inside `try` gets
+    swallowed by the handler's own `except Exception -> 500`, turning a 403
+    into a server error and, worse, hiding that the check ran at all."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    body = src.split(decorator, 1)[1]
+    body = body.split("\n@router.")[0]
+    require_at = body.find("_require(authorization")
+    try_at = body.find("\n    try:")
+    assert require_at != -1, f"{decorator} does not call _require at all"
+    assert try_at == -1 or require_at < try_at, (
+        f"{decorator} checks permissions INSIDE its try block"
+    )
+
+
+# --------------------------------------------------------------------------
+# Correlation: a trace that groups the wrong rows is worse than no trace
+# --------------------------------------------------------------------------
+
+def test_each_background_pass_gets_its_own_correlation_id():
+    """Nothing in main.py names its tasks, so the old `bg:<task name>` fallback
+    gave one long-lived loop a single id for its entire life — measured, 15
+    ACME calls across 5 ticks came out as 1 id. `related` (LIMIT 100) then
+    presents up to a hundred unrelated calls as this request's trace.
+    """
+    async def loop():
+        per_tick = []
+        for _ in range(5):
+            begin_background_trace("acme_renewals")
+            per_tick.append([_correlation_id() for _ in range(3)])
+            await asyncio.sleep(0)
+        return per_tick
+
+    ticks = asyncio.run(loop())
+    for tick in ticks:
+        assert len(set(tick)) == 1, "calls within one pass must share an id"
+    ids = [t[0] for t in ticks]
+    assert len(set(ids)) == 5, f"passes must not share an id, got {ids}"
+
+
+def test_unwrapped_background_code_does_not_collapse_onto_one_id():
+    """Erring toward too little grouping: a row that stands alone is honest, a
+    row falsely grouped with a hundred others is not."""
+    async def unwrapped():
+        request_id_context.set(None)
+        return [_correlation_id() for _ in range(4)]
+
+    ids = asyncio.run(unwrapped())
+    assert len(set(ids)) == 4
+
+
+def test_the_background_loops_that_make_outbound_calls_open_a_trace():
+    src = open(os.path.join(_BACKEND, "main.py"), encoding="utf-8").read()
+    for loop_name in ("complete_pending_acme_orders", "check_letsencrypt_renewals",
+                      "monitor_agent_status"):
+        body = src.split(f"async def {loop_name}", 1)[1].split("\nasync def ")[0]
+        assert "begin_background_trace(" in body, (
+            f"{loop_name} makes outbound calls but never opens a per-pass trace"
+        )
+
+
+# --------------------------------------------------------------------------
+# Memory: a limit, not a setting
+# --------------------------------------------------------------------------
+
+def test_queue_memory_is_bounded_even_at_the_max_body_size_ceiling():
+    """`max_body_bytes` is editable from Settings and its documented ceiling is
+    256 KB, which a row carries twice. Against the default 2 000-row queue that
+    is ~1 GiB — the entire pod limit — reachable from in-range values.
+    """
+    set_config(replace(get_config(), max_body_bytes=262144, capture_agent_success=True))
+    budget = 8 * 1024 * 1024
+    sink = _CountingSink(maxsize=2000, max_bytes=budget)
+    blob = b"x" * 262144
+    for _ in range(2000):
+        sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+
+    held = sum(r.queue_weight() for r in sink.accepted)
+    assert held <= budget, f"queue held {held} bytes against a {budget} byte budget"
+    assert sink.stats["dropped"] > 0, "over-budget rows must be dropped, and counted"
+    unbounded = 2000 * (2 * 262144 + 1400)
+    assert held < unbounded / 10, (
+        f"without the byte budget this queue would hold {unbounded // 1024 // 1024} MiB"
+    )
+
+
+_DRAIN_ROWS = 50
+
+
+def test_the_byte_budget_is_released_as_rows_drain():
+    """A budget that only ever counts up is a slow leak, not a limit.
+
+    Deliberately time-INDEPENDENT. `_collect()` stops at whichever comes first,
+    `batch_size` rows or the `flush_ms` deadline, so a batch size larger than
+    the row count makes the result a function of how fast the runner happens to
+    be. The first version of this test used batch=100/flush=10ms for 50 rows and
+    passed on a native build while failing in CI, which builds
+    linux/amd64 + linux/arm64 and therefore runs one of them under qemu
+    emulation: 25 iterations of `asyncio.wait_for` were enough to exhaust 10 ms
+    there, `_collect()` returned half a batch, and the assertion read
+    `239800 == 0` - measuring the scheduler, not the sink.
+
+    So: batch size EQUAL to the row count, so the loop exits on the count and
+    never consults the deadline; a generous flush window in case it somehow
+    does; and a drain loop rather than a single call. Nothing here depends on
+    wall-clock speed.
+    """
+    async def drain():
+        sink = RequestLogSink(2000, _DRAIN_ROWS, 5000, max_bytes=8 * 1024 * 1024)
+        blob = b"x" * 4096
+        set_config(replace(get_config(), capture_agent_success=True))
+        for _ in range(_DRAIN_ROWS):
+            sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+        assert sink.stats["queued_bytes"] > 0, "nothing was queued, so nothing is being measured"
+        assert sink._queue.qsize() == _DRAIN_ROWS, "the queue did not take every row"
+
+        drained = 0
+        while not sink._queue.empty():
+            drained += len(await sink._collect())
+        assert drained == _DRAIN_ROWS, f"drained {drained} of {_DRAIN_ROWS} rows"
+        return sink.stats["queued_bytes"]
+
+    assert asyncio.run(drain()) == 0
+
+
+def test_stats_say_the_sink_counters_are_per_worker():
+    """The sink is a module global; with UVICORN_WORKERS > 1 each process keeps
+    its own. A number that looks fleet-wide but is not understates drops by
+    exactly the worker count."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    assert '"scope"' in src.split('"sink"', 1)[1][:400], (
+        "the stats response must label the sink counters as this-worker-only"
+    )

backend/tests/test_request_log_middleware.py

@@ -0,0 +1,432 @@
+"""v1.11.0: the request/response logger must be invisible to everything below it.
+
+This is the highest-risk piece of the feature. A logging middleware that reads
+the request body the naive way DRAINS the ASGI receive channel, and the handler
+underneath then sees an empty body — `POST /api/agents/heartbeat` reads the raw
+stream itself, so every agent in the fleet would start failing its heartbeat
+because someone wanted nicer logs.
+
+The implementation therefore TEES rather than consumes. These tests drive the
+middleware over a stub ASGI app and assert that property directly: the
+downstream app sees the full body, the client sees the full response, and only
+a capped copy is kept.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from middleware.request_logger import RequestResponseLogMiddleware  # noqa: E402
+from utils.logging_config import correlation_id_context  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+
+
+_MAIN = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "main.py")
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    """Collect the rows the middleware hands to the sink, instead of writing them."""
+    rows = []
+    monkeypatch.setattr(sink_module.request_log_sink, "offer", rows.append)
+    # The middleware imports `request_log_sink` by value, so patch there too.
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl.request_log_sink, "offer", rows.append)
+    return rows
+
+
+@pytest.fixture(autouse=True)
+def default_config(monkeypatch):
+    """Every test starts from the shipped defaults, with a small body cap so the
+    truncation paths are exercised without megabyte fixtures."""
+    cfg = replace(DEFAULT_CONFIG, max_body_bytes=1024)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl, "get_config", lambda: request_log_settings._CACHE)
+    return cfg
+
+
+def set_config(monkeypatch, **overrides):
+    cfg = replace(request_log_settings._CACHE, **overrides)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    return cfg
+
+
+# --------------------------------------------------------------------------
+# A minimal ASGI harness — no TestClient, no HTTP stack, just the protocol.
+# --------------------------------------------------------------------------
+
+async def drive(app, *, method="POST", path="/api/backends", body=b"", query=b"",
+                headers=None, content_type="application/json"):
+    """Run one request through `app` and return (status, headers, body)."""
+    raw_headers = [(b"host", b"testserver")]
+    if content_type:
+        raw_headers.append((b"content-type", content_type.encode()))
+    for k, v in (headers or {}).items():
+        raw_headers.append((k.encode().lower(), v.encode()))
+
+    scope = {
+        "type": "http",
+        "asgi": {"version": "3.0"},
+        "http_version": "1.1",
+        "method": method,
+        "scheme": "http",
+        "path": path,
+        "raw_path": path.encode(),
+        "query_string": query,
+        "root_path": "",
+        "headers": raw_headers,
+        "client": ("10.1.2.3", 51234),
+        "server": ("testserver", 80),
+    }
+
+    # Deliver the body in three chunks so the tee is exercised across messages.
+    chunks = [body[i:i + max(1, len(body) // 3 or 1)] for i in range(0, len(body), max(1, len(body) // 3 or 1))] or [b""]
+    pending = list(chunks)
+
+    async def receive():
+        if pending:
+            chunk = pending.pop(0)
+            return {"type": "http.request", "body": chunk, "more_body": bool(pending)}
+        return {"type": "http.request", "body": b"", "more_body": False}
+
+    sent = {"status": None, "headers": [], "body": b""}
+
+    async def send(message):
+        if message["type"] == "http.response.start":
+            sent["status"] = message["status"]
+            sent["headers"] = message.get("headers", [])
+        elif message["type"] == "http.response.body":
+            sent["body"] += message.get("body", b"") or b""
+
+    await app(scope, receive, send)
+    return sent
+
+
+def echo_length_app(status=200, content_type=b"application/json"):
+    """Stub app that CONSUMES the whole request body and reports its length.
+
+    This is the regression shape: if the middleware drained the stream, the app
+    below it would see 0 bytes.
+    """
+    async def app(scope, receive, send):
+        total = 0
+        while True:
+            message = await receive()
+            total += len(message.get("body", b"") or b"")
+            if not message.get("more_body"):
+                break
+        payload = json.dumps({"received_bytes": total}).encode()
+        await send({"type": "http.response.start", "status": status,
+                    "headers": [(b"content-type", content_type)]})
+        await send({"type": "http.response.body", "body": payload})
+    return app
+
+
+def chunked_app(chunks, content_type=b"application/json"):
+    async def app(scope, receive, send):
+        await send({"type": "http.response.start", "status": 200,
+                    "headers": [(b"content-type", content_type)]})
+        for i, chunk in enumerate(chunks):
+            await send({"type": "http.response.body", "body": chunk,
+                        "more_body": i < len(chunks) - 1})
+    return app
+
+
+# --------------------------------------------------------------------------
+# The transparency guarantees
+# --------------------------------------------------------------------------
+
+def test_request_body_reaches_downstream_intact(captured):
+    """THE regression guard: draining the receive channel would break the raw-body
+    agent heartbeat handler."""
+    body = b"x" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=body))
+
+    assert json.loads(sent["body"])["received_bytes"] == 100_000, (
+        "the handler below the logger saw a different body length than the client sent — "
+        "the middleware consumed the receive channel instead of teeing it"
+    )
+
+
+def test_response_body_reaches_client_intact(captured):
+    chunks = [b'{"part":', b'"one",', b'"n":2}']
+    app = RequestResponseLogMiddleware(chunked_app(chunks))
+
+    sent = asyncio.run(drive(app, method="GET", body=b""))
+
+    assert sent["body"] == b"".join(chunks), "a response chunk was swallowed by the logger"
+    assert sent["status"] == 200
+
+
+def test_only_the_capped_prefix_is_captured(captured):
+    body = b"y" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=body))
+
+    row = captured[0]
+    assert row.request_body_bytes == 100_000, "the on-the-wire size must be recorded in full"
+    assert len(row.request_body_raw) <= 1024, (
+        "the middleware buffered more than max_body_bytes — memory is unbounded per request"
+    )
+
+
+def test_non_capturable_content_type_is_counted_but_not_buffered(captured):
+    app = RequestResponseLogMiddleware(chunked_app([b"\x00\x01\x02" * 500],
+                                                   content_type=b"application/octet-stream"))
+
+    asyncio.run(drive(app, method="GET", content_type=None))
+
+    row = captured[0]
+    assert row.response_body_bytes == 1500
+    assert row.response_body_raw is None, (
+        "a binary response body was buffered — this is what keeps streaming/file "
+        "responses safe"
+    )
+
+
+# --------------------------------------------------------------------------
+# What gets logged, and what does not
+# --------------------------------------------------------------------------
+
+def test_basic_row_fields(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="POST", path="/api/backends",
+                      body=b'{"name":"web"}', query=b"cluster_id=2&token=secret"))
+
+    row = captured[0]
+    assert row.direction == "inbound"
+    assert row.method == "POST"
+    assert row.path == "/api/backends"
+    assert row.status_code == 200
+    assert row.status_class == 2
+    assert row.client_ip == "10.1.2.3"
+    assert row.duration_ms >= 0
+    # The query string is scrubbed before it is stored, in the URL and the dict.
+    assert "secret" not in row.url
+    assert row.query_params["token"] == "***REDACTED***"
+    assert row.query_params["cluster_id"] == "2"
+
+
+@pytest.mark.parametrize("path", [
+    "/api/health",
+    "/api/health/deep",
+    "/api/docs",
+    "/api/openapi.json",
+    "/.well-known/acme-challenge/abc123",
+    "/api/agents/heartbeat",
+    "/favicon.ico",
+])
+def test_excluded_paths_produce_no_row(captured, path):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path=path))
+    assert captured == [], f"{path} must not be logged by default"
+
+
+def test_log_viewer_path_cannot_be_un_excluded(captured, monkeypatch):
+    """`exclude_paths` is operator-editable, so the viewer's own endpoints have a
+    hard floor — otherwise reading the log generates log entries about reading
+    the log."""
+    set_config(monkeypatch, exclude_paths=())
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path="/api/request-logs?limit=50"))
+
+    assert captured == [], (
+        "clearing exclude_paths re-enabled logging of the log viewer itself"
+    )
+
+
+def test_options_preflight_is_skipped(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="OPTIONS", path="/api/backends"))
+    assert captured == []
+
+
+def test_get_can_be_turned_off(captured, monkeypatch):
+    set_config(monkeypatch, capture_get=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="GET", path="/api/backends"))
+    assert captured == []
+
+    asyncio.run(drive(app, method="POST", path="/api/backends", body=b"{}"))
+    assert len(captured) == 1, "turning GETs off must not silence writes"
+
+
+def test_disabled_config_short_circuits_but_still_serves(captured, monkeypatch):
+    set_config(monkeypatch, enabled=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=b"hello"))
+
+    assert captured == []
+    assert sent["status"] == 200, "the kill-switch must not break request serving"
+
+
+def test_capture_bodies_off_keeps_sizes_but_drops_content(captured, monkeypatch):
+    set_config(monkeypatch, capture_bodies=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=b'{"secret":"x"}'))
+
+    row = captured[0]
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 14, "size accounting must survive with bodies off"
+
+
+# --------------------------------------------------------------------------
+# Errors and correlation
+# --------------------------------------------------------------------------
+
+def test_exception_is_recorded_as_status_class_zero_and_reraised(captured):
+    async def boom(scope, receive, send):
+        raise RuntimeError("handler exploded")
+
+    app = RequestResponseLogMiddleware(boom)
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(drive(app, method="GET"))
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a request that produced no HTTP response must be status_class 0 — that is the "
+        "sentinel the error-retention prune keys off"
+    )
+    assert row.error.startswith("RuntimeError")
+
+
+def test_correlation_id_is_seeded_before_downstream_and_reset_after(captured):
+    seen = {}
+
+    async def app(scope, receive, send):
+        seen["cid"] = correlation_id_context.get()
+        await send({"type": "http.response.start", "status": 204, "headers": []})
+        await send({"type": "http.response.body", "body": b""})
+
+    wrapped = RequestResponseLogMiddleware(app)
+    asyncio.run(drive(wrapped, method="GET"))
+
+    row = captured[0]
+    assert seen["cid"] == row.request_id[:8], (
+        "the downstream error handler would mint its own id, so X-Correlation-ID would "
+        "not match request_logs.request_id"
+    )
+    assert correlation_id_context.get() is None, (
+        "the ContextVar token was not reset — the next request on this task would inherit "
+        "a stale correlation id"
+    )
+
+
+def test_x_request_id_header_is_returned_to_the_client(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    sent = asyncio.run(drive(app, method="GET"))
+
+    names = {k.decode().lower() for k, _ in sent["headers"]}
+    assert "x-request-id" in names, (
+        "without this header a user reporting a problem has no id to quote"
+    )
+
+
+def test_error_responses_are_logged_with_their_status(captured):
+    app = RequestResponseLogMiddleware(echo_length_app(status=422))
+    asyncio.run(drive(app, body=b'{"bad":true}'))
+
+    row = captured[0]
+    assert row.status_code == 422
+    assert row.status_class == 4, "4xx must be classed as an error for retention purposes"
+
+
+def test_jwt_identity_is_resolved_without_a_database(captured):
+    """The middleware runs on every request; a DB lookup per call is not
+    acceptable, so the user is read straight out of the token claims."""
+    from datetime import datetime, timedelta
+
+    from jose import jwt
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+
+    token = jwt.encode(
+        {"user_id": 42, "username": "ops", "exp": datetime.utcnow() + timedelta(minutes=10)},
+        JWT_SECRET_KEY, algorithm=JWT_ALGORITHM,
+    )
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": f"Bearer {token}"}))
+
+    row = captured[0]
+    assert row.user_id == 42
+    assert row.username == "ops"
+
+
+def test_malformed_token_yields_an_anonymous_row(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer not.a.jwt"}))
+
+    row = captured[0]
+    assert row.user_id is None
+    assert row.username is None
+    # Logging is not an auth path — a bad token must not turn into an exception.
+
+
+def test_authorization_header_is_never_stored_verbatim(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer super-secret"}))
+
+    params = captured[0].to_params()
+    assert "super-secret" not in json.dumps(params, default=str)
+
+
+# --------------------------------------------------------------------------
+# Registration order in main.py
+# --------------------------------------------------------------------------
+
+def test_middleware_is_registered_last_so_it_is_outermost():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    log_at = src.index("app.add_middleware(RequestResponseLogMiddleware)")
+    cors_at = src.index("    CORSMiddleware,")
+
+    assert log_at > cors_at, (
+        "Starlette's add_middleware inserts at index 0, so the LAST registration is the "
+        "OUTERMOST middleware. Registering the request logger before CORS would put it "
+        "inside the stack, where it can no longer see the final client-visible response "
+        "and can no longer seed the correlation id before the error handler reads it."
+    )
+
+
+def test_env_kill_switch_guards_the_registration():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert re.search(
+        r"if REQUEST_LOG_ENABLED:\s*\n\s*app\.add_middleware\(RequestResponseLogMiddleware\)",
+        src,
+    ), (
+        "REQUEST_LOG_ENABLED must gate the add_middleware call itself, not a branch inside "
+        "the middleware — the whole point is that a disabled log costs nothing"
+    )
+
+
+def test_cors_exposes_the_request_id_header():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert "expose_headers=" in src and "X-Request-ID" in src, (
+        "without expose_headers the browser cannot read X-Request-ID on a cross-origin "
+        "deployment, so the id is unusable from the app"
+    )

backend/tests/test_request_log_migration.py

@@ -0,0 +1,251 @@
+"""v1.11.0: the request_logs migration actually runs on existing installs.
+
+Source-scan tests (the sanctioned pattern here — there is no database in this
+suite). The failure mode being pinned is specific and silent: migrations are
+gated on `applied_version >= SCHEMA_VERSION`, so forgetting the bump means the
+whole sequence is skipped on every already-deployed database and neither the
+table nor the new permissions ever appear — while a fresh install works fine,
+so it looks correct in development.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_MIGRATIONS = os.path.join(_BACKEND, "database", "migrations.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        return f.read()
+
+
+@pytest.fixture(scope="module")
+def runner_body(src):
+    """The body of _run_all_migrations_inner, where steps are registered."""
+    assert "async def _run_all_migrations_inner" in src
+    return src.split("async def _run_all_migrations_inner", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.fixture(scope="module")
+def rbac_body(src):
+    return src.split("async def update_system_roles_to_enterprise_rbac", 1)[1].split("\nasync def ", 1)[0]
+
+
+def _role_block(rbac_body, role):
+    """Slice one role's permission list out of the enterprise_roles literal."""
+    start = rbac_body.index(f"'{role}': {{")
+    end = rbac_body.index("]", rbac_body.index("'permissions': [", start))
+    return rbac_body[start:end]
+
+
+# --------------------------------------------------------------------------
+# The version gate
+# --------------------------------------------------------------------------
+
+def test_schema_version_bumped_to_at_least_11(src):
+    match = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert match, "SCHEMA_VERSION assignment not found in migrations.py"
+    assert int(match.group(1)) >= 11, (
+        "SCHEMA_VERSION was not bumped for the request_logs table. run_all_migrations() "
+        "returns early when the recorded version is already >= SCHEMA_VERSION, so every "
+        "existing deployment would skip the whole run: no request_logs table, no "
+        "requestlog.* permissions, and the feature would silently do nothing in production "
+        "while working perfectly on a fresh database."
+    )
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_both_migration_steps_are_registered(runner_body):
+    assert "await ensure_request_logs_table()" in runner_body, (
+        "ensure_request_logs_table is defined but never called from the migration runner"
+    )
+    assert "await ensure_request_log_settings()" in runner_body, (
+        "the retention defaults are never seeded, so an upgraded install has no "
+        "requestlog.* rows and Settings shows blanks"
+    )
+
+
+def test_table_is_created_before_its_settings_are_seeded(runner_body):
+    table_at = runner_body.index("await ensure_request_logs_table()")
+    seed_at = runner_body.index("await ensure_request_log_settings()")
+    assert table_at < seed_at, (
+        "the settings seed runs before the table step; if the table step then raises, the "
+        "run aborts with settings but no table"
+    )
+
+
+# --------------------------------------------------------------------------
+# The DDL itself
+# --------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def ddl_body(src):
+    return src.split("async def ensure_request_logs_table", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.mark.parametrize("fragment", [
+    "CREATE TABLE IF NOT EXISTS request_logs",
+    "id                  BIGSERIAL PRIMARY KEY",
+    "request_id          VARCHAR(64) NOT NULL",
+    "direction           VARCHAR(8)  NOT NULL",
+    "status_class        SMALLINT",
+    "created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()",
+    "request_logs_direction_check",
+    "client_ip           INET",
+])
+def test_ddl_essentials(ddl_body, fragment):
+    assert fragment in ddl_body, f"request_logs DDL is missing {fragment!r}"
+
+
+def test_ddl_is_idempotent(ddl_body):
+    assert "CREATE TABLE IF NOT EXISTS" in ddl_body
+    creates = re.findall(r"CREATE INDEX(?: IF NOT EXISTS)?", ddl_body)
+    assert creates, "no indexes are created for request_logs"
+    assert all(c == "CREATE INDEX IF NOT EXISTS" for c in creates), (
+        "an index is created without IF NOT EXISTS — the second startup would raise and "
+        "abort the whole migration run"
+    )
+
+
+def test_prune_partial_indexes_are_present(ddl_body):
+    """The retention delete is split by outcome, so a plain
+    (status_class, created_at) index would still range-scan the half it does not
+    want."""
+    assert "idx_request_logs_prune_ok" in ddl_body
+    assert "idx_request_logs_prune_err" in ddl_body
+    assert "WHERE status_class BETWEEN 1 AND 3" in ddl_body
+    assert "WHERE status_class = 0 OR status_class >= 4" in ddl_body
+
+
+def test_request_id_index_exists_for_the_trace_view(ddl_body):
+    assert "idx_request_logs_request_id" in ddl_body, (
+        "without this index, opening one request to see the outbound calls it triggered "
+        "is a sequential scan"
+    )
+
+
+def test_no_foreign_key_on_user_id(ddl_body):
+    """Deliberate deviation from the house style — see the docstring in
+    migrations.py. Pinned so it is not 'fixed' back into an FK later."""
+    user_id_line = [line for line in ddl_body.splitlines() if "user_id " in line and "INTEGER" in line]
+    assert user_id_line, "user_id column not found"
+    assert "REFERENCES" not in user_id_line[0], (
+        "an FK was added to request_logs.user_id — per-insert FK validation on the "
+        "highest-volume table in the system, and audit rows must outlive the account"
+    )
+
+
+def test_migration_step_reraises_on_failure(ddl_body):
+    """The version marker is written only after the inner sequence completes, so
+    swallowing here would stamp version 11 with no table and the gate would then
+    skip every retry, permanently."""
+    assert re.search(r"\n\s+raise\n", ddl_body), (
+        "ensure_request_logs_table swallows its exception instead of re-raising"
+    )
+
+
+def test_settings_seed_does_not_overwrite_operator_tuning(src):
+    seed_body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    assert "ON CONFLICT (key) DO NOTHING" in seed_body, (
+        "the seed uses DO UPDATE, so every upgrade would reset the operator's retention "
+        "settings back to the defaults"
+    )
+
+
+# --------------------------------------------------------------------------
+# Permission seeding
+# --------------------------------------------------------------------------
+
+def test_super_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "super_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_security_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "security_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_operator_gets_read_only(rbac_body):
+    block = _role_block(rbac_body, "operator")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" not in block, (
+        "operators should be able to read the log to debug an apply or an ACME order, but "
+        "retention policy and purge belong to the admins"
+    )
+
+
+def test_viewer_gets_neither(rbac_body):
+    block = _role_block(rbac_body, "viewer")
+    assert "requestlog" not in block, (
+        "viewer was granted a requestlog permission. Even redacted, captured request and "
+        "response bodies are a far broader disclosure surface than the read-only config "
+        "views a viewer is meant to have."
+    )
+
+
+def test_permission_strings_have_exactly_one_dot(rbac_body):
+    """get_user_permissions splits on the FIRST dot and silently drops any
+    string without one."""
+    for perm in re.findall(r"'(requestlog[^']*)'", rbac_body):
+        assert perm.count(".") == 1, f"{perm!r} is not a <resource>.<action> pair"
+
+
+def test_initial_seed_lists_stay_in_sync(src):
+    """create_initial_system_data() is overwritten by the enterprise seeder on
+    every run, but that seeder swallows all exceptions — keeping the two in sync
+    is the safety net."""
+    initial = src.split("system_roles = [", 1)[1].split("\n        ]", 1)[0]
+    assert '"requestlog.read"' in initial
+    assert '"requestlog.manage"' in initial
+
+
+# --------------------------------------------------------------------------
+# Runtime wiring
+# --------------------------------------------------------------------------
+
+def test_prune_loop_is_started_and_independent_of_the_acme_loop():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "async def prune_request_logs_loop" in main_src
+    assert "asyncio.create_task(prune_request_logs_loop())" in main_src, (
+        "the retention prune task is defined but never started, so request_logs grows "
+        "without bound"
+    )
+    loop_body = main_src.split("async def prune_request_logs_loop", 1)[1].split("\n# Production middleware", 1)[0]
+    assert "table_name = 'request_logs'" in loop_body, (
+        "the prune loop does not check for its own table, so it would log an error every "
+        "tick on a database where the migration has not run yet"
+    )
+    assert "table_name = 'letsencrypt_orders'" not in loop_body, (
+        "the prune loop was gated on the ACME table, which would disable retention "
+        "entirely on an install that never uses ACME"
+    )
+
+
+def test_sink_is_flushed_before_the_pool_closes():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    body = main_src.split("async def shutdown_event", 1)[1]
+    flush_at = body.find("request_log_sink.flush")
+    close_at = body.find("close_database_pool()")
+    assert flush_at != -1, "queued request-log rows are never flushed on shutdown"
+    assert flush_at < close_at, (
+        "the sink is flushed after the pool is closed, so the queued rows are lost. The "
+        "sink's writer is a `while True` loop and can never satisfy the generic "
+        "asyncio.wait drain, so it needs its own explicit flush first."
+    )

backend/tests/test_request_log_prune.py

@@ -0,0 +1,261 @@
+"""v1.11.0: retention actually reclaims space, and cannot be turned into an
+injection point or a 60-second lock.
+
+`request_logs` is the highest-volume table in the system, so the prune has
+three properties that are easy to get wrong and expensive to get wrong:
+
+  * the operator-supplied retention day counts are BIND PARAMETERS, never
+    string-interpolated into the SQL;
+  * deletes are BATCHED, because the pool's command_timeout is 60s and an
+    unbounded DELETE over millions of rows raises and then nothing is ever
+    pruned;
+  * the watermark is stamped only after a COMPLETE pass, so a pass that dies
+    half-way is retried instead of being recorded as done.
+
+The fake connection dispatches on the SQL text rather than on call order — an
+ordered side_effect list silently passes tests for the wrong reason as soon as
+the number of statements changes.
+"""
+import asyncio
+import json
+import os
+import sys
+from datetime import datetime, timedelta
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_prune  # noqa: E402
+from utils.request_log_prune import (  # noqa: E402
+    BATCH_SIZE,
+    MAX_BATCHES,
+    PRUNE_LOCK_KEY,
+    prune_request_logs_if_due,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_SUCCESS_MARKER = "status_class BETWEEN 1 AND 3"
+_ERROR_MARKER = "status_class = 0 OR status_class >= 4"
+_CAP_MARKER = "id <= $1"
+
+
+def _conn(*, lock=True, watermark_age_minutes=None, cutoff_id=None,
+          success_batches=None, error_batches=None, cap_batches=None,
+          fail_on=None):
+    """A fake asyncpg connection that answers by SQL shape."""
+    conn = AsyncMock()
+
+    def fetchval(sql, *args):
+        text = str(sql)
+        if "pg_try_advisory_lock" in text:
+            return lock
+        if "ORDER BY id DESC OFFSET" in text:
+            return cutoff_id
+        return None
+
+    conn.fetchval = AsyncMock(side_effect=fetchval)
+
+    if watermark_age_minutes is None:
+        conn.fetchrow = AsyncMock(return_value=None)
+    else:
+        stamp = (datetime.utcnow() - timedelta(minutes=watermark_age_minutes)).isoformat() + "Z"
+        conn.fetchrow = AsyncMock(return_value={"value": json.dumps(stamp)})
+
+    queues = {
+        _SUCCESS_MARKER: list(success_batches or ["DELETE 0"]),
+        _ERROR_MARKER: list(error_batches or ["DELETE 0"]),
+        _CAP_MARKER: list(cap_batches or ["DELETE 0"]),
+    }
+
+    def execute(sql, *args):
+        text = str(sql)
+        if fail_on and fail_on in text:
+            raise RuntimeError("statement timeout")
+        for marker, queue in queues.items():
+            if marker in text:
+                return queue.pop(0) if queue else "DELETE 0"
+        return ""
+
+    conn.execute = AsyncMock(side_effect=execute)
+    return conn
+
+
+def _run(conn, *, force=False, **cfg_overrides):
+    cfg = replace(DEFAULT_CONFIG, **cfg_overrides)
+    with patch.object(request_log_prune, "get_config", lambda: cfg), \
+         patch.object(request_log_prune, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        return asyncio.run(prune_request_logs_if_due(force=force))
+
+
+def _delete_sql(conn):
+    return [str(c.args[0]) for c in conn.execute.call_args_list
+            if "DELETE FROM request_logs" in str(c.args[0])]
+
+
+def test_skips_entirely_when_another_replica_holds_the_lock():
+    conn = _conn(lock=False)
+    counts = _run(conn)
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+    assert _delete_sql(conn) == [], (
+        "a second replica ran the prune concurrently — pg_try_advisory_lock is what keeps "
+        "N pods from all scanning the same table at once"
+    )
+
+
+def test_skips_when_the_watermark_is_still_fresh():
+    conn = _conn(watermark_age_minutes=10)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["ran"] == 0
+    assert _delete_sql(conn) == []
+
+
+def test_runs_all_three_limits_when_due():
+    conn = _conn(
+        watermark_age_minutes=120, cutoff_id=999,
+        success_batches=["DELETE 3"], error_batches=["DELETE 4"], cap_batches=["DELETE 5"],
+    )
+    counts = _run(conn, prune_interval_minutes=60, success_retention_days=7,
+                  error_retention_days=30, max_rows=500000)
+
+    sqls = _delete_sql(conn)
+    assert len(sqls) == 3, f"expected success TTL + error TTL + row cap, got {len(sqls)}"
+    assert _SUCCESS_MARKER in sqls[0]
+    assert _ERROR_MARKER in sqls[1]
+    assert _CAP_MARKER in sqls[2]
+
+    assert counts["success"] == 3
+    assert counts["error"] == 4
+    assert counts["overflow"] == 5
+    assert counts["ran"] == 1
+
+
+def test_retention_days_travel_as_bind_parameters():
+    """Injection guard: the day counts come straight from an operator-editable
+    setting, so they must never be formatted into the SQL text."""
+    conn = _conn(watermark_age_minutes=120)
+    _run(conn, prune_interval_minutes=60, success_retention_days=7, error_retention_days=30)
+
+    ttl_calls = [c for c in conn.execute.call_args_list
+                 if "created_at < NOW()" in str(c.args[0])]
+    assert len(ttl_calls) == 2
+
+    for call in ttl_calls:
+        assert "($1 || ' days')::INTERVAL" in str(call.args[0]), (
+            "the retention window is interpolated into the SQL string instead of bound — "
+            "an operator-supplied value reaching the parser is an injection point"
+        )
+
+    assert ttl_calls[0].args[1] == "7"
+    assert ttl_calls[1].args[1] == "30"
+    assert ttl_calls[0].args[2] == BATCH_SIZE
+
+
+def test_deletes_are_batched_until_a_short_batch():
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}", f"DELETE {BATCH_SIZE}", "DELETE 12"],
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * 2 + 12, (
+        "the batch loop stopped early or double-counted"
+    )
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == 3, "the loop must stop on the first short batch"
+
+
+def test_batch_loop_respects_the_ceiling():
+    """A table so far behind that every batch comes back full must still hand the
+    connection back rather than looping forever."""
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}"] * (MAX_BATCHES * 3),
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * MAX_BATCHES
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == MAX_BATCHES
+
+
+def test_watermark_is_not_stamped_when_a_step_fails():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=42, fail_on=_CAP_MARKER)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert stamps == [], (
+        "a partially-completed pass stamped the watermark, so the remainder would not be "
+        "retried until the next interval"
+    )
+    assert counts["ran"] == 0
+
+
+def test_watermark_is_stamped_after_a_complete_pass():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert len(stamps) == 1
+    # args = (sql, key, json_value)
+    assert stamps[0].args[1] == "requestlog.last_pruned_at"
+    assert stamps[0].args[2].startswith('"'), (
+        "the watermark must be stored as a JSON string — the ::jsonb cast rejects a bare "
+        "timestamp, and the reader json.loads() it back"
+    )
+    assert counts["ran"] == 1
+
+
+def test_advisory_lock_is_released_even_on_failure():
+    conn = _conn(watermark_age_minutes=120, fail_on=_SUCCESS_MARKER)
+    _run(conn, prune_interval_minutes=60)
+
+    unlocks = [c for c in conn.execute.call_args_list if "pg_advisory_unlock" in str(c.args[0])]
+    assert unlocks, "the advisory lock was leaked — every later pass on any replica would skip"
+    assert unlocks[0].args[1] == PRUNE_LOCK_KEY
+
+
+def test_never_raises_when_the_pool_is_exhausted():
+    with patch.object(request_log_prune, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        counts = asyncio.run(prune_request_logs_if_due())
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+
+
+def test_row_cap_is_a_noop_when_the_table_is_smaller_than_the_cap():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None, cap_batches=["DELETE 77"])
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["overflow"] == 0, (
+        "the cap deleted rows even though OFFSET max_rows found no cutoff — that would "
+        "truncate a table that is under the limit"
+    )
+    assert not any(_CAP_MARKER in s for s in _delete_sql(conn))
+
+
+def test_force_bypasses_the_watermark():
+    """The manual purge button must not be a no-op just because the scheduled
+    pass ran a minute ago."""
+    conn = _conn(watermark_age_minutes=1, cutoff_id=None,
+                 success_batches=["DELETE 1"], error_batches=["DELETE 2"])
+    counts = _run(conn, force=True, prune_interval_minutes=1440)
+
+    assert counts["ran"] == 1
+    assert counts["success"] == 1
+    assert counts["error"] == 2
+
+
+def test_lock_key_does_not_collide_with_the_existing_ones():
+    # 18181818 drafts cap, 18181819 wizard create, 18181820 apply,
+    # 0x41434D45 per-ACME-order, 1836016242 migration.
+    assert PRUNE_LOCK_KEY not in (18181818, 18181819, 18181820, 0x41434D45, 1836016242)

backend/tests/test_request_log_real_payloads.py

@@ -0,0 +1,289 @@
+"""v1.11.0: redaction pinned against THIS codebase's real payloads.
+
+test_request_log_redaction.py pins the RULES — which key names match, which
+value shapes fire. It passed 72/72 while six real endpoints of this application
+still wrote secrets to `request_logs`, because a rule test proves the rule, not
+the coverage. Every case here is built from an actual handler's request or
+response shape, with the field names taken from the source and named in the
+docstring, so a future change to redaction is measured against what this system
+actually sends rather than against what someone remembered to imagine.
+
+Method note: the payloads go through `decode_body()`, the same entry point the
+writer task uses, rather than calling `redact()` directly. That is deliberate —
+two of the findings below only appear on the way in (an oversized body never
+reaches `redact()` as a dict at all, it arrives as one `_raw` string), so a test
+that starts from a dict would report a pass on a payload that leaks in
+production.
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    decode_body,
+    is_secret_key,
+    redact_headers,
+)
+
+VRRP_SECRET = "S3cr3tVrrpPass!"
+TOTP_SECRET = "JBSWY3DPEHPK3PXP"
+STATS_PASSWORD = "StatsPa55word"
+USERLIST_HASH = "$6$rounds=5000$abcdefgh$XyZ"
+JWT = (
+    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
+    ".eyJzdWIiOiIxIiwidXNlcm5hbWUiOiJhZG1pbiJ9"
+    ".dQw4w9WgXcQdQw4w9WgXcQdQw4w9WgXcQ"
+)
+PEM_KEY = (
+    "-----BEGIN RSA PRIVATE KEY-----\n"
+    + "MIIEowIBAAKCAQEA" + "A" * 200 + "\n"
+    + "-----END RSA PRIVATE KEY-----\n"
+)
+
+# The rendered file `GET /api/agents/{n}/keepalived-config` hands to an agent,
+# and the one `POST /api/agents/{n}/keepalived-discovery` sends back.
+KEEPALIVED_CONF = f"""! Managed by HAProxy OpenManager
+vrrp_instance VI_1 {{
+    state MASTER
+    interface eth0
+    virtual_router_id 51
+    priority 200
+    advert_int 1
+    authentication {{
+        auth_type PASS
+        auth_pass {VRRP_SECRET}
+    }}
+    virtual_ipaddress {{
+        10.20.30.40/24
+    }}
+}}
+"""
+
+# A production haproxy.cfg as the agent uploads it verbatim from the node
+# (`config_content=$(cat "$config_path")` -> POST .../config-response).
+HAPROXY_CFG = f"""global
+    log stdout local0
+    stats socket /var/run/haproxy.sock mode 660
+
+userlist admins
+    user ops password {USERLIST_HASH}
+    user dev insecure-password Hunter2Plain
+
+listen stats
+    bind *:8404
+    stats enable
+    stats auth admin:{STATS_PASSWORD}
+    stats uri /stats
+
+backend web
+    server web1 10.0.0.1:80 check
+"""
+
+
+def _capture(payload, *, cap=8192, content_type="application/json"):
+    """Run a payload through the capture path exactly as the writer does.
+
+    `cap` is `requestlog.max_body_bytes`. Bodies larger than it arrive
+    truncated, do not parse as JSON, and land in the `{"_raw": ...}` fallback —
+    which is the common case for config uploads and the case a dict-based test
+    never exercises.
+    """
+    body = json.dumps(payload).encode()
+    value, truncated = decode_body(body[:cap], content_type, len(body))
+    return json.dumps(value, default=str), truncated
+
+
+def _assert_absent(rendered, *secrets):
+    for secret in secrets:
+        assert secret not in rendered, (
+            f"{secret!r} reached request_logs. Rendered row: {rendered[:400]}"
+        )
+
+
+# --------------------------------------------------------------------------
+# The VRRP password. routers/vip.py: "the secret never leaves the server in
+# cleartext ... only the at-rest Fernet token and the agent-delivery endpoint
+# ever see the real value."
+# --------------------------------------------------------------------------
+
+def test_vip_create_body_does_not_store_auth_pass():
+    """POST/PUT /api/vip — `payload.auth_pass`, routers/vip.py:585,679."""
+    rendered, _ = _capture({
+        "name": "vip-prod", "virtual_ip": "10.20.30.40", "interface": "eth0",
+        "virtual_router_id": 51, "auth_pass": VRRP_SECRET,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_keepalived_config_delivery_does_not_store_the_rendered_secret():
+    """GET /api/agents/{n}/keepalived-config — `keepalived.config_content`.
+
+    Polled on the SSL cadence, so an unmasked capture rewrites the secret to the
+    audit table roughly 576 times a day per member node.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "status": "available",
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "keepalived": {
+            "vip_id": 3, "vip_name": "vip-prod",
+            "config_content": KEEPALIVED_CONF, "config_hash": "abc123",
+        },
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "auth_pass" in rendered, "the directive should stay visible, only its value masked"
+    assert "vrrp_instance VI_1" in rendered, "masking must not destroy the rest of the config"
+
+
+def test_keepalived_discovery_body_does_not_store_the_found_secret():
+    """POST /api/agents/{n}/keepalived-discovery — `config_content`.
+
+    routers/agent.py already pops auth_pass out of the parsed analysis,
+    Fernet-encrypts it into its own column and stores only
+    `vip_discoveries.raw_config_masked`. Capturing the request that produced all
+    that, unmasked, would put the plaintext straight back next to it.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "exists": True, "is_managed": False,
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "config_content": KEEPALIVED_CONF,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_auth_pass_is_masked_when_the_body_is_too_large_to_parse():
+    """The truncated `_raw` path, where line breaks are the escape `\\n`.
+
+    A value pattern that stops only at a REAL newline runs to the end of the
+    string here: no leak, but the whole remainder of the config is masked and
+    the row is useless. Both properties are asserted.
+    """
+    payload = {"config_content": KEEPALIVED_CONF + "backend b\n    server s1 10.0.0.1:80 check\n" * 400}
+    rendered, truncated = _capture(payload)
+    assert truncated, "this fixture must exercise the truncated path"
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "server s1 10.0.0.1:80" in rendered, (
+        "masking ran past the end of the auth_pass line and ate the rest of the config"
+    )
+
+
+# --------------------------------------------------------------------------
+# TOTP. routers/mfa.py logs `{"secret_len": ...}` with the comment
+# "NEVER log the secret itself".
+# --------------------------------------------------------------------------
+
+def test_mfa_enroll_response_does_not_store_the_totp_secret_in_either_field():
+    """POST /api/mfa/enroll — returns `secret` AND `otpauth_uri`.
+
+    Redacting one while the same value sits in the other is not redaction.
+    """
+    rendered, _ = _capture({
+        "secret": TOTP_SECRET,
+        "otpauth_uri": f"otpauth://totp/OpenManager:admin?secret={TOTP_SECRET}&issuer=OpenManager",
+        "qr_size": 256,
+    })
+    _assert_absent(rendered, TOTP_SECRET)
+
+
+def test_userinfo_credentials_in_a_url_are_dropped():
+    rendered, _ = _capture({"webhook": "https://svc:Sup3rSecret@hooks.example.com/notify?api_key=abc123"})
+    _assert_absent(rendered, "Sup3rSecret", "abc123")
+    assert "hooks.example.com" in rendered, "the host is the diagnostic value; keep it"
+
+
+# --------------------------------------------------------------------------
+# HAProxy config. We never RENDER credentials into one, but the agent uploads
+# the node's real file and the operator can paste one.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("cap,label", [(8192, "truncated _raw path"), (10 ** 6, "parsed path")])
+def test_uploaded_haproxy_config_masks_credentials_on_both_paths(cap, label):
+    """POST /api/configuration/agents/{n}/config-response, and
+    POST /api/config/validate."""
+    rendered, _ = _capture({"config_content": HAPROXY_CFG, "config_path": "/etc/haproxy/haproxy.cfg"}, cap=cap)
+    _assert_absent(rendered, STATS_PASSWORD, USERLIST_HASH, "Hunter2Plain")
+    assert "stats auth admin:" in rendered, f"[{label}] the account name is diagnostic; keep it"
+    assert "server web1 10.0.0.1:80" in rendered, f"[{label}] the rest of the config must survive"
+
+
+@pytest.mark.parametrize("prose", [
+    "invalid password format",
+    "the password must be at least 8 characters",
+    "authentication failed for user admin",
+])
+def test_ordinary_prose_is_not_mangled(prose):
+    """Over-matching would blank the messages the log exists to show."""
+    rendered, _ = _capture({"detail": prose})
+    assert prose in rendered, f"redaction damaged an ordinary message: {rendered}"
+
+
+# --------------------------------------------------------------------------
+# Regressions guarding what already worked, so a later rule change cannot
+# quietly trade one of these away for one of the above.
+# --------------------------------------------------------------------------
+
+def test_login_exchange_stores_neither_the_password_nor_the_token():
+    req, _ = _capture({"username": "admin", "password": "hunter2hunter2"})
+    _assert_absent(req, "hunter2hunter2")
+    res, _ = _capture({"access_token": JWT, "token_type": "bearer", "user": {"id": 1}})
+    _assert_absent(res, JWT)
+
+
+def test_private_key_is_redacted_even_under_an_innocent_key_name():
+    """The value-shape guard is the net under the key-name rules."""
+    rendered, _ = _capture({"blob": PEM_KEY, "note": "backup"})
+    _assert_absent(rendered, "MIIEowIBAAKCAQEA")
+
+
+def test_dns_provider_credentials_are_redacted():
+    cf, _ = _capture({"provider": "cloudflare", "api_token": "cf_live_abcdefghijklmnop", "zone_id": "z1"})
+    _assert_absent(cf, "cf_live_abcdefghijklmnop")
+    gd, _ = _capture({"provider": "godaddy", "api_key": "gd_key_1234567890", "api_secret": "gd_secret_098"})
+    _assert_absent(gd, "gd_key_1234567890", "gd_secret_098")
+
+
+def test_innocent_urls_survive_untouched():
+    """Scrubbing must not rewrite the ACME URLs an operator reads back."""
+    for url in (
+        "https://acme-v02.api.letsencrypt.org/directory",
+        "https://acme-v02.api.letsencrypt.org/acme/acct/12345",
+    ):
+        rendered, _ = _capture({"directory_url": url})
+        assert url in rendered, f"an innocent URL was rewritten: {rendered}"
+
+
+def test_credential_headers_are_presence_only_and_the_rest_are_dropped():
+    out = redact_headers({
+        "authorization": f"Bearer {JWT}",
+        "x-api-key": "agt_" + "a" * 32,
+        "cookie": "session=abc123",
+        "user-agent": "curl/8.4.0",
+        "x-forwarded-for": "10.20.30.5",
+        "x-internal-secret": "not-on-the-allowlist",
+    })
+    rendered = json.dumps(out)
+    _assert_absent(rendered, JWT, "agt_" + "a" * 32, "abc123", "not-on-the-allowlist")
+    assert out["user-agent"] == "curl/8.4.0"
+    assert out["x-forwarded-for"] == "10.20.30.5"
+    assert "x-internal-secret" not in out, "an unlisted header must be dropped, not kept"
+
+
+@pytest.mark.parametrize("key", ["auth_pass", "authPass", "auth-pass", "AUTH_PASS"])
+def test_auth_pass_key_matches_in_every_spelling(key):
+    assert is_secret_key(key), (
+        f"{key!r} normalizes to something no rule matches. 'password' is not a "
+        f"substring of 'authpass' and the bare 'auth' entry is an exact match."
+    )
+
+
+@pytest.mark.parametrize("key", [
+    "monkey", "key_suffix", "payload_size", "nonce_count", "keyboard_layout",
+    "config_path", "authenticated", "author",
+])
+def test_innocent_field_names_are_still_kept(key):
+    """The other half of the trade: over-redaction blanks the fields the
+    feature exists to show."""
+    assert not is_secret_key(key)

backend/tests/test_request_log_redaction.py

@@ -0,0 +1,262 @@
+"""v1.11.0: nothing secret reaches request_logs.
+
+The request/response log stores bodies and headers, so redaction is the single
+control standing between "operators can debug a failing ACME order" and "the
+audit table is a credential store". These tests pin both halves of that: the
+things that MUST be redacted, and the innocent field names that must NOT be
+(over-matching would silently blank out the fields the feature exists to show).
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    REDACTED,
+    decode_body,
+    is_capturable_content_type,
+    is_secret_key,
+    redact,
+    redact_headers,
+    safe_error_text,
+    scrub_query_string,
+    scrub_url,
+)
+
+
+# --------------------------------------------------------------------------
+# Key matching
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("key", [
+    "password", "PASSWORD", "Pass_Word", "passwd", "pwd",
+    # api_token is the literal field name of the Cloudflare provider credential
+    # (services/dns_providers/cloudflare.py) — it must never survive a round trip.
+    "token", "access_token", "refreshToken", "MFA_TOKEN",
+    "api_token", "agent_token", "csrf_token", "session_token",
+    "api_key", "API-KEY", "apiKey", "x-api-key",
+    "secret", "client_secret", "eab_hmac_key",
+    "private_key", "cert_private_key", "csr_private_key", "jwk_private_key",
+    "authorization", "cookie", "set-cookie",
+    "signature", "protected", "payload", "nonce", "replay-nonce",
+    "key_authorization", "backup_codes", "totp_secret",
+    "stats_password", "credentials", "encryption_key",
+])
+def test_secret_keys_are_detected(key):
+    assert is_secret_key(key), f"{key!r} must be treated as a secret field name"
+
+
+@pytest.mark.parametrize("key", [
+    # Every one of these has a secret-looking substring but is innocent. If any
+    # starts redacting, the log stops being useful for the exact debugging it
+    # was built for.
+    "key_suffix", "monkey", "keyboard", "turkey",
+    "payload_size", "nonce_count",
+    "public_key_id", "keys_total",
+    "name", "status_code", "duration_ms", "domain", "directory_url",
+])
+def test_innocent_keys_are_not_redacted(key):
+    assert not is_secret_key(key), (
+        f"{key!r} was redacted by over-matching — the log would blank out a field "
+        f"operators need"
+    )
+
+
+# --------------------------------------------------------------------------
+# Recursive body redaction
+# --------------------------------------------------------------------------
+
+def test_nested_dicts_and_lists_are_redacted_recursively():
+    body = {
+        "user": {"username": "admin", "password": "hunter2"},
+        "accounts": [
+            {"email": "a@example.com", "eab_hmac_key": "s3cr3t"},
+            {"email": "b@example.com", "api_token": "cf-token"},
+        ],
+        "cluster_id": 3,
+    }
+    out = redact(body)
+
+    assert out["user"]["username"] == "admin"
+    assert out["user"]["password"] == REDACTED
+    assert out["accounts"][0]["email"] == "a@example.com"
+    assert out["accounts"][0]["eab_hmac_key"] == REDACTED
+    assert out["accounts"][1]["api_token"] == REDACTED
+    assert out["cluster_id"] == 3
+
+
+def test_depth_limit_stops_runaway_nesting():
+    deep = current = {}
+    for _ in range(20):
+        current["child"] = {}
+        current = current["child"]
+    current["password"] = "leak"
+
+    out = redact(deep)
+    flattened = json.dumps(out)
+    assert "***DEPTH_LIMIT***" in flattened
+    assert "leak" not in flattened
+
+
+def test_node_budget_bounds_a_very_wide_body():
+    wide = {f"field_{i}": i for i in range(5000)}
+    out = redact(wide)
+    assert out.get("_node_limit") is True
+    assert len(out) < 5000, "node budget did not bound a pathologically wide body"
+
+
+def test_pem_private_key_is_redacted_by_value_shape():
+    body = {"blob": "-----BEGIN RSA PRIVATE KEY-----\n" + "A" * 200 + "\n-----END RSA PRIVATE KEY-----"}
+    out = redact(body)
+    assert out["blob"] == REDACTED, (
+        "a PEM private key under an innocent key name was stored verbatim"
+    )
+
+
+def test_jwt_shaped_string_is_redacted_by_value_shape():
+    jwt_like = "eyJhbGciOiJIUzI1NiJ9." + "a" * 40 + "." + "b" * 40
+    out = redact({"data": jwt_like})
+    assert out["data"] == REDACTED
+
+
+def test_long_strings_are_truncated_with_a_marker():
+    out = redact({"note": "x" * 9000})
+    assert out["note"].endswith("chars]")
+    assert len(out["note"]) < 9000
+
+
+def test_redact_never_raises_on_odd_input():
+    class Weird:
+        def __repr__(self):
+            raise RuntimeError("boom")
+
+    # Non-serializable leaf values must pass straight through, not explode.
+    assert redact({"x": Weird()}) is not None
+
+
+# --------------------------------------------------------------------------
+# Headers (allowlist)
+# --------------------------------------------------------------------------
+
+def test_headers_use_an_allowlist_with_presence_markers():
+    out = redact_headers({
+        "Content-Type": "application/json",
+        "User-Agent": "curl/8.0",
+        "Authorization": "Bearer super-secret-token",
+        "Cookie": "session=abc",
+        "X-Custom-Internal": "some value",
+    })
+
+    assert out["content-type"] == "application/json"
+    assert out["user-agent"] == "curl/8.0"
+    # Presence is useful when debugging a 401; the value is not.
+    assert out["authorization"] == REDACTED
+    assert out["cookie"] == REDACTED
+    # Not on the allowlist and not a known credential header -> dropped entirely.
+    assert "x-custom-internal" not in out
+
+
+def test_redact_headers_handles_none():
+    assert redact_headers(None) is None
+    assert redact_headers({}) is None
+
+
+# --------------------------------------------------------------------------
+# URLs and query strings
+# --------------------------------------------------------------------------
+
+def test_query_string_secrets_are_scrubbed():
+    scrubbed, as_dict = scrub_query_string("token=abc123&page=2&api_key=xyz")
+    assert "abc123" not in scrubbed
+    assert "xyz" not in scrubbed
+    assert "page=2" in scrubbed
+    assert as_dict["token"] == REDACTED
+    assert as_dict["page"] == "2"
+
+
+def test_scrub_url_strips_userinfo_and_query_secrets():
+    out = scrub_url("https://user:hunter2@api.example.com:8443/v1/zones?api_key=abc&page=1")
+    assert "hunter2" not in out
+    assert "user" not in out.split("/v1")[0].replace("api.example.com", "")
+    assert "abc" not in out
+    assert "api.example.com:8443" in out
+    assert "page=1" in out
+
+
+def test_scrub_url_drops_the_fragment():
+    # Fragments never reach a server, and they are a classic token carrier.
+    assert "#" not in scrub_url("https://example.com/x?a=1#access_token=leak")
+
+
+# --------------------------------------------------------------------------
+# Body decoding, capping, truncation marker
+# --------------------------------------------------------------------------
+
+def test_decode_body_parses_and_redacts_json():
+    raw = json.dumps({"username": "admin", "password": "hunter2"}).encode()
+    value, truncated = decode_body(raw, "application/json", len(raw))
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+    assert truncated is False
+
+
+def test_decode_body_marks_truncation_with_the_original_size():
+    full = b"x" * 20000
+    captured = full[:1024]
+    value, truncated = decode_body(captured, "text/plain", len(full))
+    assert truncated is True
+    assert value["_truncated"] is True
+    assert value["_original_bytes"] == 20000
+
+
+def test_decode_body_wraps_non_json_as_raw_object():
+    value, _ = decode_body(b"plain text response", "text/plain", 19)
+    assert value == {"_raw": "plain text response"}
+
+
+def test_decode_body_survives_truncated_json():
+    # A JSON body cut off at the cap will not parse — keep the prefix rather
+    # than losing the field entirely.
+    value, truncated = decode_body(b'{"a": "bb', "application/json", 500)
+    assert truncated is True
+    assert "_raw" in value
+
+
+def test_decode_body_parses_form_encoded():
+    value, _ = decode_body(b"username=admin&password=hunter2",
+                           "application/x-www-form-urlencoded", 30)
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+
+
+def test_decode_body_returns_none_for_empty():
+    assert decode_body(b"", "application/json", 0) == (None, False)
+    assert decode_body(None, "application/json", 0) == (None, False)
+
+
+def test_binary_content_types_are_not_capturable():
+    assert is_capturable_content_type("application/json")
+    assert is_capturable_content_type("application/json; charset=utf-8")
+    assert is_capturable_content_type("text/plain")
+    assert not is_capturable_content_type("application/octet-stream")
+    assert not is_capturable_content_type("image/png")
+    assert not is_capturable_content_type("text/event-stream")
+
+
+# --------------------------------------------------------------------------
+# Error rendering
+# --------------------------------------------------------------------------
+
+def test_safe_error_text_type_only_hides_the_message():
+    exc = ValueError("https://api.godaddy.com/v1/domains/secret-zone/records failed")
+    assert safe_error_text(exc, type_only=True) == "ValueError"
+    assert "godaddy" not in safe_error_text(exc, type_only=True)
+
+
+def test_safe_error_text_includes_the_message_when_allowed():
+    text = safe_error_text(RuntimeError("connection refused"))
+    assert text.startswith("RuntimeError")
+    assert "connection refused" in text

backend/tests/test_request_log_router_auth.py

@@ -0,0 +1,205 @@
+"""v1.11.0: the request log API is gated, and its routes resolve.
+
+Two distinct failure modes are pinned here.
+
+**Auth.** The table holds redacted-but-real request and response bodies for
+every user, so an unauthenticated or under-privileged caller must never get a
+row. There is no database in this suite, so the behavioural checks assert only
+that an anonymous call is rejected before any DB work — which is exactly the
+property that matters — and a source scan covers the per-endpoint permission.
+
+**Route order.** `/{log_id}` is a single-segment path and FastAPI matches in
+declaration order, so declaring it before `/settings`, `/stats` or `/purge`
+makes those three unreachable (they parse as a log id and 422). This is the
+mirror image of the shadowing trap already present in routers/settings.py.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+REJECT = (401, 403, 422)
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_ROUTER, encoding="utf-8") as f:
+        return f.read()
+
+
+# --------------------------------------------------------------------------
+# Behavioural: nothing is readable without credentials
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("method,path", [
+    ("get", "/api/request-logs"),
+    ("get", "/api/request-logs/1"),
+    ("get", "/api/request-logs/stats"),
+    ("get", "/api/request-logs/settings"),
+    ("put", "/api/request-logs/settings"),
+    ("post", "/api/request-logs/purge"),
+])
+def test_anonymous_access_is_rejected(client, method, path):
+    res = getattr(client, method)(path) if method != "put" else client.put(path, json={})
+    assert res.status_code in REJECT, (
+        f"{method.upper()} {path} returned {res.status_code} without an Authorization "
+        f"header — the request log contains captured bodies for every user"
+    )
+
+
+def test_a_garbage_token_is_rejected(client):
+    res = client.get("/api/request-logs", headers={"authorization": "Bearer not-a-token"})
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# Source scan: per-endpoint permission
+# --------------------------------------------------------------------------
+
+def _handler_body(src, decorator):
+    start = src.index(decorator)
+    rest = src[start + len(decorator):]
+    end = rest.find("\n@router.")
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.mark.parametrize("decorator,action", [
+    ('@router.get("/settings")', "manage"),
+    ('@router.put("/settings")', "manage"),
+    ('@router.get("/stats")', "read"),
+    ('@router.post("/purge")', "manage"),
+    ('@router.get("")', "read"),
+    ('@router.get("/{log_id}")', "read"),
+])
+def test_every_endpoint_enforces_its_permission(src, decorator, action):
+    body = _handler_body(src, decorator)
+    assert f'_require(authorization, "{action}")' in body, (
+        f"{decorator} does not enforce requestlog.{action}"
+    )
+
+
+def test_require_helper_raises_403_not_a_silent_pass(src):
+    helper = src.split("async def _require", 1)[1].split("\nasync def ", 1)[0]
+    assert "check_user_permission" in helper
+    assert "status_code=403" in helper
+    assert "current_user=current_user" in helper, (
+        "the admin bypass is skipped, so every call pays an extra SELECT on users"
+    )
+
+
+# --------------------------------------------------------------------------
+# Route declaration order
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("literal", ['@router.get("/settings")', '@router.put("/settings")',
+                                     '@router.get("/stats")', '@router.post("/purge")'])
+def test_literal_routes_are_declared_before_the_catch_all(src, literal):
+    catch_all = src.index('@router.get("/{log_id}")')
+    assert src.index(literal) < catch_all, (
+        f"{literal} is declared after GET /{{log_id}}. FastAPI matches in declaration "
+        f"order and /{{log_id}} is a single-segment path, so it would swallow this route "
+        f"and the request would fail parsing 'settings' as an int."
+    )
+
+
+def test_list_route_is_declared_before_the_catch_all(src):
+    assert src.index('@router.get("")') < src.index('@router.get("/{log_id}")')
+
+
+# --------------------------------------------------------------------------
+# Query construction
+# --------------------------------------------------------------------------
+
+def test_filters_are_bound_never_interpolated(src):
+    """User-supplied filters reach the WHERE clause; they must arrive as $n
+    parameters."""
+    body = _handler_body(src, '@router.get("")')
+    # The only f-string interpolation allowed into SQL is the placeholder index
+    # and the assembled clause list, never a raw value.
+    for match in re.findall(r'add\("([^"]+)"', body):
+        assert "{n}" in match, f"filter clause {match!r} does not use a bound placeholder"
+
+
+def test_list_endpoint_scopes_non_privileged_callers(src):
+    """A caller with only `requestlog.read` sees their own rows plus the fleet's.
+
+    Widened from own-rows-only during review, deliberately. The `operator` role
+    is granted requestlog.read to "debug failing applies and ACME orders", but
+    an apply fails on the NODE and the node reports it over its own API key, so
+    the row carrying the diagnosis has `user_id IS NULL` — own-rows-only hid it
+    from exactly the role the grant was written for.
+
+    What must NOT widen is the part this test was written to protect: another
+    USER's captured bodies. Both halves are asserted below.
+    """
+    # Comments explain what the clause deliberately does NOT do, so match on
+    # code only — otherwise the prose describing the rule fails the test for it.
+    body = "\n".join(
+        line for line in _handler_body(src, '@router.get("")').splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "if not can_manage:" in body
+    assert "direction = 'inbound'" in body, (
+        "outbound rows are not scoped at all, so a caller with only "
+        "requestlog.read would see every CA and DNS call the backend ever made"
+    )
+    assert "user_id = $" in body, (
+        "a caller with only requestlog.read can see every other user's captured "
+        "request bodies"
+    )
+    assert "TARGET_INBOUND_AGENT" in body, (
+        "agent rows are hidden from requestlog.read, which is the one thing the "
+        "operator grant exists for"
+    )
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL rather than on target would also expose anonymous "
+        "traffic — failed logins and the usernames they carry, unauthenticated "
+        "probes — to any requestlog.read holder"
+    )
+
+
+def test_detail_endpoint_applies_the_same_scoping(src):
+    body = _handler_body(src, '@router.get("/{log_id}")')
+    assert "can_manage" in body
+    assert "404" in body, (
+        "the detail endpoint should 404 rather than 403 for a row the caller may not see, "
+        "so it does not confirm which ids exist"
+    )
+
+
+def test_list_response_omits_bodies(src):
+    """A 200-row page carrying two 8 KB JSONB blobs per row is a multi-megabyte
+    response; bodies belong to the detail endpoint."""
+    columns = src.split("_LIST_COLUMNS = ", 1)[1].split('"""', 2)[1]
+    assert "request_body," not in columns
+    assert "response_body," not in columns
+    assert "request_body_bytes" in columns, "the size is still useful in the list"
+
+
+def test_count_is_bounded(src):
+    body = _handler_body(src, '@router.get("")')
+    assert "LIMIT {count_cap}" in body or "count_cap" in body, (
+        "an unbounded COUNT(*) over request_logs is a sequential scan on every page change"
+    )
+    assert "total_is_estimate" in body
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_router_is_registered_in_main():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "from routers.request_logs import router as request_logs_router" in main_src
+    assert "app.include_router(request_logs_router)" in main_src, (
+        "the router is imported but never mounted, so every endpoint 404s"
+    )

backend/tests/test_request_log_settings.py

@@ -0,0 +1,258 @@
+"""v1.11.0: the retention policy the operator sees is the policy that runs.
+
+Two things drift silently and are caught here:
+
+  1. The defaults live in TWO places — the seed SQL in migrations.py and the
+     dataclass in utils/request_log_settings.py. If they disagree, a fresh
+     install and an upgraded install behave differently, which is the worst
+     kind of bug to chase.
+  2. Values in `system_settings` are operator-editable and arrive from asyncpg
+     as raw JSON *strings*. Anything out of range, mistyped or hand-edited must
+     be clamped rather than crash the writer loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    RequestLogConfig,
+    config_from_mapping,
+    get_config,
+    normalize_exclude_paths,
+    refresh_config,
+    set_config,
+)
+
+_MIGRATIONS = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "database", "migrations.py"
+)
+
+
+def _seeded_defaults():
+    """Parse the ('requestlog.x', 'value', ...) tuples out of the seed SQL."""
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        src = f.read()
+
+    body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    out = {}
+    for key, raw in re.findall(r"\('requestlog\.(\w+)', '(.*?)', 'requestlog'", body):
+        try:
+            out[key] = json.loads(raw)
+        except json.JSONDecodeError:
+            out[key] = raw
+    return out
+
+
+# --------------------------------------------------------------------------
+# Defaults must not drift between the seed and the code
+# --------------------------------------------------------------------------
+
+def test_seed_and_dataclass_defaults_agree():
+    seeded = _seeded_defaults()
+    assert seeded, "could not parse the requestlog seed rows out of migrations.py"
+
+    code = DEFAULT_CONFIG.as_dict()
+    for key, seed_value in seeded.items():
+        assert key in code, f"migrations seeds requestlog.{key} but RequestLogConfig has no such field"
+        assert code[key] == seed_value, (
+            f"requestlog.{key} default drifted: migrations.py seeds {seed_value!r} but "
+            f"RequestLogConfig has {code[key]!r}. A fresh install and an upgraded install "
+            f"would then behave differently."
+        )
+
+    for key in code:
+        assert key in seeded, (
+            f"RequestLogConfig has {key!r} but migrations.py does not seed requestlog.{key} — "
+            f"existing installs would silently fall back to the in-code default"
+        )
+
+
+def test_log_viewer_is_excluded_by_default():
+    assert "/api/request-logs" in DEFAULT_EXCLUDE_PATHS
+    assert "/api/health" in DEFAULT_EXCLUDE_PATHS
+    assert "/.well-known/acme-challenge" in DEFAULT_EXCLUDE_PATHS, (
+        "the ACME challenge endpoint returns key_authorization — logging it would store "
+        "the challenge secret"
+    )
+    assert "/api/agents/heartbeat" in DEFAULT_EXCLUDE_PATHS, (
+        "the agent heartbeat is the highest-volume POST in the system; logging it by "
+        "default would dominate the table"
+    )
+
+
+def test_error_retention_defaults_longer_than_success_retention():
+    assert DEFAULT_CONFIG.error_retention_days > DEFAULT_CONFIG.success_retention_days, (
+        "the whole point of splitting the two is to keep failures around after the "
+        "ordinary traffic has aged out"
+    )
+
+
+# --------------------------------------------------------------------------
+# Coercion and clamping
+# --------------------------------------------------------------------------
+
+def test_raw_json_strings_from_asyncpg_are_parsed():
+    cfg = config_from_mapping({
+        "enabled": True,
+        "max_body_bytes": 4096,
+        "sample_rate": 0.25,
+        "success_retention_days": 3,
+        "exclude_paths": ["/api/health", "/metrics"],
+    })
+    assert cfg.enabled is True
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.25
+    assert cfg.success_retention_days == 3
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+
+@pytest.mark.parametrize("raw,expected", [
+    ("true", True), ("false", False), ("1", True), ("0", False),
+    ("on", True), ("off", False), (1, True), (0, False), (True, True),
+])
+def test_boolean_coercion_accepts_hand_written_values(raw, expected):
+    cfg = config_from_mapping({"enabled": raw})
+    assert cfg.enabled is expected
+
+
+@pytest.mark.parametrize("field,value,expected", [
+    ("max_body_bytes", 10_000_000, 262144),
+    ("max_body_bytes", -5, 0),
+    ("success_retention_days", 0, 1),
+    ("success_retention_days", 9999, 365),
+    ("error_retention_days", 0, 1),
+    ("max_rows", 10, 1000),
+    ("prune_interval_minutes", 1, 5),
+    ("prune_interval_minutes", 99999, 1440),
+])
+def test_out_of_range_values_are_clamped_not_rejected(field, value, expected):
+    """A bad value in the table must not disable logging or crash the writer —
+    it is clamped to the nearest sane bound."""
+    cfg = config_from_mapping({field: value})
+    assert getattr(cfg, field) == expected
+
+
+@pytest.mark.parametrize("value,expected", [(1.5, 1.0), (-0.2, 0.0), ("0.4", 0.4)])
+def test_sample_rate_is_clamped(value, expected):
+    assert config_from_mapping({"sample_rate": value}).sample_rate == expected
+
+
+def test_garbage_values_fall_back_to_the_default():
+    cfg = config_from_mapping({"max_body_bytes": "not-a-number", "sample_rate": "abc"})
+    assert cfg.max_body_bytes == DEFAULT_CONFIG.max_body_bytes
+    assert cfg.sample_rate == DEFAULT_CONFIG.sample_rate
+
+
+def test_exclude_paths_shape_is_enforced():
+    out = normalize_exclude_paths(
+        ["/good", "no-leading-slash", "/" + "x" * 500, 42, "/also-good"],
+        DEFAULT_EXCLUDE_PATHS,
+    )
+    assert out == ("/good", "/also-good")
+
+
+def test_exclude_paths_count_is_bounded():
+    out = normalize_exclude_paths([f"/p{i}" for i in range(500)], DEFAULT_EXCLUDE_PATHS)
+    assert len(out) <= 64
+
+
+def test_empty_exclude_paths_falls_back_rather_than_logging_everything():
+    """An empty list would re-enable logging of health checks and the docs, and
+    flood the table — treat it as 'not configured'."""
+    assert normalize_exclude_paths([], DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+    assert normalize_exclude_paths(None, DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+
+
+def test_partial_mapping_keeps_the_other_defaults():
+    cfg = config_from_mapping({"sample_rate": 0.5})
+    assert cfg.sample_rate == 0.5
+    assert cfg.success_retention_days == DEFAULT_CONFIG.success_retention_days
+    assert cfg.enabled is DEFAULT_CONFIG.enabled
+
+
+# --------------------------------------------------------------------------
+# refresh_config
+# --------------------------------------------------------------------------
+
+def test_refresh_config_parses_the_raw_jsonb_strings_asyncpg_returns():
+    conn = AsyncMock()
+    conn.fetch = AsyncMock(return_value=[
+        {"key": "requestlog.enabled", "value": "false"},
+        {"key": "requestlog.max_body_bytes", "value": "4096"},
+        {"key": "requestlog.sample_rate", "value": "0.5"},
+        {"key": "requestlog.exclude_paths", "value": '["/api/health","/metrics"]'},
+    ])
+
+    with patch.object(request_log_settings, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.5
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+    set_config(DEFAULT_CONFIG)
+
+
+def test_refresh_config_keeps_the_previous_snapshot_on_db_failure():
+    """A transient pool error must not silently flip logging on or off."""
+    known = RequestLogConfig(enabled=False, sample_rate=0.1)
+    set_config(known)
+
+    with patch.object(request_log_settings, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.sample_rate == 0.1
+    set_config(DEFAULT_CONFIG)
+
+
+def test_get_config_is_synchronous_and_needs_no_database():
+    """The middleware calls this on every request; it must never await."""
+    assert not asyncio.iscoroutinefunction(get_config)
+    assert isinstance(get_config(), RequestLogConfig)
+
+
+# --------------------------------------------------------------------------
+# The Pydantic model the API exposes
+# --------------------------------------------------------------------------
+
+def test_api_model_defaults_match_the_dataclass():
+    from routers.request_logs import RequestLogSettings
+
+    model = RequestLogSettings().model_dump()
+    code = DEFAULT_CONFIG.as_dict()
+    for key, value in code.items():
+        assert model[key] == value, f"API model default for {key} disagrees with RequestLogConfig"
+
+
+@pytest.mark.parametrize("payload", [
+    {"max_body_bytes": 999999},
+    {"success_retention_days": 0},
+    {"error_retention_days": 400},
+    {"sample_rate": 1.5},
+    {"max_rows": 10},
+    {"prune_interval_minutes": 1},
+    {"exclude_paths": ["no-slash"]},
+    {"exclude_paths": ["/" + "x" * 300]},
+])
+def test_api_model_rejects_out_of_range_input(payload):
+    from pydantic import ValidationError
+
+    from routers.request_logs import RequestLogSettings
+
+    with pytest.raises(ValidationError):
+        RequestLogSettings(**payload)

backend/tests/test_request_log_sink.py

@@ -0,0 +1,276 @@
+"""v1.11.0: the batching writer must never slow down or break a request.
+
+One row per API call is the highest write volume in the system and the asyncpg
+pool (min=10/max=50) is shared with every handler and four background loops. So
+the hot path enqueues and returns; a single writer task batches and inserts.
+The properties pinned here:
+
+  * `offer()` never blocks and never raises — a full queue drops and counts;
+  * the parameter list stays aligned with the INSERT placeholders (a column
+    added to one and not the other would fail every write at runtime, in
+    production, with the migration already applied);
+  * a failed batch is dropped with a warning rather than killing the loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_settings  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+from utils.request_log_sink import (  # noqa: E402
+    RequestLogRow,
+    RequestLogSink,
+    _INSERT_SQL,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+
+def _row(**overrides):
+    base = dict(
+        request_id="abc123",
+        direction="inbound",
+        method="POST",
+        url="/api/backends",
+        path="/api/backends",
+        status_code=200,
+        duration_ms=12,
+        created_at=datetime(2026, 8, 11, 9, 0, tzinfo=timezone.utc),
+    )
+    base.update(overrides)
+    return RequestLogRow(**base)
+
+
+@pytest.fixture(autouse=True)
+def defaults(monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(sink_module, "get_config", lambda: request_log_settings._CACHE)
+
+
+def _set(monkeypatch, **overrides):
+    monkeypatch.setattr(request_log_settings, "_CACHE", replace(DEFAULT_CONFIG, **overrides))
+
+
+# --------------------------------------------------------------------------
+# SQL / parameter alignment
+# --------------------------------------------------------------------------
+
+def test_insert_placeholders_match_the_column_list():
+    columns = _INSERT_SQL.split("(", 1)[1].split(")", 1)[0]
+    n_columns = len([c for c in columns.split(",") if c.strip()])
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+
+    assert n_columns == n_placeholders, (
+        f"the INSERT names {n_columns} columns but binds {n_placeholders} placeholders — "
+        f"every write would fail at runtime, on a database where the migration has "
+        f"already succeeded"
+    )
+
+
+def test_row_produces_exactly_as_many_params_as_the_insert_binds():
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+    assert len(_row().to_params()) == n_placeholders, (
+        "RequestLogRow.to_params() drifted from _INSERT_SQL"
+    )
+
+
+def test_jsonb_params_are_serialized_strings_not_dicts():
+    """No JSONB codec is registered on this pool, so JSONB values travel as text
+    and are cast in SQL — handing asyncpg a dict raises."""
+    row = _row(
+        query_params={"page": "2"},
+        request_headers={"content-type": "application/json"},
+        request_body_value={"name": "web"},
+    )
+    params = row.to_params()
+
+    for value in params:
+        assert not isinstance(value, (dict, list)), (
+            f"{value!r} was passed as a Python container; asyncpg cannot bind it to a "
+            f"jsonb parameter"
+        )
+
+    assert json.loads(params[6]) == {"page": "2"}
+
+
+def test_client_ip_is_never_a_placeholder_string():
+    """client_ip is an INET column: 'unknown' or a comma-joined X-Forwarded-For
+    raises on INSERT."""
+    params = _row(client_ip=None).to_params()
+    assert params[12] is None
+
+
+def test_status_class_is_zero_when_there_was_no_response():
+    assert _row(status_code=None).status_class == 0
+    assert _row(status_code=204).status_class == 2
+    assert _row(status_code=503).status_class == 5
+
+
+# --------------------------------------------------------------------------
+# offer(): the hot path
+# --------------------------------------------------------------------------
+
+def test_offer_drops_and_counts_when_the_queue_is_full():
+    sink = RequestLogSink(maxsize=3, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(10):
+            sink.offer(_row())
+
+    asyncio.run(run())
+
+    assert sink.stats["queued"] == 3
+    assert sink.stats["dropped"] == 7, (
+        "a full queue must drop and count, never block the request or raise"
+    )
+
+
+def test_offer_never_raises_on_a_broken_row():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(None)  # not a RequestLogRow at all
+
+    asyncio.run(run())  # must not raise
+
+
+def test_offer_respects_the_kill_switch(monkeypatch):
+    _set(monkeypatch, enabled=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    asyncio.run(_offer(sink, _row()))
+    assert sink.stats["queued"] == 0
+
+
+def test_offer_respects_the_per_direction_switches(monkeypatch):
+    _set(monkeypatch, capture_outbound=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row(direction="outbound", target="acme"))
+        sink.offer(_row(direction="inbound"))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 1
+
+
+def test_sampling_never_drops_errors(monkeypatch):
+    """A sample rate of zero must still capture every failure — that is the whole
+    point of sampling successes only."""
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(20):
+            sink.offer(_row(status_code=200))
+        for _ in range(5):
+            sink.offer(_row(status_code=500))
+        for _ in range(5):
+            sink.offer(_row(status_code=None))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 10, (
+        "sampling removed error rows; only 1xx/2xx/3xx inbound traffic may be sampled out"
+    )
+
+
+def test_sampling_does_not_touch_outbound_rows(monkeypatch):
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row(direction="outbound", target="acme", status_code=200))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 5, (
+        "outbound calls are low-volume and high-value; sampling them away hides which CA "
+        "or DNS call was made"
+    )
+
+
+def test_capture_bodies_off_strips_the_payload_before_queueing(monkeypatch):
+    _set(monkeypatch, capture_bodies=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    row = _row(request_body_raw=b'{"a":1}', request_body_bytes=7)
+
+    asyncio.run(_offer(sink, row))
+
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 7, "the size must survive so growth is still measurable"
+
+
+async def _offer(sink, row):
+    sink.offer(row)
+
+
+# --------------------------------------------------------------------------
+# The writer
+# --------------------------------------------------------------------------
+
+def test_a_batch_is_written_with_one_executemany():
+    conn = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            return await sink.flush(timeout=1.0)
+
+    written = asyncio.run(run())
+
+    assert written == 5
+    assert conn.executemany.await_count == 1, (
+        "rows were inserted one at a time; that is one pool acquire per API call and the "
+        "pool has 50 connections"
+    )
+    sql, params = conn.executemany.await_args.args
+    assert "INSERT INTO request_logs" in sql
+    assert len(params) == 5
+
+
+def test_a_failed_batch_does_not_kill_the_writer():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("relation does not exist"))
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())  # must not raise
+    assert sink.stats["failed_batches"] == 1
+
+
+def test_the_connection_is_released_even_when_the_write_fails():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("boom"))
+    release = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", release):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())
+    assert release.await_count == 1, "a failed batch leaked a pooled connection"
+
+
+def test_flush_on_an_empty_queue_is_a_noop():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    assert asyncio.run(sink.flush(timeout=0.1)) == 0

backend/tests/test_router_path_shadowing.py

@@ -0,0 +1,141 @@
+"""
+v1.10.6 — literal API paths must never be declared after a parameterised one.
+
+Found in production on the v1.10.4 VIP-adoption feature: `@router.get("/discoveries")`
+sat at the bottom of routers/vip.py, below `@router.get("/{vip_id}")`. FastAPI matches
+routes in DECLARATION order, so every `GET /api/vip/discoveries` was answered by the
+`/{vip_id}` handler, which declares `vip_id: int` and therefore rejected the request with
+422 before `list_vip_discoveries` ever ran.
+
+Nothing about that failure was visible. The agents reported their discoveries correctly,
+the rows landed in `vip_discoveries`, and the HA/VIP page treats any non-OK response as
+"nothing to show" — so the adoption feature simply did not exist as far as the UI was
+concerned, with no error anywhere.
+
+These tests are a STATIC source scan on purpose: no imports, no app construction, no DB.
+They therefore also cover routers that cannot be imported in a bare test environment, and
+they keep covering routes added in the future.
+"""
+import pathlib
+import re
+
+import pytest
+
+ROUTERS_DIR = pathlib.Path(__file__).resolve().parents[1] / "routers"
+
+# `@router.get("/x")`, `@some_router.post("/x", ...)` — the path is the first string arg.
+_DECORATOR = re.compile(r'^@(?:\w+)\.(get|post|put|delete|patch)\(\s*[\'"]([^\'"]*)[\'"]')
+
+
+def _routes(source: str):
+    """[(line_no, verb, path)] in declaration order."""
+    out = []
+    for line_no, line in enumerate(source.splitlines(), 1):
+        match = _DECORATOR.match(line)
+        if match:
+            out.append((line_no, match.group(1), match.group(2)))
+    return out
+
+
+def _shadows(earlier: str, later: str) -> bool:
+    """True if `earlier` (declared first) swallows the literal path `later`.
+
+    Only literal paths can be silently swallowed, and only by a path that has the same
+    number of segments where every non-placeholder segment matches. The collection route
+    ("" or "/") is its own path and never collides.
+    """
+    if not later.strip("/") or not earlier.strip("/"):
+        return False
+    if "{" in later:
+        return False
+    if "{" not in earlier:
+        return False
+    earlier_segments = earlier.strip("/").split("/")
+    later_segments = later.strip("/").split("/")
+    if len(earlier_segments) != len(later_segments):
+        return False
+    return all(
+        e.startswith("{") or e == l
+        for e, l in zip(earlier_segments, later_segments)
+    )
+
+
+def _shadowed_routes(path: pathlib.Path):
+    routes = _routes(path.read_text())
+    found = []
+    for index, (line_no, verb, route_path) in enumerate(routes):
+        for prior_line, prior_verb, prior_path in routes[:index]:
+            if prior_verb == verb and _shadows(prior_path, route_path):
+                found.append(
+                    f"{path.name}:{line_no} {verb.upper()} {route_path} is swallowed by "
+                    f"{prior_path} declared at line {prior_line}"
+                )
+    return found
+
+
+# ----------------------------------------------------------------------------
+# 1. The specific regression: /api/vip/discoveries must outrank /{vip_id}
+# ----------------------------------------------------------------------------
+
+def test_vip_discoveries_declared_before_vip_id():
+    routes = _routes((ROUTERS_DIR / "vip.py").read_text())
+    get_paths = [path for _line, verb, path in routes if verb == "get"]
+
+    assert "/discoveries" in get_paths, "the discoveries endpoint disappeared"
+    assert "/{vip_id}" in get_paths, "the get-one endpoint disappeared"
+    assert get_paths.index("/discoveries") < get_paths.index("/{vip_id}"), (
+        "GET /discoveries is declared after GET /{vip_id}; FastAPI will route "
+        "/api/vip/discoveries into get_vip and answer 422, silently emptying the "
+        "adoption panel"
+    )
+
+
+# ----------------------------------------------------------------------------
+# 2. The general guard: no literal path anywhere is shadowed
+# ----------------------------------------------------------------------------
+
+def test_no_literal_route_is_shadowed_in_any_router():
+    problems = []
+    for router_file in sorted(ROUTERS_DIR.glob("*.py")):
+        problems.extend(_shadowed_routes(router_file))
+
+    assert not problems, (
+        "literal route(s) declared after a parameterised route that swallows them:\n  "
+        + "\n  ".join(problems)
+    )
+
+
+# ----------------------------------------------------------------------------
+# 3. The detector itself must actually detect (guards against a vacuous pass)
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize(
+    "earlier,later,expected",
+    [
+        ("/{vip_id}", "/discoveries", True),        # the v1.10.4 bug
+        ("/{vip_id}", "/{other}", False),           # two placeholders never shadow
+        ("/{vip_id}", "", False),                   # collection route is its own path
+        ("/{vip_id}/apply", "/adopt", False),       # different segment counts
+        ("/{vip_id}/apply", "/adopt/now", False),   # literal mismatch in segment 2
+        ("/{vip_id}/{action}", "/adopt/now", True), # both segments placeheld
+        ("/vips", "/discoveries", False),           # literal never shadows a literal
+    ],
+)
+def test_shadow_detector_semantics(earlier, later, expected):
+    assert _shadows(earlier, later) is expected
+
+
+def test_detector_flags_the_original_declaration_order():
+    """A synthetic file in the pre-fix order must be reported, so a future refactor that
+    breaks the detector cannot make the guard above pass vacuously."""
+    source = (
+        '@router.get("")\n'
+        "async def list_vips(): ...\n"
+        '@router.get("/{vip_id}")\n'
+        "async def get_vip(vip_id: int): ...\n"
+        '@router.get("/discoveries")\n'
+        "async def list_vip_discoveries(): ...\n"
+    )
+    routes = _routes(source)
+    assert [verb for _l, verb, _p in routes] == ["get", "get", "get"]
+    assert _shadows(routes[1][2], routes[2][2]) is True

backend/tests/test_security_advisories_auth.py

@@ -0,0 +1,223 @@
+"""Regression tests for the 2026-07 security advisories.
+
+Covers:
+- GHSA-7rhv-c5pc-69r8 (CRITICAL RCE): agent script-template management must
+  require the agents.version permission, not merely authentication.
+- GHSA-3p5c-m5m4-mjpx (missing auth): agent data-plane endpoints must require a
+  valid X-API-Key, and operator/UI endpoints must require a JWT. An anonymous
+  caller must never get a 200 with sensitive data.
+
+These are behavioral assertions via FastAPI's TestClient. The auth checks were
+deliberately moved ahead of any DB access, so an unauthenticated request is
+rejected without needing a database — the same approach as the existing
+test_ssl_list_endpoint_auth.py. Accepted rejection statuses are 401/403/422
+(never 200-with-data).
+"""
+import re
+import os
+import pytest
+
+REJECT = (401, 403, 422)
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: agent data-plane endpoints must reject a missing X-API-Key
+# --------------------------------------------------------------------------
+
+def test_agent_config_requires_api_key(client):
+    """GET /api/agents/{name}/config leaked the full haproxy.cfg without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/config")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: agent config served without X-API-Key ({res.status_code})"
+    )
+
+
+def test_agent_ssl_certificates_requires_api_key(client):
+    """GET /api/agents/{name}/ssl-certificates leaked SSL private keys without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/ssl-certificates")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: SSL certs (private keys!) served without X-API-Key ({res.status_code})"
+    )
+    if res.status_code == 200:
+        assert "private_key_content" not in res.text
+
+
+def test_agent_upgrade_status_requires_api_key(client):
+    res = client.get("/api/agents/prod-haproxy-1/upgrade-status")
+    assert res.status_code in REJECT
+
+
+def test_agent_pending_requests_requires_api_key(client):
+    res = client.get("/api/configuration/agents/prod-haproxy-1/pending-requests")
+    assert res.status_code in REJECT
+
+
+def test_agent_heartbeat_by_name_requires_api_key(client):
+    res = client.post("/api/agents/heartbeat", json={"name": "rogue-poc"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless heartbeat/auto-register accepted ({res.status_code})"
+    )
+
+
+def test_agent_heartbeat_by_id_requires_api_key(client):
+    res = client.post("/api/agents/1/heartbeat", json={"name": "spoofed"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless by-id heartbeat state-spoof accepted ({res.status_code})"
+    )
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: operator/UI endpoints must reject a missing JWT
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_requires_jwt(client):
+    """GET /api/agents (RCE read-back channel) was served without a JWT."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", ["/api/health/deep", "/api/health/agents", "/api/health/clusters"])
+def test_detailed_health_requires_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, f"{path} served without a JWT ({res.status_code})"
+
+
+def test_simple_health_stays_public(client):
+    """The liveness probe endpoint (/api/health) must remain UNAUTHENTICATED.
+
+    It reports 200 when healthy and 503 when the DB is unreachable (as in this
+    no-DB test env); what matters for the k8s probe is that it is never gated
+    behind auth (401/403). We only added auth to /api/health/{deep,agents,clusters}.
+    """
+    res = client.get("/api/health")
+    assert res.status_code not in (401, 403), (
+        f"Regression: /api/health liveness probe now requires auth ({res.status_code}) — "
+        f"this breaks k8s liveness/readiness"
+    )
+
+
+def test_dashboard_stats_requires_jwt(client):
+    res = client.get("/api/dashboard-stats/stats?cluster_id=1")
+    assert res.status_code in REJECT
+
+
+def test_ssl_config_versions_requires_jwt(client):
+    res = client.get("/api/ssl/certificates/1/config-versions")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-7rhv (CRITICAL RCE): script-template management
+# --------------------------------------------------------------------------
+
+def test_script_template_write_requires_auth(client):
+    """Anonymous POST must be rejected outright."""
+    res = client.post("/api/agents/script-templates/linux",
+                      json={"script_content": "#!/bin/bash\nid", "version": "9.9.9"})
+    assert res.status_code in REJECT
+
+
+def test_script_template_read_requires_auth(client):
+    res = client.get("/api/agents/script-templates/linux")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): sibling endpoints exposing the SAME class of data
+# (found during post-merge review — must also require a JWT)
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/haproxy-cluster-pools/1/agents",   # full agent inventory — same class as GET /api/agents
+    "/api/pools",
+    "/api/haproxy-cluster-pools",
+    "/api/dashboard/stats",
+    "/api/dashboard/overview",               # optional-auth pattern — leaked stats/names/alerts anonymously
+    "/api/haproxy/stats",
+    "/api/waf/rules",
+    "/api/health/errors",
+])
+def test_sibling_inventory_endpoints_require_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c (round 2) regression: {path} served without a JWT ({res.status_code}) — "
+        f"anonymous access to inventory/topology/WAF/error data"
+    )
+
+
+def test_pool_agents_no_anonymous_inventory_leak(client):
+    """The richest bypass: /api/haproxy-cluster-pools/{id}/agents must not leak inventory."""
+    res = client.get("/api/haproxy-cluster-pools/1/agents")
+    assert res.status_code in REJECT
+    if res.status_code == 200:
+        assert "ip_address" not in res.text and "hostname" not in res.text
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): agent webhooks must return 401 (not a 200 error body)
+# for anonymous callers — the auth raise must propagate, not be swallowed.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/agents/some-agent/config-applied",
+    "/api/agents/some-agent/config-validation-failed",
+    "/api/agents/some-agent/config-sync",
+])
+def test_agent_webhooks_reject_anonymous_with_401(client, path):
+    res = client.post(path, json={})
+    assert res.status_code in REJECT, (
+        f"{path} returned {res.status_code} for an anonymous caller — the auth "
+        f"rejection must be a 401/403, not a swallowed 200 error body"
+    )
+    # Specifically must NOT be a 200 "status: error" body.
+    assert res.status_code != 200
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): GET /api/agents must accept EITHER a JWT OR an agent
+# X-API-Key. Anonymous (neither) is still rejected — agents send a key, so a
+# JWT-only gate would break them (verified end-to-end in the localtest smoke).
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_still_rejects_fully_anonymous(client):
+    """No JWT and no X-API-Key -> 401 (the agent-key accept path needs a valid key)."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+def test_generate_uninstall_script_requires_auth(client):
+    """Agent-management endpoint must not be anonymously reachable (JWT or agent key)."""
+    res = client.get("/api/agents/generate-uninstall-script/linux")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", [
+    "/api/config/validate",
+    "/api/config/optimize",
+    "/api/config/templates/default/generate",
+])
+def test_config_compute_endpoints_require_auth(client, path):
+    """Config compute endpoints (run a HAProxy validator on caller input) were
+    optional-auth; now require a JWT. The dependency rejects before body parsing."""
+    res = client.post(path, json={})
+    assert res.status_code in REJECT
+
+
+def test_script_template_write_enforces_agents_version_permission():
+    """Static guarantee: the write handler checks agents.version (not just authN).
+
+    A behavioral 403-for-viewer test would need a seeded DB + a minted viewer JWT;
+    instead we assert the permission gate is present in source, mirroring the
+    existing audit-style source tests. This is the core RCE fix (GHSA-7rhv).
+    """
+    src_path = os.path.join(os.path.dirname(__file__), "..", "routers", "agent.py")
+    with open(src_path, "r") as f:
+        src = f.read()
+    # Isolate the save_agent_script_template handler body.
+    m = re.search(r"async def save_agent_script_template\(.*?\n(.*?)\n@router\.", src, re.DOTALL)
+    assert m, "save_agent_script_template handler not found"
+    body = m.group(1)
+    assert 'check_user_permission' in body and '"agents", "version"' in body, (
+        "GHSA-7rhv regression: script-template WRITE no longer enforces the "
+        "agents.version permission — any JWT holder could poison the root install script"
+    )

backend/tests/test_site_wizard_round11.py

@@ -276,14 +276,16 @@ def test_categorize_routes_directives_correctly():
 
 def test_emit_buckets_flushed_in_canonical_order():
     """The flush block at end of frontend processing must list buckets
-    in: prelude → stick → tcp_req → acl → http_req → http_resp →
+    in: prelude → filter → stick → tcp_req → acl → http_req → http_resp →
     redirect → use_be → default_be. Pre-fix `http-request` rules
     interleaved with `use_backend` rules in source order, producing
-    HAProxy parser warnings."""
+    HAProxy parser warnings. (Issue #38 added the `filter` bucket, flushed
+    right after `prelude` so SPOE `filter` lines precede `send-spoe-group`.)"""
     src = _gen_src()
     flush_match = re.search(
         r'for\s+_bucket_key\s+in\s+\(\s*'
         r'"prelude"\s*,\s*'
+        r'"filter"\s*,\s*'
         r'"stick"\s*,\s*'
         r'"tcp_req"\s*,\s*'
         r'"acl"\s*,\s*'

backend/tests/test_spoe_filter_import.py

@@ -0,0 +1,203 @@
+"""
+Issue #38 regression tests: HAProxy SPOE `filter` + frontend `log-format` support.
+
+Bug: the bulk-config parser recognised only a fixed set of frontend directives,
+so `filter spoe engine coraza config ...` and `log-format ...` were silently
+dropped on import / manual edit. This regenerated a config missing the SPOE
+engine definition, so HAProxy failed with
+"unable to find SPOE engine 'coraza' used by the send-spoe-group 'coraza-req'".
+
+These tests verify the end-to-end fix without requiring a database:
+1. parser captures `filter` + `log-format` into the new ParsedFrontend fields;
+2. `http-request send-spoe-group` is still preserved (regression guard);
+3. the generator's directive categoriser + bucket flush order emit `filter`
+   BEFORE the `http-request send-spoe-group` rules and keep `log-format`;
+4. reject/rollback restores the new columns;
+5. a non-SPOE frontend is completely unaffected (zero-impact).
+"""
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.haproxy_config_parser import parse_haproxy_config, ParsedFrontend
+from services.haproxy_config import _categorize_haproxy_directive
+from models.frontend import FrontendConfig
+
+
+# The exact frontend/backend config reported in Issue #38 (Coraza-SPOA).
+ISSUE_38_CONFIG = r"""
+frontend web-frontend
+    bind *:8073
+    mode http
+        log-format "%ci:%cp\ [%t]\ %ft\ %b/%s\ %ST\ %B\ %{+Q}r\ %[var(txn.coraza.id)]\ waf-hit:\ %[var(txn.coraza.fail)]"
+        filter spoe engine coraza config /etc/haproxy/coraza.cfg
+      http-request set-var(txn.coraza.app) str(haproxy_waf)
+     http-request send-spoe-group coraza coraza-req
+    http-request deny if { var(txn.coraza.fail) -m int eq 1 }
+    default_backend web-backend
+
+backend web-backend
+    balance roundrobin
+    mode http
+    server server1 192.168.1.10:443 weight 100 ssl verify none
+
+backend coraza-spoa
+    mode tcp
+    option spop-check
+    server coraza_spoa 192.168.12.21:9000
+"""
+
+
+def _get_frontend(parse_result, name):
+    for fe in parse_result.frontends:
+        if fe.name == name:
+            return fe
+    return None
+
+
+class TestParserCapturesSpoe:
+    def test_filter_and_log_format_captured(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe is not None, "web-frontend should be parsed and kept"
+        assert fe.filters is not None
+        assert "filter spoe engine coraza config /etc/haproxy/coraza.cfg" in fe.filters
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format")
+        # the escaped/quoted format string must be preserved verbatim
+        assert "%[var(txn.coraza.fail)]" in fe.log_format
+
+    def test_send_spoe_group_still_preserved(self):
+        # Regression guard: http-request rules (incl. send-spoe-group) must
+        # still be collected into request_headers as before.
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe.request_headers is not None
+        assert "send-spoe-group coraza coraza-req" in fe.request_headers
+
+    def test_multiple_filters_preserved_in_order(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    filter compression
+    filter spoe engine coraza config /etc/haproxy/coraza.cfg
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        lines = fe.filters.split("\n")
+        assert lines == [
+            "filter compression",
+            "filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+        ]
+
+    def test_log_format_sd_variant_captured(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    log-format-sd "[exampleSDID@1234 field=value]"
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format-sd")
+
+
+class TestGeneratorOrderingContract:
+    """The generator routes directives into ordered buckets. Verify SPOE
+    correctness at the (pure) categoriser + documented flush-order level."""
+
+    def test_filter_routes_to_filter_bucket(self):
+        assert _categorize_haproxy_directive("    filter spoe engine coraza config /x.cfg") == "filter"
+
+    def test_send_spoe_group_routes_to_http_req(self):
+        assert _categorize_haproxy_directive("    http-request send-spoe-group coraza coraza-req") == "http_req"
+
+    def test_log_format_routes_to_prelude(self):
+        assert _categorize_haproxy_directive('    log-format "%ci:%cp"') == "prelude"
+        assert _categorize_haproxy_directive('    log-format-sd "[x]"') == "prelude"
+
+    def test_flush_order_places_filter_before_http_req(self):
+        # The bucket flush order is the single source of truth for emission
+        # ordering. Assert `filter` is flushed before `http_req` (and after
+        # `prelude`), guaranteeing `filter ...` renders before
+        # `http-request send-spoe-group ...`.
+        src = _read_source("services/haproxy_config.py")
+        m = re.search(r"for _bucket_key in \((.*?)\):", src, re.DOTALL)
+        assert m, "bucket flush loop not found"
+        order = re.findall(r'"(\w+)"', m.group(1))
+        assert "filter" in order, "new 'filter' bucket missing from flush order"
+        assert order.index("prelude") < order.index("filter") < order.index("http_req")
+
+
+class TestModelAndRollback:
+    def test_model_has_passthrough_fields(self):
+        fc = FrontendConfig(
+            name="f", bind_port=80,
+            filters="filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+            log_format='log-format "%ci"',
+        )
+        assert fc.filters.startswith("filter spoe")
+        assert fc.log_format.startswith("log-format")
+
+    def test_dataclass_defaults_none(self):
+        fe = ParsedFrontend(name="f")
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_rollback_restores_new_columns(self):
+        # Reject/rollback of a frontend UPDATE must restore the new columns,
+        # otherwise the rejected (new) filters/log_format would persist.
+        src = _read_source("utils/entity_snapshot.py")
+        assert "log_format = $" in src
+        assert "filters = $" in src
+        assert "old_values.get('log_format')" in src
+        assert "old_values.get('filters')" in src
+
+
+class TestZeroImpact:
+    def test_non_spoe_frontend_unaffected(self):
+        cfg = """
+frontend plain
+    bind *:80
+    mode http
+    option httplog
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "plain")
+        # No filter / log-format present → new fields stay None (no behaviour change)
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_spop_check_backend_roundtrips_without_warning(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        be = next((b for b in result.backends if b.name == "coraza-spoa"), None)
+        assert be is not None, "coraza-spoa backend should import"
+        assert be.mode == "tcp"
+        assert be.options and "option spop-check" in be.options
+        # spop-check is now a known option → no spurious 'unknown option' warning
+        assert not any(
+            "coraza-spoa" in w and "spop-check" in w and "Unknown" in w
+            for w in result.warnings
+        )
+
+
+def _read_source(relpath):
+    base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    with open(os.path.join(base, relpath), "r", encoding="utf-8") as fh:
+        return fh.read()

backend/tests/test_ssrf_guard.py

@@ -0,0 +1,68 @@
+"""Unit tests for the SSRF guard (GHSA-3vh4-gvxx-wm2p).
+
+The guard protects server-side fetches of ACME `directory_url` values. This
+project uses only public ACME CAs, so every non-public IP must be rejected.
+Tests avoid real network/DNS by using IP literals and scheme checks.
+"""
+import asyncio
+import pytest
+
+from utils.ssrf_guard import is_public_ip, assert_public_url, SSRFValidationError
+
+
+# ---- is_public_ip -----------------------------------------------------------
+
+@pytest.mark.parametrize("ip", [
+    "8.8.8.8", "1.1.1.1", "93.184.216.34",  # public
+])
+def test_public_ips_allowed(ip):
+    assert is_public_ip(ip) is True
+
+
+@pytest.mark.parametrize("ip", [
+    "127.0.0.1",            # loopback
+    "10.0.0.5",             # RFC1918
+    "172.19.0.1",           # RFC1918 (the SSRF PoC docker gateway)
+    "192.168.1.1",          # RFC1918
+    "169.254.169.254",      # link-local / cloud metadata
+    "0.0.0.0",              # unspecified
+    "::1",                  # IPv6 loopback
+    "fe80::1",              # IPv6 link-local
+    "::ffff:127.0.0.1",     # IPv4-mapped IPv6 loopback (R18c bypass)
+    "::ffff:169.254.169.254",  # IPv4-mapped metadata
+    "not-an-ip",            # garbage
+])
+def test_non_public_ips_rejected(ip):
+    assert is_public_ip(ip) is False
+
+
+# ---- assert_public_url ------------------------------------------------------
+
+def _raises(url):
+    with pytest.raises(SSRFValidationError):
+        asyncio.run(assert_public_url(url))
+
+
+def test_rejects_non_https_scheme():
+    # The SSRF PoC used http:// against an internal listener.
+    _raises("http://172.19.0.1:2121/internal-secret")
+    _raises("http://8.8.8.8/")            # even a public IP over http is refused
+    _raises("file:///etc/passwd")
+    _raises("gopher://8.8.8.8/")
+
+
+def test_rejects_private_ip_literals():
+    _raises("https://127.0.0.1/")
+    _raises("https://10.0.0.5/")
+    _raises("https://169.254.169.254/latest/meta-data/")
+    _raises("https://[::1]/")
+
+
+def test_rejects_empty_or_hostless():
+    _raises("")
+    _raises("https://")
+
+
+def test_allows_public_ip_literal_https():
+    # A public IP literal over https must pass (no DNS needed).
+    asyncio.run(assert_public_url("https://8.8.8.8/directory"))