fix(vip): make the Adoptable tag name the real blocker (v1.10.11)

The tag and the disabled Adopt button were computed by two separate ladders and
could disagree. Seen on a live pair: one node's keepalived.conf had an
unbalanced brace, so it was excluded from the instance; its partner was then
tagged "MASTER missing" - technically true, because the unreadable node's
state MASTER had not been counted - while the actual reason (the peer cannot be
taken over, so adopting would strand it) sat only in the button's tooltip. The
label pointed the operator at the wrong node.

groupState now makes ONE ordered decision and returns the label, its colour and
the reason together, so the tag can never describe a different condition than
the one disabling the button. A group held up by a node that references the
same address but cannot be adopted with it reads "blocked by peer"; two MASTERs
is distinguished from none.

Display only: the endpoint's checks and refusals are untouched.

Backend suite: 1366 passed, 152 skipped. Frontend build clean.

.env.template

@@ -17,6 +17,34 @@ REDIS_URL=redis://redis:6379
 # Change this to a strong random string in production
 SECRET_KEY=your-secret-key-change-this-in-production
 
+# ----------------------------------------------------------------------------
+# Optional per-purpose encryption keys.
+#
+# Every secret the application stores is encrypted at rest with Fernet. Each class
+# derives its own key, so rotating one never affects another. If a variable below is
+# unset, that class's key is derived from SECRET_KEY via HKDF — which works, but means
+# rotating SECRET_KEY makes the existing values of that class UNDECRYPTABLE. Set an
+# explicit key (urlsafe-base64, 32 bytes) in production if you want independent
+# rotation. Generate one with:
+#   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
+# ----------------------------------------------------------------------------
+
+# VRRP secrets for HA/VIP (Issue #27).
+# VIP_ENCRYPTION_KEY=
+
+# TOTP secrets for multi-factor authentication (Issue #18).
+# MFA_ENCRYPTION_KEY=
+
+# Per-account DNS provider credentials for ACME DNS-01 (Issue #35).
+# Rotating this without re-entering credentials makes DNS-01 renewals fail until the
+# affected accounts' credentials are re-saved in ACME Automation.
+# DNS_PROVIDER_ENCRYPTION_KEY=
+
+# Private keys of PENDING CSRs, held only until the signed certificate is imported
+# (Issue #53). Rotating this while CSRs are out for signature makes those CSRs
+# unusable — they must be deleted and re-created.
+# CSR_ENCRYPTION_KEY=
+
 # ============================================================================
 # PUBLIC URL CONFIGURATION
 # ============================================================================
@@ -31,6 +59,13 @@ PUBLIC_URL=http://localhost:8000
 
 # Management base URL (defaults to PUBLIC_URL if not set)
 # Override this if your management interface is on a different URL
+#
+# This is also the last-resort fallback for the ACME HTTP-01 challenge backend,
+# i.e. the address written into haproxy.cfg as `server _acme_mgmt <host>:<port>`.
+# That address is resolved BY HAPROXY, ON THE HAPROXY NODE. If HAProxy runs
+# anywhere other than this machine, localhost points at the wrong box and HTTP-01
+# validation fails while DNS-01 keeps working. Set a routable address, with the
+# port, e.g. MANAGEMENT_BASE_URL=http://10.90.1.4:8080
 MANAGEMENT_BASE_URL=http://localhost:8000
 
 # ============================================================================
@@ -54,6 +89,15 @@ AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
 # Config sync interval in seconds
 AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 
+# ============================================================================
+# BACKEND PERFORMANCE
+# ============================================================================
+# Number of uvicorn worker processes for the backend API (default: 1).
+# On multi-core hosts, setting this to the core count (e.g. 2) lets the API
+# use all cores. Safe to increase: background tasks are multi-replica safe
+# (the k8s deployment already runs 2+ replicas via HPA).
+UVICORN_WORKERS=1
+
 # ============================================================================
 # CORS CONFIGURATION
 # ============================================================================

.github/workflows/docker-build.yml

@@ -7,6 +7,8 @@ on:
 jobs:
   build_and_push:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
 
     steps:
       - name: checkout
@@ -19,13 +21,17 @@ jobs:
       - name: read product version
         id: prodversion
         run: |
-          VERSION=$(jq -r .version version.json)
+          VERSION=$(jq -r .version backend/version.json)
           if [ -z "$VERSION" ] || [ "$VERSION" = "null" ]; then
-            echo "Failed to read product version from version.json" >&2
+            echo "Failed to read product version from backend/version.json" >&2
             exit 1
           fi
           echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
 
+      # version.json now lives at backend/version.json (inside the ./backend build
+      # context), so `COPY . .` bakes it into the image directly — no staging step
+      # is needed and the backend reports the correct version in every deployment,
+      # not just this workflow's builds.
       - name: set up qemu
         uses: docker/setup-qemu-action@v3
 
@@ -62,3 +68,30 @@ jobs:
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.version.outputs.TAG }}
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.prodversion.outputs.VERSION }}
 
+      # Keep the GitHub Releases/Tags in sync with version.json. The docker
+      # images above are tagged with the product version, but nothing here
+      # created the matching git tag, so the repo's Tags/Releases drifted
+      # behind (stuck at the last manually-created tag). After the images are
+      # pushed, cut a Release (which also creates the tag) for the current
+      # version.json, but only if one does not already exist, so re-runs
+      # without a version bump are a no-op.
+      - name: create github release from version.json
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          VERSION="${{ steps.prodversion.outputs.VERSION }}"
+          TAG="v${VERSION}"
+          RELEASE_NAME=$(jq -r '.releaseName // empty' backend/version.json)
+          if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "Release $TAG already exists, skipping."
+          else
+            TITLE="$TAG"
+            [ -n "$RELEASE_NAME" ] && TITLE="$TAG — $RELEASE_NAME"
+            gh release create "$TAG" \
+              --repo "$GITHUB_REPOSITORY" \
+              --target "$GITHUB_SHA" \
+              --title "$TITLE" \
+              --notes "Automated release for $TAG (from version.json)."
+            echo "Created release $TAG"
+          fi
+

.gitignore

@@ -39,9 +39,6 @@ venv.bak/
 *.sqlite
 *.sqlite3
 
-# Docker
-.dockerignore
-
 # IDE
 .vscode/
 .idea/

LICENSE

@@ -1,22 +1,679 @@
-MIT License
+HAProxy OpenManager
+Copyright (C) 2025-2026  Taylan Bakırcıoğlu and HAProxy OpenManager Contributors
 
-Copyright (c) 2025 HAProxy OpenManager Contributors
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+---
 
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.

UPGRADE_GUIDE.md

@@ -1,3 +1,424 @@
+# Upgrade Notes — v1.10.11 (Adoptable tag names the real blocker)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The *Adoptable* tag and the disabled *Adopt* button were derived separately, so they could
+  name different problems. A pair blocked by a peer whose config could not be parsed showed
+  **MASTER missing**, because the unreadable node's `state MASTER` had not been counted — true,
+  but it sent the operator to the wrong node. Both now come from one ordered decision.
+- New label **blocked by peer** for a group held up by a node that references the same address
+  but cannot be taken over with it. Two MASTERs is now distinct from none.
+- Display only. The endpoint's checks and refusals are unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.10 (Adoption blockers listed once per instance)
+
+**Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.
+
+- The adoption panel merged every member's blocker list, so a two-node pair showed each shared
+  problem twice. The two files report different line numbers for the same directive, so exact
+  de-duplication did not collapse them. Blockers are now merged on the message with the leading
+  `line N:` ignored.
+- Display only. The endpoint already evaluated the combined set across all nodes, and what it
+  accepts or refuses is unchanged.
+
+**Rollback:** safe; purely presentational.
+
+---
+
+# Upgrade Notes — v1.10.9 (Adoption refuses to strand a node)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent change.
+
+- **Adoption will not leave a node behind.** v1.10.8 resolved the whole VRRP instance, but only
+  from nodes it could parse, that were enabled and that were in the same pool. Anything else fell
+  out of the set silently while its peers were rewritten. Adoption now refuses if any reported
+  `keepalived.conf` mentions the virtual address and is not among the nodes being taken over, and
+  says which node and why.
+- **The nodes must agree on the shared fields.** `prefix_length`, unicast/multicast mode, HAProxy
+  tracking and the VRRP password live on the VIP and are re-rendered onto every member, so one
+  node's value used to be imposed on the rest. A disagreement is now refused with both values
+  shown.
+- **The takeover authorisation is now retired on acknowledgement.** It is the permission to
+  overwrite a `keepalived.conf` that does not carry our marker. It was never cleared, so it stayed
+  valid for that exact file content indefinitely; restoring the pre-adoption file would have been
+  overwritten again without fresh approval. It is now dropped once the member acks our rendered
+  config, gated on the acked hash matching `applied_config_hash` so a failed deploy cannot strand
+  the VIP.
+- **Nothing to do on upgrade.** Existing adopted VIPs keep working; their authorisation is retired
+  on the next successful acknowledgement.
+
+**Rollback:** safe. No schema or data migration; reverting restores the previous (more permissive)
+adoption checks.
+
+---
+
+# Upgrade Notes — v1.10.8 (VIP adoption takes the whole VRRP instance)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact: nothing about what the agent reports or how it takes a
+config over changes.
+
+- **Adoption is now per VRRP instance, not per node.** Every node in the pool reporting the same
+  `virtual_router_id` and virtual address becomes a member of one VIP, each with the role,
+  priority and interface its own `keepalived.conf` declares, and each with its own one-shot
+  takeover hash. The panel lists one row per instance.
+- **Why this mattered:** single-node adoption could not produce a working pair. The BACKUP alone
+  failed apply, the MASTER alone left the peer unmanaged and the peer could not then be adopted
+  (VRID collision). On a **unicast** instance it was worse than inconvenient: the render drops
+  the unicast block when there are no peers, so the adopted node fell back to multicast while its
+  peer stayed unicast and both could hold the address.
+- **New refusals, each with the reason in the message:** the group does not have exactly one
+  MASTER; the nodes disagree on `advert_int`; a declared unicast peer is not among the nodes being
+  adopted; a node is already a member of a live VIP.
+- **Apply Management "View Change" now renders the adopt diff correctly.** It did not recognise
+  the `adopt` action and fell through to the generic HAProxy diff, which compared the staged
+  `keepalived.conf` against the cluster's previous `haproxy.cfg` and showed the whole HAProxy
+  config as removed. Alarming, but display-only — nothing was ever applied from that view.
+- **Rejecting an adoption is recoverable again.** It used to hide the node from the panel
+  permanently. Nothing clears `vip_discoveries.adopted_vip_id`, a VIP is only soft-deleted so the
+  column's `ON DELETE SET NULL` never fires, and the agent does not re-report a file whose hash
+  has not changed. Adoptability is now derived from whether the linked VIP is still active.
+- **If you adopted a VIP on 1.10.4-1.10.7**, check it before applying: it may have only one
+  member. Add the peer from the VIP's edit form, or reject the pending adoption and adopt again —
+  the node reappears in the panel under this release.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous single-node
+adoption behaviour.
+
+---
+
+# Upgrade Notes — v1.10.7 (HA / VIP follows the selected cluster)
+
+**Backend + frontend, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No agent impact.
+
+- **The HA / VIP page ignored the cluster picker.** Both the VIP table and the *Unmanaged
+  keepalived detected* panel queried the whole fleet, so on an install with more than one
+  cluster the lists never changed when the selection did. Both now pass `cluster_id`, mapped to
+  the cluster's pool the same way `GET /api/vip?cluster_id=` already worked for Apply
+  Management.
+- **Behaviour change worth knowing:** the VIP table is now scoped to the selected cluster. It
+  used to show every VIP in the fleet. If you relied on the fleet-wide view, the API still
+  supports it — `GET /api/vip` and `GET /api/vip/discoveries` without `cluster_id` return
+  everything, unchanged.
+- **API compatibility:** `cluster_id` is optional on both endpoints. Existing integrations that
+  do not send it behave exactly as before.
+
+**Rollback:** safe. The change is a query parameter plus the page that sends it; reverting
+restores the fleet-wide lists and touches no data.
+
+---
+
+# Upgrade Notes — v1.10.6 (VIP adoption panel was unreachable)
+
+**One backend fix, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. No API-shape change, no frontend change and zero agent impact.
+
+- **v1.10.4's adoption panel never appeared.** `GET /discoveries` was declared after
+  `GET /{vip_id}` in `routers/vip.py`. FastAPI matches routes in declaration order, so the
+  discovery list was routed into the get-one-VIP handler, which declares `vip_id: int` and
+  answered **422** before the real handler ran. The HA/VIP page treats any non-OK response as
+  "nothing to show", so the feature was invisible with no error in any log.
+- **Nothing was lost.** The agent side always worked: discoveries were reported and stored in
+  `vip_discoveries`. Deploy this backend and the rows appear immediately — no agent upgrade, no
+  re-sync of the agent script, no re-report needed.
+- **If you are upgrading straight from 1.10.3 or earlier**, follow the v1.10.4 notes below as
+  well: that release does bump `SCHEMA_VERSION` (10 → 11), which re-seeds the four built-in
+  roles, and its agent script has to reach the nodes before discovery starts.
+- **Regression guard.** A static source scan now fails the build if any literal API path in any
+  router is declared after a parameterised route that would swallow it. The whole router tree is
+  clean as of this release.
+
+**Rollback:** safe and immediate. The change is a route declaration order plus a test; reverting
+to 1.10.5 restores the previous (broken-panel) behaviour and touches no data.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.5 (HTTP-01 challenge backend on split deployments)
+
+**Bug fixes, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are **not**
+re-seeded. No API-shape change and zero agent impact.
+
+- **HTTP-01 could fail silently when HAProxy runs on different hosts than the management stack.**
+  The rendered config wrote `server _acme_mgmt <mgmt>:8080` from a value that defaults to
+  loopback — and HAProxy resolves that address **on the HAProxy node**, so it pointed at the wrong
+  box. Every check still reported success. The per-cluster `acme_backend_url` now has a UI field
+  (Cluster Management), changing it actually mints a config version, and the value is validated at
+  the write boundary.
+- **A config-generation failure could be pushed to agents as the cluster's whole `haproxy.cfg`.**
+  The generator reported failure by *returning* `# Error ...` instead of raising, and the apply
+  path hashed that comment and stored it as an APPLIED version. Both persisting call sites now
+  refuse with 422 and leave the running config in force. **This is worth knowing even if you never
+  touch ACME**, since any exception in the generator could trigger it.
+- **`frontends.mode` is nullable and was interpolated raw**, emitting a literal `mode None` that
+  HAProxy rejects — which fails the whole cluster config, not just that frontend. Normalised now.
+- **Cluster creation ignored the ACME fields**: a cluster created with ACME switched on came back
+  switched off, with no error.
+- **`docker-compose.yml` hardcoded `PUBLIC_URL` / `MANAGEMENT_BASE_URL`**, so a value in your
+  `.env` or host environment was silently ignored. They are interpolated now, with the previous
+  literals as defaults, so behaviour is unchanged unless you actually set them.
+- **Diagnostics stop over-reporting health.** The port-80 check now reads the body, so a reverse
+  proxy answering 200 with a web page is no longer counted as a working challenge endpoint. Every
+  new condition is a **warning, never a failure** — the Site Wizard blocks submit on a failing
+  check, so a new failing condition would have locked installs on upgrade day.
+- **Rollback:** downgrade freely. No schema or data change.
+
+---
+
+---
+
+# Upgrade Notes — v1.10.4 (Adopt an existing keepalived VIP)
+
+**Additive, but this release DOES bump the schema — read the role warning below.** Nothing on any
+node changes until you adopt a VIP and apply it.
+
+- **Schema:** `SCHEMA_VERSION` bumps to `11`, so on first start the (idempotent) migration
+  sequence re-runs once and adds **one new table** (`vip_discoveries`) plus two additive columns
+  (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`). **No existing table is
+  altered**, no existing row changes, and the admin password is not reset.
+- **⚠️ Built-in roles are re-seeded to their defaults** — the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump. If you customised `super_admin` / `operator` / `security_admin` /
+  `viewer`, **re-apply those changes after upgrading**. (The three previous releases did not bump
+  the version, so this is the first re-seed since v1.9.0.) No new permission strings are
+  introduced: discovery and adoption are governed by the existing `vip.read` / `vip.create`.
+- **⚠️ The Linux agent script changed, and discovery does not start until nodes run it.** The
+  fallback latest Linux agent version moves `2.0.0` → `2.1.0`, so nodes will pull the new script
+  through the normal agent-upgrade path. The addition is **read-only**: the agent reads the
+  `keepalived.conf` it does not own and reports it, rate-limited to once per content change. It
+  writes nothing new to the node. Until a node has upgraded, it simply never appears under
+  *Unmanaged keepalived detected*.
+- **Nothing is taken over implicitly.** The agent still refuses to overwrite a `keepalived.conf`
+  that lacks OpenManager's ownership marker. Adoption authorises exactly **one** takeover of
+  exactly the file that was analysed, pinned to its md5: if the file changes between adoption and
+  Apply, the agent refuses again and reports `externally_managed` rather than clobbering your
+  edit. Re-adopt to pick up the current file.
+- **Adoption can refuse, on purpose.** It replaces the file with OpenManager's render, so anything
+  the renderer cannot reproduce would be destroyed. Those directives are listed as blockers —
+  `notify_*` failover hooks, `vrrp_sync_group`, LVS `virtual_server` sections, a second address in
+  one instance, a custom `track_script`, extra `global_defs`. You can accept that loss explicitly
+  with a tick, but a value that is *unknown* rather than lost (an absent `virtual_router_id`, or
+  an address with no prefix length) cannot be waived — the VRID is fatal to guess and the prefix
+  has to be supplied, because picking a netmask for a live VIP would change its routing.
+- **Multi-node VIPs need every node.** Adoption covers the node that reported. Its unicast peers
+  hold their own `keepalived.conf`, so adopt or add them as members before applying — otherwise
+  the render has no peers. The UI says so after a successful adopt.
+- **Secrets:** the reported config may contain the VRRP `auth_pass`. It is split at ingest — the
+  password is Fernet-encrypted into its own column (same key path as `vip_instances`,
+  `VIP_ENCRYPTION_KEY` falling back to a key derived from `SECRET_KEY`) and the stored copy of the
+  file has it masked, so nothing readable through the API, the UI preview or a DB dump carries it
+  in cleartext.
+- **Rollback:** downgrading to 1.10.3 leaves `vip_discoveries` as an unused table and the two new
+  columns unread; managed VIPs keep working. One caveat: a VIP adopted on 1.10.4 but **not yet
+  applied** loses its takeover authorisation on downgrade, so the node's original config stays in
+  place and the VIP sits PENDING — harmless, but re-adopt after upgrading again. Agents already on
+  script 2.1.0 keep reporting discoveries to an endpoint that no longer exists; the report fails
+  quietly and nothing on the node is affected.
+
+---
+
+# Upgrade Notes — v1.10.3 (Multi-account ACME wizard fix)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change, no
+environment variable, zero agent impact. Installations with a single ACME account behave exactly as
+before.
+
+- **What was broken:** with **more than one** ACME account registered, the *Request ACME
+  Certificate* wizard did not honour the account you selected. Choosing an HTTP-01 account still
+  submitted a DNS-01 request, which the API rejected with
+  `The selected ACME account has no DNS provider configured for DNS-01.` The *Review* step also
+  named the default account rather than the chosen one, so the mismatch was invisible before
+  submitting.
+- **Default account:** the wizard previously previewed the **oldest** valid account while the
+  backend uses the **newest** (`ORDER BY created_at DESC`). If you never picked an account
+  explicitly and have several, requests were already going to the newest one — only the preview was
+  wrong. The wizard now previews that same account, marks it `(default)`, and sends `account_id`
+  explicitly so the two can no longer diverge.
+- **Wildcard guard:** the client-side "wildcard requires a DNS-01 account" block silently stopped
+  applying on the *Review* step. Requests were still rejected by the backend, so nothing incorrect
+  was ever issued — you now get the warning before submitting instead of an error after.
+- **No action needed on existing certificates or orders.** Nothing about issuance, renewal or the
+  stored accounts changes; only how the wizard resolves which account a new request uses.
+- **Rollback:** downgrade freely. This release changes frontend behaviour only.
+
+---
+
+# Upgrade Notes — v1.10.2 (Dark mode fixes on Apply Management)
+
+**Frontend only. Nothing to do on upgrade.** No schema, no `SCHEMA_VERSION` bump, no API change,
+no environment variable, zero agent impact. Light mode is byte-identical: every colour swapped in
+this release resolves, under the default algorithm, to exactly the literal it replaced
+(`colorWarningBg` → `#fffbe6`, `colorSuccessBg` → `#f6ffed`, `colorErrorBg` → `#fff2f0`, …), so
+only dark mode changes.
+
+- **Apply Management panels** were painted with light-mode colour literals, so in dark mode the
+  "Pending Changes" box rendered as a cream panel with light text on it. Measured contrast was
+  **1.03:1** — effectively invisible. It is now **11.50:1**. The same class of bug affected the
+  diff rows in *View Change* (2.21:1 and 2.99:1, now 5.49:1 and 4.01:1), the ACME/pending version
+  panels, the VIP pending-delete row and the agent-error recommendation box.
+- **Static confirm dialogs came up white in dark mode.** In Ant Design 5 the static
+  `Modal.confirm` / `message` / `notification` APIs render into their own detached root and never
+  see the app's `ConfigProvider`, so they always used the light algorithm. This release registers
+  `ConfigProvider.config({ holderRender })` once at the app root, which fixes **every** static
+  dialog in the app (12 components use them), not just Apply Management.
+- **Rollback:** downgrade freely. This release changes rendering only.
+
+---
+
+# Upgrade Notes — v1.10.1 (CSR private key encrypted at rest)
+
+**Backward compatible.** Nothing to do on upgrade, and nothing changes for existing clusters,
+agents or certificates:
+
+- **Schema:** **no `SCHEMA_VERSION` bump and no migration.** The Fernet token replaces the PEM
+  inside the *existing* `ssl_csrs.private_key_pem` TEXT column. As in v1.10.0, this means the
+  four built-in roles are **not** re-seeded, so any customization of `super_admin` / `operator` /
+  `security_admin` / `viewer` survives.
+- **Existing pending CSRs keep working.** Rows written before this release hold a raw PEM and are
+  still read transparently, so a CSR that is already out for signature can be imported normally
+  after the upgrade. There is no data migration and no downtime step. Those rows stay plaintext
+  until they are imported (which NULLs the key) — if you want everything encrypted immediately,
+  delete and re-create any long-pending CSRs.
+- **Scope:** this covers the PENDING CSR key only. It is the one key in the system that sits idle
+  for the whole signing window and is never transmitted. `ssl_certificates.private_key_content`
+  and the ACME order keys are unchanged, because agents must receive those in plaintext on every
+  poll.
+- **Optional env:** `CSR_ENCRYPTION_KEY` (see `.env.template`). If unset, the key is derived from
+  `SECRET_KEY` via HKDF with its own info string, so it is independent of the VIP, MFA and DNS
+  provider keys.
+  - **⚠️ Rotating `SECRET_KEY` while `CSR_ENCRYPTION_KEY` is unset makes pending CSR keys
+    unrecoverable.** Import then fails with an explicit "delete this CSR and create a new one"
+    error rather than a misleading key-mismatch. Set an explicit `CSR_ENCRYPTION_KEY` if you
+    rotate `SECRET_KEY`. Certificates already imported are unaffected — their key lives on the
+    certificate row.
+- **API / UI / agents:** unchanged. No CSR endpoint ever returned the private key before or now,
+  and nothing about the CSR tab changes.
+- **Rollback:** the application downgrades cleanly — 1.10.0 starts normally against the same
+  database and every other feature is unaffected. The one casualty is a CSR **created on 1.10.1
+  and still pending**: 1.10.0 has no decrypt step, so it hands the Fernet token straight to the
+  key-pairing check. Measured on a real downgrade, the import then fails with
+  `HTTP 500 — Could not verify the certificate/key pair: key parse failed (encrypted?)`; it does
+  **not** silently pair the wrong key, and it does not corrupt anything. Import or delete CSRs
+  created on 1.10.1 before downgrading. Certificates already imported are unaffected, since their
+  key lives on the certificate row, and CSRs created before 1.10.1 are plaintext and still work.
+
+---
+
+# Upgrade Notes — v1.10.0 (GoDaddy DNS-01 provider)
+
+**Backward compatible & additive.** Nothing changes unless you select **GoDaddy** as an ACME
+account's DNS provider:
+
+- **Schema:** **no `SCHEMA_VERSION` bump.** The GoDaddy credentials (API Key + Secret) are stored
+  as two keys inside the *existing* encrypted
+  `letsencrypt_account_dns_credentials.credentials_encrypted` blob — no new table, no new column,
+  no migration.
+- **✅ Built-in roles are NOT re-seeded.** The re-seed warning in the v1.9.0 notes below is
+  triggered by a `SCHEMA_VERSION` bump. This release does not bump it, so any customization you
+  made to `super_admin` / `operator` / `security_admin` / `viewer` survives untouched.
+- **Permissions / API shape:** unchanged. `GET /api/letsencrypt/dns-providers` simply returns one
+  extra entry in its `providers` array; every request and response shape is identical, and the
+  credential form is rendered from that schema, so there is no frontend behaviour change either.
+- **Environment:** no new variable. GoDaddy credentials use the same Fernet-at-rest path as
+  Cloudflare (`DNS_PROVIDER_ENCRYPTION_KEY`, falling back to a key derived from `SECRET_KEY`).
+- **Agents:** zero agent changes. DNS-01 is invisible to agents; an issued certificate follows the
+  normal PENDING → Apply Management → agent pull pipeline exactly as before.
+- **Using it:** the API Key must be a **Production** key from `developer.godaddy.com/keys` (the
+  first key that dashboard issues is an OTE/test key and is rejected), the zone must be in the same
+  GoDaddy account, and that account needs at least one registered domain before GoDaddy permits DNS
+  API access. A Personal Access Token also works — paste it as the Key and leave the Secret blank.
+  Credentials are checked against the GoDaddy API before they are stored, so an invalid, OTE or
+  ineligible key fails at save time. Note the check is a **read**: a Personal Access Token that has
+  `domains.domain:read` but not `domains.dns:update` saves successfully and only fails at the first
+  publish, with a 403 in the order timeline.
+- **Rollback:** don't select GoDaddy. Existing Manual and Cloudflare accounts and all HTTP-01
+  issuance are untouched. **Downgrading after adopting GoDaddy is not a no-op**: on 1.9.0
+  `godaddy` is not a known provider, so any account still set to it degrades to the manual-confirm
+  path (in-flight DNS-01 orders wait for a confirmation nobody can give and expire after 48h, and
+  renewals stop), and the cleanup sweep marks published TXT records cleaned without removing them.
+  Before downgrading, switch affected accounts back to Manual or Cloudflare and let the reconcile
+  sweep remove outstanding `_acme-challenge` records first. The stored credential row itself is
+  inert — an encrypted blob for an unknown provider.
+
+---
+
+# Upgrade Notes — v1.9.0 (CSR creation)
+
+**Backward compatible & additive.** Upgrading to v1.9.0 changes nothing for existing
+clusters/agents until you create a CSR:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `10`, so on first start the (idempotent)
+  migration sequence re-runs once and adds **one new table** (`ssl_csrs`) plus its
+  indexes. **No existing table is altered**, existing rows are untouched, and the
+  **admin password is not reset** (the default-user seeding is guarded by an
+  existence check, not an upsert). No new permission strings are introduced — all
+  CSR endpoints are governed by the existing `ssl.create` / `ssl.read` /
+  `ssl.delete` permissions.
+- **⚠️ Built-in roles are re-seeded to their defaults (pre-existing behaviour of
+  every `SCHEMA_VERSION` bump — verified in a v1.8.10 → v1.9.0 upgrade drill).**
+  Because the version gate re-runs the whole sequence, `update_system_roles_to_enterprise_rbac()`
+  issues an unconditional `UPDATE roles SET … permissions = <defaults> WHERE name = …`
+  for the four **built-in** roles (`super_admin`, `operator`, `security_admin`,
+  `viewer`). **Any customization you made to a built-in role is reverted.** In the
+  drill, an `operator` role that had been narrowed by removing `apply.execute` and
+  `config.bulk_import` came back with both restored (57 → 59 permissions).
+  - **Roles you created yourself are NOT affected** — the re-seed matches on the four
+    built-in names only.
+  - This is not new in v1.9.0: it happens on every release that bumps
+    `SCHEMA_VERSION` (v1.7.0, v1.8.0, v1.8.8 …). It is documented as intentional at
+    `backend/database/migrations.py` (the "BUMP THIS … OR seeded/role data" note) —
+    the migration is treated as the authority on built-in-role contents.
+  - **If you have hardened a built-in role, do this:** export it before upgrading
+    (`GET /api/roles`), then re-apply your changes after the first start
+    (`PUT /api/roles/{id}`) — or, preferably, move your customization into a
+    purpose-made custom role, which survives every upgrade.
+- **Key storage:** CSR private keys are stored in the database like every other key
+  in the system (`ssl_certificates.private_key_content` and the ACME order keys).
+  The key is never returned by any CSR API endpoint, and after a successful import
+  the CSR row's key copy is set to NULL (the key then lives only on the certificate
+  row).
+- **Agents:** zero agent changes. Agents never read the new table; a CSR becomes
+  visible to agents only after its signed certificate is imported **and** applied via
+  Apply Management (the standard PENDING pipeline).
+- **Rollback:** simply don't use the CSR tab. The `ssl_csrs` table is inert when
+  empty; downgrading the application leaves it as an ignored extra table.
+
+---
+
+# Upgrade Notes — v1.7.0 (HA / VIP Keepalived management, Issue #27)
+
+**Backward compatible & opt-in.** Upgrading to v1.7.0 changes nothing for existing
+clusters/agents until you create a VIP:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `3`, so on first start the (idempotent)
+  migration sequence re-runs once and adds two **new** tables (`vip_instances`,
+  `vip_members`) plus an additive `vip_instances.applied_snapshot` column (enables
+  rejecting a pending VIP change and restoring the previous applied state). No existing
+  table is altered. Existing rows and the **admin password
+  are not reset** (default users are create-if-missing). The only data effect is that
+  the **four built-in system roles** (`super_admin`/`operator`/`security_admin`/`viewer`)
+  are re-seeded to their canonical permission sets **plus** the new `vip.*` permissions —
+  this is the long-standing behavior of the role seeder; **custom roles are untouched**.
+- **Agents:** the agent script gains an opt-in keepalived deploy that is a **no-op** on
+  any node without an applied VIP, and it **never overwrites a hand-managed
+  `/etc/keepalived/keepalived.conf`** (it reports "externally managed" instead).
+- **Scope:** VRRP VIPs target bare-metal / VMware / on-prem L2 networks. On AWS/Azure/GCP
+  the cloud fabric doesn't honor VRRP/gratuitous-ARP; the UI surfaces this. Ensure host
+  firewalls permit VRRP (IP protocol 112).
+- **Optional env:** `VIP_ENCRYPTION_KEY` (see `.env.template`) — if unset, the VRRP secret
+  encryption key is derived from `SECRET_KEY` (like MFA).
+
+No rollback steps are required to *disable* the feature: simply don't create VIPs (or
+delete them — agents tear down their managed keepalived on the next poll).
+
+---
+
 # Agent Upgrade Guide - Dashboard Stats Fix
 
 ## Problem

backend/Dockerfile

@@ -37,5 +37,13 @@ USER appuser
 # Expose port
 EXPOSE 8000
 
-# Run the application in production mode (without --reload)
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+# Run the application in production mode (without --reload).
+# UVICORN_WORKERS (default 1) opts into multiple worker processes on multi-core
+# hosts; with 1 worker uvicorn runs in-process, identical to the flagless CMD
+# this replaces. Falls back to WEB_CONCURRENCY when UVICORN_WORKERS is unset
+# because flagless uvicorn honored WEB_CONCURRENCY (uvicorn config.py) — this
+# keeps any deployment that relied on it byte-for-byte compatible. Background
+# tasks are multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as
+# already exercised by the k8s HPA deployment. `exec` keeps uvicorn as PID 1
+# so signal handling is unchanged.
+CMD ["sh", "-c", "exec uvicorn main:app --host 0.0.0.0 --port 8000 --workers ${UVICORN_WORKERS:-${WEB_CONCURRENCY:-1}}"]

backend/auth_middleware.py

@@ -1,4 +1,4 @@
-from fastapi import HTTPException, status
+from fastapi import HTTPException, status, Header
 from typing import Optional, Dict, Any
 from jose import jwt
 import logging
@@ -92,6 +92,18 @@ async def get_current_user_from_token(authorization: Optional[str] = None) -> Op
             detail="Authentication failed"
         )
 
+async def require_authenticated_user(authorization: Optional[str] = Header(None)) -> Dict[str, Any]:
+    """FastAPI dependency: require a valid operator JWT, else 401.
+
+    Reads the Authorization header itself, so it can be attached at router or
+    route level to gate operator/UI endpoints that must not be public:
+        APIRouter(..., dependencies=[Depends(require_authenticated_user)])
+        @router.get(..., dependencies=[Depends(require_authenticated_user)])
+    Any authenticated user passes (no fine-grained RBAC here) — this restores the
+    pre-existing "logged-in users only" expectation without changing role access.
+    """
+    return await get_current_user_from_token(authorization)
+
 async def get_current_user_from_token_no_exception(authorization: Optional[str] = None) -> Optional[Dict[str, Any]]:
     """
     Get current user from JWT token without raising HTTPException.

backend/database/migrations.py

@@ -50,9 +50,62 @@ async def ensure_agents_table():
         await conn.execute("ALTER TYPE config_status ADD VALUE IF NOT EXISTS 'DELETION';")
         logger.info("Ensured REJECTED and DELETION values exist in config_status enum.")
 
-        # First, create essential tables if they don't exist
-        await create_essential_tables(conn)
-        
+        # First, create essential tables if they don't exist.
+        #
+        # Rolling-restart resilience: create_essential_tables() runs idempotent
+        # CREATE ... IF NOT EXISTS statements on every startup. Its CREATE INDEX
+        # block needs a SHARE lock that conflicts with concurrent writes (e.g.
+        # agent heartbeats updating backend_servers/agents). During a redeploy a
+        # writer can hold that lock, so the DDL blocked for the full 60s
+        # command_timeout -> TimeoutError -> startup crash -> crash-loop.
+        #
+        # Fix: fail fast on locks (lock_timeout), retry briefly, and on
+        # persistent contention SKIP the idempotent bootstrap and continue — on
+        # an established DB the objects already exist; a fresh DB has no writers
+        # so the first attempt always succeeds. lock_timeout is scoped to this
+        # call and RESET afterwards, so every other migration below keeps its
+        # original (wait-indefinitely) behavior. Non-lock errors still propagate
+        # (genuine schema problems must NOT be masked).
+        import asyncio as _asyncio
+        _lock_excs = (_asyncio.TimeoutError,)
+        try:
+            import asyncpg as _asyncpg
+            _lock_excs = _lock_excs + (
+                _asyncpg.exceptions.LockNotAvailableError,
+                _asyncpg.exceptions.QueryCanceledError,
+            )
+        except Exception:
+            pass
+        try:
+            await conn.execute("SET lock_timeout = '10s'")
+        except Exception:
+            pass
+        try:
+            for _attempt in range(1, 4):
+                try:
+                    await create_essential_tables(conn)
+                    break
+                except _lock_excs as _lock_err:
+                    if _attempt < 3:
+                        logger.warning(
+                            f"create_essential_tables: lock contention "
+                            f"(attempt {_attempt}/3), retrying in 3s "
+                            f"({type(_lock_err).__name__})"
+                        )
+                        await _asyncio.sleep(3)
+                    else:
+                        logger.warning(
+                            "create_essential_tables: persistent lock contention; "
+                            "skipping idempotent schema bootstrap and continuing "
+                            "startup (objects already exist on an established DB) "
+                            f"({type(_lock_err).__name__})"
+                        )
+        finally:
+            try:
+                await conn.execute("RESET lock_timeout")
+            except Exception:
+                pass
+
         # Ensure status column exists in config_versions table
         status_column_exists = await conn.fetchval("""
             SELECT 1 FROM information_schema.columns 
@@ -141,7 +194,14 @@ async def ensure_agents_table():
             'use_backend_rules': "ALTER TABLE frontends ADD COLUMN use_backend_rules JSONB DEFAULT '[]'::jsonb;",
             'request_headers': "ALTER TABLE frontends ADD COLUMN request_headers TEXT;",
             'response_headers': "ALTER TABLE frontends ADD COLUMN response_headers TEXT;",
-            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;"
+            'maxconn': "ALTER TABLE frontends ADD COLUMN maxconn INTEGER;",
+            # Issue #38: SPOE filter directives (e.g. Coraza WAF) and frontend
+            # log-format were silently dropped on bulk-import / manual edit
+            # because the parser recognised only a fixed set of directives.
+            # These nullable TEXT columns persist them verbatim (multi-line for
+            # `filters`), mirroring the request_headers/options passthrough.
+            'log_format': "ALTER TABLE frontends ADD COLUMN log_format TEXT;",
+            'filters': "ALTER TABLE frontends ADD COLUMN filters TEXT;"
         }
         
         for col, query in frontend_columns.items():
@@ -431,6 +491,7 @@ async def ensure_agents_table():
                 connection_type VARCHAR(50) DEFAULT 'agent',
                 stats_socket_path VARCHAR(500) DEFAULT '/run/haproxy/admin.sock',
                 haproxy_config_path VARCHAR(500) DEFAULT '/etc/haproxy/haproxy.cfg',
+                keepalived_config_path VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf',
                 pool_id INTEGER REFERENCES haproxy_cluster_pools(id) ON DELETE SET NULL,
                 haproxy_user VARCHAR(255) DEFAULT 'haproxy',
                 haproxy_group VARCHAR(255) DEFAULT 'haproxy',
@@ -1298,6 +1359,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk', 'apply.emergency',
                     'agents.read', 'agents.create', 'agents.update', 'agents.delete', 'agents.script', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.create', 'clusters.update', 'clusters.delete', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.upload', 'config.backup', 'config.restore', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'users.read', 'users.create', 'users.update', 'users.delete', 'users.password', 'users.roles',
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
@@ -1319,6 +1381,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk',
                     'agents.read', 'agents.update', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read'
@@ -1336,6 +1399,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history',
                     'agents.read', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
@@ -1354,6 +1418,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.history',
                     'agents.read',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read',
@@ -1639,13 +1704,142 @@ async def ensure_agent_activity_logs_table():
             await close_database_connection(conn)
         # Don't raise - this is not critical for system operation
 
+# Schema-version gate for the migration runner.
+#
+# >>> BUMP THIS whenever you add/modify ANY step in _run_all_migrations_inner()
+# >>> that changes the schema (table/column/index/constraint) OR seeded/role data
+# >>> (e.g. update_system_roles_to_enterprise_rbac). Otherwise the new step will
+# >>> NOT run on databases already marked at the current version.
+#
+# When the DB already records >= this version, run_all_migrations() skips the
+# whole (lock-heavy) idempotent sequence, so redeploys/scale-ups issue NO DDL and
+# a concurrently-serving replica's traffic cannot block ALTER / CREATE INDEX (the
+# rolling-deploy startup crash that motivated this gate).
+#
+# Backward compatibility (the product runs at many versions across companies):
+#  - First start on this code: no marker -> applied_version is NULL -> the FULL
+#    sequence runs (upgrades any prior version), THEN the marker is written. So
+#    upgrading from any older version is unaffected.
+#  - The marker is written ONLY after _run_all_migrations_inner() completes with
+#    no exception, so an interrupted/failed migration never marks an incomplete
+#    schema as done — the next start retries.
+#  - Behavior change vs the historical "re-run every idempotent ensure_* on every
+#    start": once marked, same-version restarts no longer re-run (and therefore no
+#    longer auto-repair manual drift). To force a re-run, bump SCHEMA_VERSION or
+#    delete the schema_migrations row.
+#
+# v1.7.0 (Issue #27 — HA/VIP Keepalived management): bumped 1 -> 2 so the new
+# additive ensure_vip_tables() step (two brand-new tables) actually runs on
+# databases already marked at version 1. The whole re-run is idempotent.
+# v1.7.0 self-review: bumped 2 -> 3 so the additive `applied_snapshot` column on
+# vip_instances (enables VIP reject/restore-to-previous) lands on DBs marked at 2.
+# v1.7.0 self-review: bumped 3 -> 4 for the additive `keepalived_config_path` column on
+# haproxy_clusters (cluster-driven keepalived.conf path, like haproxy_config_path).
+# v1.7.0 self-review: bumped 4 -> 5 to drop the table-level UNIQUE on vip_instances.name
+# and replace it with a partial unique index (active rows only), so a soft-deleted VIP's
+# name is reusable — consistent with the address/VRID partial indexes. Idempotent re-run.
+# v1.7.2: bumped 5 -> 6 for the additive `purge_on_teardown` column on vip_instances
+# (opt-in "also uninstall the keepalived package on delete"; default FALSE keeps the safe
+# graceful-teardown behaviour). Additive + idempotent.
+# v1.7.2: bumped 6 -> 7 for the additive `pending_delete` column on vip_instances
+# (approval-gated VIP deletion: a delete is staged for Apply Management and the VIP keeps
+# running until APPROVED, so an agent never tears down without explicit human approval).
+# v1.8.0 (Issue #35 — ACME DNS-01 challenge support): bumped 7 -> 8 for additive DNS-01
+# columns on letsencrypt_accounts/letsencrypt_orders/acme_challenges and the brand-new
+# letsencrypt_account_dns_credentials table (ensure_letsencrypt_dns_credentials step).
+# All additive + idempotent; default challenge_type 'http-01' keeps existing flows byte-identical.
+# v1.8.8 (Issue #38 — SPOE filter + frontend log-format): bumped 8 -> 9 for the additive
+# `log_format` + `filters` TEXT columns on `frontends` (frontend_columns loop). Without this
+# bump, already-deployed databases (version >= 8) skip the whole migration run and never gain
+# the columns, so the frontends SELECT/INSERT would fail. Additive + idempotent + nullable;
+# existing rows stay NULL and render byte-identical.
+# v1.9.0 (CSR creation): bumped 9 -> 10 for the brand-new `ssl_csrs` table
+# (ensure_ssl_csrs_table step). Holds a locally generated private key + CSR PEM
+# until the operator imports the CA-signed certificate; the import creates a
+# normal ssl_certificates row and NULLs the key copy here. Additive + idempotent;
+# no existing table is altered, agents never read this table.
+# v1.10.4 (VIP adoption): bumped 10 -> 11 for the new `vip_discoveries` table plus two
+# additive columns (`vip_instances.adopted_at`, `vip_members.takeover_expected_hash`).
+# Holds the keepalived.conf an agent found already on a node so an existing VIP can be
+# adopted instead of retyped. Additive + idempotent; no existing table is altered and no
+# existing row changes. NOTE for the upgrade notes: a SCHEMA_VERSION bump re-seeds the four
+# built-in roles to their defaults, so role customizations are lost on this upgrade.
+SCHEMA_VERSION = 11
+
+
 async def run_all_migrations():
-    """Run all database migrations"""
+    """Run all database migrations.
+
+    Hardened for multiple backend replicas / rolling deploys:
+    - A session-level advisory lock serializes the run so only one pod migrates
+      at a time (others wait, then hit the version gate and skip). It is
+      session-scoped, so it auto-releases if a pod dies mid-migration.
+    - A schema-version marker (schema_migrations) gates the run: when the DB is
+      already at SCHEMA_VERSION the whole idempotent sequence is skipped, so no
+      DDL is issued and a serving replica's traffic can't block it.
+    If the advisory lock or marker can't be used, we fall back to running the
+    (idempotent) migrations rather than crashing startup.
+    """
     logger.info("Starting database migrations...")
-    
+    MIGRATION_ADVISORY_LOCK_KEY = 1836016242  # single-key advisory space ("migr"); distinct from the (ns,id) locks used elsewhere
+    lock_conn = None
+    lock_acquired = False
+    try:
+        lock_conn = await get_database_connection()
+        try:
+            await lock_conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            lock_acquired = True
+            logger.info("Acquired migration advisory lock (migrations serialized across pods)")
+        except Exception as _lock_e:
+            logger.warning(f"Could not acquire migration advisory lock; proceeding (migrations are idempotent): {_lock_e}")
+
+        # Schema-version gate: skip the lock-heavy sequence if the DB is current.
+        applied_version = None
+        try:
+            await lock_conn.execute("""
+                CREATE TABLE IF NOT EXISTS schema_migrations (
+                    id INTEGER PRIMARY KEY DEFAULT 1,
+                    version INTEGER NOT NULL,
+                    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                    CONSTRAINT schema_migrations_singleton CHECK (id = 1)
+                )
+            """)
+            applied_version = await lock_conn.fetchval("SELECT version FROM schema_migrations WHERE id = 1")
+        except Exception as _mk_e:
+            logger.warning(f"schema_migrations marker unavailable; running full migrations: {_mk_e}")
+            applied_version = None
+
+        if applied_version is not None and applied_version >= SCHEMA_VERSION:
+            logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
+            return
+
+        await _run_all_migrations_inner()
+
+        try:
+            await lock_conn.execute("""
+                INSERT INTO schema_migrations (id, version, applied_at)
+                VALUES (1, $1, CURRENT_TIMESTAMP)
+                ON CONFLICT (id) DO UPDATE SET version = EXCLUDED.version, applied_at = EXCLUDED.applied_at
+            """, SCHEMA_VERSION)
+            logger.info(f"Recorded schema version {SCHEMA_VERSION} in schema_migrations.")
+        except Exception as _wr_e:
+            logger.warning(f"Could not record schema version marker (migrations still applied): {_wr_e}")
+    finally:
+        if lock_acquired and lock_conn is not None:
+            try:
+                await lock_conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            except Exception:
+                pass
+        if lock_conn is not None:
+            await close_database_connection(lock_conn)
+
+
+async def _run_all_migrations_inner():
+    """The full idempotent migration sequence. Runs under the migration advisory
+    lock and is gated by the schema-version marker in run_all_migrations()."""
     # First, ensure basic database schema exists
     await run_init_sql()
-    
+
     # Then run additional migrations
     await ensure_agents_table()
     await ensure_config_versions_metadata_column()
@@ -1684,6 +1878,9 @@ async def run_all_migrations():
     await ensure_system_settings_table()
     await ensure_acme_tables()
     await ensure_acme_columns_on_existing_tables()
+    # Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+    # MUST run after ensure_acme_tables() (FK references letsencrypt_accounts).
+    await ensure_letsencrypt_dns_credentials()
     # Issue #11 cleanup: must run AFTER acme_tables/columns to ensure FK refs exist
     await cleanup_orphan_acme_challenge_backend()
     # v1.5.0 Feature A (ACME diagnostics) + Feature B (site wizard)
@@ -1700,8 +1897,323 @@ async def run_all_migrations():
     # (cluster_id, bind_address, bind_port) WHERE is_active.
     await ensure_frontends_bind_unique_constraint()
 
+    # Issue #18 — TOTP MFA (v1.6.0): additive columns + 3 new tables
+    await ensure_mfa_columns()
+
+    # Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
+    # MUST run after its FK targets (haproxy_cluster_pools/agents/users), all created above.
+    await ensure_vip_tables()
+
+    # v1.9.0 — CSR creation: brand-new ssl_csrs table. FK-references
+    # ssl_certificates/users, both created above.
+    await ensure_ssl_csrs_table()
+
     logger.info("Database migrations completed successfully.")
 
+
+async def ensure_ssl_csrs_table():
+    """v1.9.0 — CSR (Certificate Signing Request) creation. Additive only:
+    one brand-new table (ssl_csrs) + indexes. No ALTER of any existing table,
+    so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (ssl_certificates, users)
+    are created earlier in the sequence.
+
+    A CSR row holds a locally generated private key + CSR PEM until the
+    operator imports the CA-signed certificate. The import creates a normal
+    ssl_certificates row (source='csr', last_config_status='PENDING') and
+    NULLs the private_key_pem copy here — the key then lives only on the
+    certificate row, like every other key in the system. Agents never read
+    this table: the agent SSL delivery endpoint selects from
+    ssl_certificates only, so a pending CSR can never leak to an agent.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS ssl_csrs (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(100) NOT NULL,
+                common_name VARCHAR(253) NOT NULL,
+                subject JSONB NOT NULL DEFAULT '{}'::jsonb,
+                sans JSONB NOT NULL DEFAULT '[]'::jsonb,
+                key_algorithm VARCHAR(20) NOT NULL DEFAULT 'rsa-2048',
+                csr_pem TEXT NOT NULL,
+                private_key_pem TEXT,
+                status VARCHAR(20) NOT NULL DEFAULT 'pending',
+                ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL,
+                completed_at TIMESTAMP,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT ssl_csrs_status_check CHECK (status IN ('pending', 'completed'))
+            );
+        """)
+
+        # Only PENDING CSRs reserve their name: the name becomes the
+        # ssl_certificates.name (and thus /etc/ssl/haproxy/{name}.pem on every
+        # agent) at import time, so two open CSRs must not target the same
+        # cert name. Completed CSRs are history and may share a name across
+        # reissues — mirrors the uq_vip_name_active partial-index rationale.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_ssl_csrs_name_pending ON ssl_csrs(name) WHERE status = 'pending';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_status ON ssl_csrs(status);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_ssl_csrs_cert ON ssl_csrs(ssl_certificate_id);"
+        )
+
+        logger.info("ssl_csrs table ensured (v1.9.0 CSR creation)")
+    except Exception as e:
+        logger.error(f"Error ensuring ssl_csrs table: {e}")
+        # Re-raise (ensure_ssl_cluster_junction_table precedent): this step is
+        # part of the SCHEMA_VERSION=10 bump, and run_all_migrations() records
+        # the marker only after the inner sequence completes cleanly. Swallowing
+        # a failure here would stamp version 10 with no ssl_csrs table, and the
+        # version gate would then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_mfa_columns():
+    """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
+
+    All operations are idempotent (ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS).
+    Default behavior preserved: every existing user gets mfa_enabled=FALSE, so login
+    flow is byte-identical for accounts that don't opt in.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            ALTER TABLE users
+                ADD COLUMN IF NOT EXISTS mfa_enabled BOOLEAN DEFAULT FALSE NOT NULL,
+                ADD COLUMN IF NOT EXISTS mfa_method VARCHAR(20),
+                ADD COLUMN IF NOT EXISTS mfa_secret_encrypted TEXT,
+                ADD COLUMN IF NOT EXISTS mfa_enrolled_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_totp_step BIGINT;
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_backup_codes (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                code_hash VARCHAR(255) NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute("""
+            CREATE INDEX IF NOT EXISTS idx_mfa_backup_codes_user
+              ON mfa_backup_codes(user_id);
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_logins (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                challenge_token VARCHAR(64) UNIQUE NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                ip_address INET
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_token ON mfa_pending_logins(challenge_token);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_expires ON mfa_pending_logins(expires_at);"
+        )
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_enrollments (
+                user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+                secret_encrypted TEXT NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_enroll_expires ON mfa_pending_enrollments(expires_at);"
+        )
+
+        logger.info("✅ MFA migration completed (Issue #18 — Phase 1)")
+    except Exception as e:
+        logger.error(f"Failed to ensure MFA columns: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_user_activity_logs_table
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+async def ensure_vip_tables():
+    """Issue #27 — HA/VIP (Keepalived) management (v1.7.0). Additive only:
+    two brand-new tables (vip_instances, vip_members) + indexes. No ALTER of any
+    existing table, so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (haproxy_cluster_pools, agents,
+    users) are created earlier in the sequence — this function is registered LAST.
+
+    Backward-compat: a cluster/agent with no VIP row is unaffected; the agent
+    delivery endpoint returns 'not_configured' for every node without a membership.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Cluster-driven keepalived.conf path (mirrors haproxy_config_path): additive +
+        # idempotent, with a universal default so operators need set nothing. The agent
+        # pulls this from its cluster, exactly like the HAProxy paths.
+        await conn.execute(
+            "ALTER TABLE haproxy_clusters ADD COLUMN IF NOT EXISTS keepalived_config_path "
+            "VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf';")
+
+        # VIP instance: one row per virtual IP (one VRRP group), anchored to a pool.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_instances (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(255) NOT NULL,
+                description TEXT,
+                pool_id INTEGER NOT NULL REFERENCES haproxy_cluster_pools(id) ON DELETE CASCADE,
+                virtual_ip VARCHAR(45) NOT NULL,
+                prefix_length INTEGER NOT NULL DEFAULT 24,
+                virtual_router_id INTEGER NOT NULL,
+                advert_int INTEGER NOT NULL DEFAULT 1,
+                auth_pass_encrypted TEXT,
+                use_unicast BOOLEAN NOT NULL DEFAULT TRUE,
+                track_haproxy BOOLEAN NOT NULL DEFAULT TRUE,
+                is_active BOOLEAN NOT NULL DEFAULT TRUE,
+                last_config_status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
+                applied_snapshot JSONB,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_vrid_range CHECK (virtual_router_id BETWEEN 1 AND 255)
+            );
+        """)
+        # Additive (idempotent) for DBs that created vip_instances before applied_snapshot
+        # existed (v1.7.0 self-review): holds the field-level state as of the last Apply so
+        # a pending edit can be rejected and fully reverted to the previous applied state.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS applied_snapshot JSONB;")
+        # Opt-in package removal (v1.7.2): when an operator deletes a VIP and explicitly ticks
+        # "also uninstall keepalived from the node(s)", we set this flag so the teardown
+        # delivery tells the agent to purge the OS package. Default FALSE = the safe enterprise
+        # default (stop+disable+remove our config, but KEEP the package). Additive + idempotent;
+        # a node we never managed stays untouched regardless.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Approval-gated deletion (v1.7.2): deleting a RUNNING VIP from the UI does NOT take
+        # effect immediately — it sets pending_delete=TRUE and stages a vip-*-delete version
+        # for Apply Management. The VIP stays is_active=TRUE (agents keep serving it, NOTHING
+        # is torn down) until the operator APPROVES; only then does apply flip is_active=FALSE
+        # and the agents tear down. Reject clears the flag and the VIP keeps running untouched.
+        # This guarantees an agent never tears a VIP down without an explicit human approval —
+        # protecting production. Additive + idempotent.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS pending_delete BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Uniqueness as PARTIAL indexes on active rows so a soft-deleted VIP frees its
+        # name/address/VRID for immediate reuse (a table-level UNIQUE would keep blocking it).
+        # NAME (v1.7.0 self-review): the original CREATE used a table-level UNIQUE on name,
+        # which left a soft-deleted VIP's name blocked (you couldn't re-create a VIP with the
+        # same name) — inconsistent with addr/VRID. Drop that constraint and use a partial
+        # index instead. Idempotent: no-op on a fresh table (no inline UNIQUE) and on re-run.
+        await conn.execute("ALTER TABLE vip_instances DROP CONSTRAINT IF EXISTS vip_instances_name_key;")
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_name_active ON vip_instances(name) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_addr_active ON vip_instances(virtual_ip) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_vrid_active ON vip_instances(pool_id, virtual_router_id) WHERE is_active=TRUE;"
+        )
+
+        # Per-node membership: which agents participate + their VRRP role/priority,
+        # the applied (delivered) config snapshot, and the agent's deploy ack.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_members (
+                id SERIAL PRIMARY KEY,
+                vip_id INTEGER NOT NULL REFERENCES vip_instances(id) ON DELETE CASCADE,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                network_interface VARCHAR(64) NOT NULL,
+                role VARCHAR(10) NOT NULL DEFAULT 'BACKUP',
+                priority INTEGER NOT NULL DEFAULT 100,
+                applied_config_content TEXT,
+                applied_config_hash VARCHAR(64),
+                last_deploy_state VARCHAR(24),
+                last_deploy_message TEXT,
+                last_deploy_hash VARCHAR(64),
+                last_deploy_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_member_role CHECK (role IN ('MASTER','BACKUP')),
+                CONSTRAINT vip_member_priority_range CHECK (priority BETWEEN 1 AND 254),
+                CONSTRAINT vip_member_unique UNIQUE (vip_id, agent_id)
+            );
+        """)
+        # Last line of defense against split-brain: at most one MASTER per VIP.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_one_master ON vip_members(vip_id) WHERE role='MASTER';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_vip ON vip_members(vip_id);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_agent ON vip_members(agent_id);"
+        )
+
+        # ── v1.10.4 — VIP adoption: what the agent found already on the node ──────────
+        # A node with a hand-maintained keepalived.conf reports it here so an existing VIP can
+        # be adopted instead of retyped. One row per agent (the file is per-node); the agent
+        # only reports a config it does NOT own, and only when the content changed.
+        #
+        # SECRETS: `raw_config` is stored MASKED (auth_pass replaced) because it is served to
+        # the UI. The real VRRP password is Fernet-encrypted in auth_pass_encrypted, mirroring
+        # vip_instances, so adoption can carry it into the managed VIP without it ever being
+        # readable through the API or a DB dump. `analysis` is the parser output with auth_pass
+        # stripped out.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_discoveries (
+                id SERIAL PRIMARY KEY,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                config_path VARCHAR(500) NOT NULL,
+                config_hash VARCHAR(64) NOT NULL,
+                is_managed BOOLEAN NOT NULL DEFAULT FALSE,
+                raw_config_masked TEXT,
+                auth_pass_encrypted TEXT,
+                analysis JSONB,
+                parse_error TEXT,
+                adopted_vip_id INTEGER REFERENCES vip_instances(id) ON DELETE SET NULL,
+                reported_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_discovery_agent_unique UNIQUE (agent_id)
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_discoveries_agent ON vip_discoveries(agent_id);"
+        )
+        # Adoption provenance + the one-shot takeover authorisation. The agent refuses to
+        # overwrite a keepalived.conf that lacks our ownership marker, which is exactly the
+        # guard adoption has to pass. Rather than weaken it, an adopted VIP carries the hash of
+        # the file we analysed: the agent takes over ONLY if the file on disk still hashes to
+        # that value, so a config that changed after adoption is never clobbered.
+        await conn.execute(
+            "ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS adopted_at TIMESTAMP;")
+        await conn.execute(
+            "ALTER TABLE vip_members ADD COLUMN IF NOT EXISTS takeover_expected_hash VARCHAR(64);")
+
+        logger.info("✅ VIP tables ensured (Issue #27 — HA/VIP Keepalived management)")
+    except Exception as e:
+        logger.error(f"Failed to ensure VIP tables: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_mfa_columns
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 async def add_ssl_certificate_id_to_backend_servers():
     """Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
     conn = None
@@ -3278,6 +3790,23 @@ async def ensure_acme_columns_on_existing_tables():
             ('last_attempt_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS last_attempt_at TIMESTAMPTZ"),
             # Commit 3a: track auto-completion task lock/poll timestamps for atomic claim across replicas
             ('orders_updated_at_idx', "CREATE INDEX IF NOT EXISTS idx_letsencrypt_orders_status_updated ON letsencrypt_orders(status, updated_at) WHERE status = 'valid' AND ssl_certificate_id IS NULL"),
+            # Issue #35 (v1.8.0 — ACME DNS-01): per-account challenge method + DNS provider selection
+            ('acct_challenge_type', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('acct_dns_provider', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS dns_provider VARCHAR(50)"),
+            # per-order challenge method + bounded DNS-01 retry chain (dns01_parent_order_id is a PLAIN INTEGER, not a FK,
+            # to avoid a self-referential cascade interacting with account/order bulk DELETEs)
+            ('order_challenge_type', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('order_dns01_attempts', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_attempts INTEGER DEFAULT 0"),
+            ('order_dns01_last_attempt_at', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_last_attempt_at TIMESTAMPTZ"),
+            ('order_dns01_parent_order_id', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_parent_order_id INTEGER"),
+            ('order_dns01_retry_claimed', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_retry_claimed BOOLEAN DEFAULT FALSE"),
+            # per-challenge DNS-01 lifecycle state
+            ('chal_challenge_type', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('chal_dns_txt_value', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_txt_value TEXT"),
+            ('chal_dns_record_published', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_published BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_record_cleaned', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_cleaned BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_published_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_published_at TIMESTAMPTZ"),
+            ('chal_manual_confirm_deadline', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS manual_confirm_deadline TIMESTAMPTZ"),
         ]:
             try:
                 await conn.execute(sql)
@@ -3293,6 +3822,34 @@ async def ensure_acme_columns_on_existing_tables():
         logger.error(f"Error adding ACME columns: {e}")
 
 
+async def ensure_letsencrypt_dns_credentials():
+    """Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+
+    Idempotent (CREATE TABLE IF NOT EXISTS). FK to letsencrypt_accounts (created earlier by
+    ensure_acme_tables). Credentials are Fernet-encrypted at rest (backend/utils/dns_credentials.py);
+    only the provider name + timestamps are ever surfaced to the API.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS letsencrypt_account_dns_credentials (
+                id SERIAL PRIMARY KEY,
+                account_id INTEGER NOT NULL UNIQUE REFERENCES letsencrypt_accounts(id) ON DELETE CASCADE,
+                dns_provider VARCHAR(50) NOT NULL,
+                credentials_encrypted TEXT NOT NULL,
+                created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
+            )
+        """)
+        logger.info("Ensured letsencrypt_account_dns_credentials table")
+        await close_database_connection(conn)
+    except Exception as e:
+        if conn:
+            await close_database_connection(conn)
+        logger.error(f"Error ensuring letsencrypt_account_dns_credentials: {e}")
+
+
 async def cleanup_orphan_acme_challenge_backend():
     """
     Issue #11: One-time cleanup of orphan `_acme_challenge_backend` rows that may

backend/main.py

@@ -8,8 +8,13 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.5.0", "releaseName": "ACME Diagnostics & Site Wizard", "releaseDate": "2026-05-08"}
-for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
+# Single source of truth: backend/version.json, which sits next to this module and is baked into
+# every image by `COPY . .` (build context ./backend) — no pipeline staging needed. The literal
+# below is only a last-resort "file missing" marker; it is deliberately NOT a real version so it can
+# never silently drift out of sync (this exact drift showed a stale version after v1.8.5/v1.8.6).
+# Keep the canonical version ONLY in backend/version.json — test_version_consistency.py enforces it.
+_version_info = {"version": "unknown", "releaseName": "unknown", "releaseDate": ""}
+for _vpath in [os.path.join(os.path.dirname(__file__), "version.json"), "/app/version.json"]:
     try:
         with open(_vpath) as _vf:
             _version_info = json.load(_vf)
@@ -40,6 +45,9 @@ from routers.settings import router as settings_router
 from routers.letsencrypt import router as letsencrypt_router
 from routers.acme_diagnostics import router as acme_diagnostics_router
 from routers.site_wizard import router as site_wizard_router
+from routers.mfa import router as mfa_router
+from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
+from routers.csr import router as csr_router  # v1.9.0 — CSR creation (in-app key+CSR generation, signed-cert import)
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -302,6 +310,21 @@ async def complete_pending_acme_orders():
                         WHERE (
                             status IN ('pending', 'processing', 'ready')
                             OR (status = 'valid' AND ssl_certificate_id IS NULL)
+                            -- Issue #35: bounded DNS-01 retry. ONLY dns-01 invalids with remaining
+                            -- budget + elapsed backoff are claimed; http-01 invalids are NEVER matched
+                            -- (their existing skip-and-log is preserved).
+                            OR (
+                                status = 'invalid' AND challenge_type = 'dns-01'
+                                AND ssl_certificate_id IS NULL
+                                AND COALESCE(dns01_retry_claimed, FALSE) = FALSE
+                                AND COALESCE(dns01_attempts, 0) < 3
+                                AND (
+                                    dns01_last_attempt_at IS NULL
+                                    OR dns01_last_attempt_at < NOW() - (
+                                        (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
+                                        || ' minutes')::INTERVAL
+                                )
+                            )
                         )
                         AND created_at > NOW() - INTERVAL '7 days'
                         AND (updated_at IS NULL OR updated_at < NOW() - INTERVAL '30 seconds')
@@ -335,9 +358,17 @@ async def complete_pending_acme_orders():
                 continue
 
             logger.info(f"[ACME-COMPLETE] Claimed {len(claimed_ids)} order(s) for completion: {claimed_ids}")
-            
+
+            from services.dns01_orchestrator import (
+                advance_dns01_order, retry_invalid_dns01, reconcile_dns01_cleanup,
+            )
+
             for oid in claimed_ids:
                 try:
+                    # Issue #35: advance the DNS-01 publish->confirm->respond state machine for
+                    # pending dns-01 orders (no-op for http-01 or non-pending orders).
+                    await advance_dns01_order(oid)
+
                     status_info = await acme_svc.check_order_status(oid)
                     current_status = status_info.get('status')
 
@@ -350,12 +381,21 @@ async def complete_pending_acme_orders():
                         result = await _complete_certificate(oid)
                         logger.info(f"[ACME-COMPLETE] Order {oid} completed - {result.get('message', '')}")
                     elif current_status == 'invalid':
-                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid, skipping")
+                        # Issue #35: bounded DNS-01 fresh-order retry (no-op for http-01).
+                        await retry_invalid_dns01(oid)
+                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid")
                     elif current_status in ('pending', 'processing'):
                         logger.info(f"[ACME-COMPLETE] Order {oid} still {current_status}, will retry next cycle")
                 except Exception as poll_err:
                     logger.error(f"[ACME-COMPLETE] Failed to complete order {oid}: {poll_err}")
 
+            # Issue #35: best-effort cleanup of TXT records left published on terminal orders
+            # (covers a failed cleanup or the kill-switch being flipped off). NOT gated by the switch.
+            try:
+                await reconcile_dns01_cleanup()
+            except Exception as rec_err:
+                logger.debug(f"[ACME-COMPLETE] DNS-01 reconcile skipped: {rec_err}")
+
             # NOTE: v1.5.0 wizard-staged processing now runs BEFORE the
             # claimed_ids early-continue above (Bulgu #2 fix), so it executes
             # every cycle regardless of pending/processing volume.
@@ -671,7 +711,9 @@ async def check_letsencrypt_renewals():
                     skip = False
                     try:
                         order = await conn2.fetchrow(
-                            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1",
+                            "SELECT o.account_id, o.domains, o.cluster_ids, o.challenge_type, a.dns_provider "
+                            "FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id "
+                            "WHERE o.id = $1",
                             order_id
                         )
                         if order:
@@ -687,15 +729,37 @@ async def check_letsencrypt_renewals():
                             if existing:
                                 logger.info(f"[ACME-RENEWAL] Skipping cert {cert['id']} - order {existing['id']} already in progress")
                                 skip = True
+                            elif (order['challenge_type'] == 'dns-01'):
+                                # Issue #35: manual DNS-01 cannot auto-renew unattended; and for an
+                                # automated provider, don't re-mint hourly if a recent retry chain already
+                                # exhausted its budget (avoids tripping the CA new-order rate limit).
+                                if (order['dns_provider'] or 'manual') == 'manual':
+                                    logger.warning(f"[ACME-RENEWAL] cert {cert['id']} uses manual DNS-01; cannot auto-renew unattended (publish the TXT and renew manually)")
+                                    skip = True
+                                else:
+                                    exhausted = await conn2.fetchrow("""
+                                        SELECT id FROM letsencrypt_orders
+                                        WHERE domains::text = $1::text AND challenge_type = 'dns-01'
+                                        AND status = 'invalid' AND COALESCE(dns01_attempts, 0) >= 3
+                                        AND created_at > NOW() - INTERVAL '24 hours'
+                                        LIMIT 1
+                                    """, json.dumps(domains))
+                                    if exhausted:
+                                        logger.warning(f"[ACME-RENEWAL] cert {cert['id']} DNS-01 renewal recently failed (check DNS); skipping re-mint for 24h")
+                                        skip = True
                     finally:
                         await close_database_connection(conn2)
 
                     if not order or skip:
                         continue
 
-                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids)
-                    await acme_svc.respond_to_challenges(new_order['order_id'])
-                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} for cert {cert['id']} ({cert['name']})")
+                    challenge_type = order['challenge_type'] or 'http-01'
+                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids, challenge_type=challenge_type)
+                    # http-01 responds immediately (token served continuously); dns-01 is driven by the
+                    # orchestrator AFTER the TXT is published (never respond before publish).
+                    if challenge_type != 'dns-01':
+                        await acme_svc.respond_to_challenges(new_order['order_id'])
+                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} ({challenge_type}) for cert {cert['id']} ({cert['name']})")
                 except Exception as cert_err:
                     logger.error(f"[ACME-RENEWAL] Failed to initiate renewal for cert {cert['id']}: {cert_err}")
 
@@ -820,6 +884,7 @@ app.include_router(config_router)  # Configuration management
 app.include_router(maintenance_router, prefix="/api", tags=["maintenance"])  # Database cleanup & maintenance
 app.include_router(auth_router)
 app.include_router(user_router)
+app.include_router(mfa_router)
 app.include_router(frontend_router)
 app.include_router(backend_router)
 app.include_router(cluster_router)
@@ -828,12 +893,14 @@ app.include_router(dashboard_stats_router)  # HAProxy stats dashboard
 app.include_router(agent_router)
 app.include_router(waf_router)
 app.include_router(ssl_router)
+app.include_router(csr_router)  # v1.9.0: CSR creation (in-app key+CSR generation, signed-cert import)
 app.include_router(security_router)
 app.include_router(configuration_router)
 app.include_router(settings_router)
 app.include_router(letsencrypt_router)
 app.include_router(acme_diagnostics_router)  # v1.5.0 Issue #13: ACME Diagnostic Panel
 app.include_router(site_wizard_router)  # v1.5.0 Issue #14: New Site Setup Wizard
+app.include_router(vip_router)  # v1.7.0 Issue #27: HA/VIP (Keepalived) management
 
 
 # Legacy URL alias: /api/proxied-hosts/* → 308 redirect to /api/sites/*.
@@ -1044,14 +1111,24 @@ async def get_version():
     return _version_info
 
 @app.get("/.well-known/acme-challenge/{token}")
-async def serve_acme_challenge(token: str):
+async def serve_acme_challenge(token: str, request: Request):
     """Serve ACME HTTP-01 challenge token. Public endpoint, no auth required."""
-    logger.info(f"ACME-CHALLENGE: Incoming request for token={token[:32]}...")
+    # Log who reached us. When HTTP-01 fails, the first question is always "did the
+    # request get here at all?" — and the answer separates a broken challenge-backend
+    # address (nothing arrives) from a wrong response (arrives, wrong body). The peer
+    # is normally the HAProxy node; X-Forwarded-For carries the CA when the frontend
+    # sets `option forwardfor`.
+    _peer = request.client.host if request.client else 'unknown'
+    _xff = request.headers.get('x-forwarded-for') or '-'
+    logger.info(
+        f"ACME-CHALLENGE: Incoming request for token={token[:32]}... "
+        f"peer={_peer} xff={_xff} host={request.headers.get('host') or '-'}"
+    )
     conn = None
     try:
         conn = await get_database_connection()
         row = await conn.fetchrow(
-            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) LIMIT 1",
+            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) AND (challenge_type = 'http-01' OR challenge_type IS NULL) LIMIT 1",
             token
         )
         if row:

backend/middleware/error_handler.py

@@ -126,10 +126,26 @@ class GlobalExceptionHandler:
             except Exception as body_error:
                 logger.debug(f"Could not extract raw body for debugging: {body_error}")
         
-        # Enhanced log message for agent heartbeats
-        log_message = f"Validation error: {str(exc)}"
+        # Build a sanitized log summary. The raw `str(exc)` from Pydantic
+        # contains the user-supplied `input` value for each failed field —
+        # which leaks secrets like TOTP codes, backup codes, mfa_token, and
+        # passwords to plaintext logs. Use only field NAMES + types here;
+        # `validation_details` (already sanitized to {field, message, type})
+        # is attached separately for downstream structured logging.
+        _field_names = [
+            err.get("field", "unknown")
+            for err in error_details.get("validation_errors", [])
+        ]
+        _err_count = len(error_details.get("validation_errors", []))
+        log_message = (
+            f"Validation error: {_err_count} field(s) failed validation: "
+            f"[{', '.join(_field_names)}]"
+        )
         if agent_name != "unknown":
-            log_message = f"Agent '{agent_name}' heartbeat validation error: {str(exc)}"
+            log_message = (
+                f"Agent '{agent_name}' heartbeat validation error: "
+                f"{_err_count} field(s) failed: [{', '.join(_field_names)}]"
+            )
         
         # Log validation error with enhanced details
         log_with_correlation(

backend/middleware/mfa_rate_limit_key.py

@@ -0,0 +1,147 @@
+"""MFA rate-limit key extraction (user-aware + ingress-aware).
+
+Problem with the default ``slowapi.util.get_remote_address``:
+
+  * Behind an ingress / reverse proxy / load balancer, every request looks
+    like it originates from the same upstream IP (the proxy itself). A
+    5/minute IP-bucket therefore becomes a 5/minute *whole-organization*
+    bucket. In a 500-user enterprise rolling out MFA, this stalls the
+    rollout to a trickle.
+
+Strategy (combined B + C from the design review):
+
+  1. **User-aware key (preferred):** if the request carries a valid Bearer
+     JWT, derive the bucket from ``user:<id>``. Each authenticated user
+     gets an isolated bucket, regardless of source IP. A malicious user
+     burning their own quota cannot starve the other 499.
+
+  2. **Trusted-proxy XFF fallback:** if the request is unauthenticated
+     (e.g. ``/login`` flow, future endpoints) and the TCP peer is in
+     ``MFA_TRUSTED_PROXY_CIDRS``, peel the first hop off ``X-Forwarded-For``.
+     This preserves real-IP buckets behind a known ingress without
+     accepting spoofed headers from the public internet.
+
+  3. **Default fallback:** plain ``request.client.host`` (slowapi default).
+
+JWT decode is intentionally signature-verified (replay/spoof protection)
+and *sync* — slowapi's decorator hook is sync, and our JWT library
+(python-jose) is sync as well. Failed verification silently downgrades
+to IP-based bucketing — never crashes the decorator.
+
+Env vars:
+  * ``MFA_TRUSTED_PROXY_CIDRS`` — comma-separated CIDR list of trusted
+    upstream proxies. Empty (default) disables XFF parsing entirely,
+    which is the safe choice when the deployment topology is unknown.
+    Examples:
+        ``MFA_TRUSTED_PROXY_CIDRS=10.0.0.0/8,172.16.0.0/12``
+        ``MFA_TRUSTED_PROXY_CIDRS=192.168.0.0/16``
+"""
+from __future__ import annotations
+
+import logging
+import os
+from ipaddress import ip_address, ip_network
+from typing import List
+
+from fastapi import Request
+from jose import jwt
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_trusted_cidrs() -> List:
+    """Parse the trusted-proxy CIDR list at import time.
+
+    A malformed entry is logged and skipped — we never crash the process
+    over a typo in operational config.
+    """
+    raw = os.getenv("MFA_TRUSTED_PROXY_CIDRS", "").strip()
+    if not raw:
+        return []
+    nets = []
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        try:
+            nets.append(ip_network(entry, strict=False))
+        except ValueError:
+            logger.warning(
+                "MFA_TRUSTED_PROXY_CIDRS: ignoring invalid CIDR %r", entry
+            )
+    return nets
+
+
+_TRUSTED_NETS = _parse_trusted_cidrs()
+
+
+def _peer_ip(request: Request) -> str:
+    """The TCP peer IP — never raises; falls back to ``0.0.0.0``."""
+    return request.client.host if request.client else "0.0.0.0"
+
+
+def _is_trusted_peer(peer: str) -> bool:
+    if not _TRUSTED_NETS:
+        return False
+    try:
+        peer_ip = ip_address(peer)
+    except ValueError:
+        return False
+    return any(peer_ip in net for net in _TRUSTED_NETS)
+
+
+def _real_ip(request: Request) -> str:
+    """If TCP peer is in a trusted proxy CIDR, peel off the first
+    ``X-Forwarded-For`` IP; otherwise return the peer.
+
+    ``X-Forwarded-For`` from an *untrusted* peer is intentionally ignored —
+    accepting it would let any client spoof their bucket.
+    """
+    peer = _peer_ip(request)
+    if not _is_trusted_peer(peer):
+        return peer
+    xff = request.headers.get("X-Forwarded-For")
+    if not xff:
+        return peer
+    first = xff.split(",")[0].strip()
+    return first or peer
+
+
+def _user_id_from_jwt(request: Request) -> str | None:
+    """Sync JWT decode → ``user_id`` (or ``sub``) claim. None on failure.
+
+    Uses the same secret + algorithm as ``auth_middleware`` so a token that
+    is valid for the API surface is also valid for the rate-limit key.
+    Bad / missing / expired tokens silently return None — slowapi falls
+    back to IP bucketing.
+    """
+    auth = request.headers.get("Authorization", "")
+    if not auth.startswith("Bearer "):
+        return None
+    token = auth[7:].strip()
+    if not token or token in {"null", "undefined"}:
+        return None
+    # Late import to avoid pulling jose into module-load when not needed.
+    try:
+        from config import JWT_ALGORITHM, JWT_SECRET_KEY
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None
+    uid = payload.get("user_id") or payload.get("sub")
+    if uid is None:
+        return None
+    return str(uid)
+
+
+def mfa_rate_limit_key(request: Request) -> str:
+    """slowapi ``key_func`` for MFA endpoints.
+
+    Order:
+      1. Authenticated → ``user:<id>``
+      2. Trusted-proxy XFF → ``ip:<first hop>``
+      3. TCP peer → ``ip:<peer>``
+    """
+    uid = _user_id_from_jwt(request)
+    if uid is not None:
+        return f"user:{uid}"
+    return f"ip:{_real_ip(request)}"

backend/middleware/mfa_rate_limits.py

@@ -0,0 +1,79 @@
+"""MFA rate-limit configuration (env-overridable).
+
+Best-practice pattern:
+  - Secure-by-default values live in code (kept in sync with the threat model).
+  - Operations can override per-environment via env vars (ConfigMap on K8s)
+    WITHOUT a code change / re-release.
+  - All limits funnel through a single named constant so the decorator stays
+    declarative (``@limiter.limit(MFA_LIMITS.enroll_start)``).
+
+Env-var precedence::
+
+    MFA_RATE_LIMIT_<NAME>  >  default in code
+
+slowapi limit string syntax: ``<count>/<period>`` where period is
+``second|minute|hour|day``. Example: ``"5/minute"``.
+
+NOTE: slowapi binds limits at import time. A change to an env var requires a
+backend restart (rolling restart on K8s, ``docker compose restart backend``
+locally). This is consistent with how ``SECRET_KEY`` / ``MFA_ENCRYPTION_KEY``
+behave.
+"""
+from __future__ import annotations
+
+import logging
+import os
+import re
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+# slowapi limit-string format guard. Keeps a typo from silently disabling
+# rate-limiting at process start.
+_LIMIT_RE = re.compile(r"^\d+/(second|minute|hour|day)$")
+
+
+def _env(name: str, default: str) -> str:
+    """Read ``MFA_RATE_LIMIT_<NAME>``; fall back to ``default``.
+
+    Validates the limit string. On bad input, logs a warning and returns the
+    secure default instead of crashing the process.
+    """
+    value = os.getenv(f"MFA_RATE_LIMIT_{name}", default).strip()
+    if not _LIMIT_RE.match(value):
+        logger.warning(
+            "MFA_RATE_LIMIT_%s='%s' is not a valid slowapi limit string "
+            "(expected '<n>/<second|minute|hour|day>'); using default '%s'.",
+            name, value, default,
+        )
+        return default
+    return value
+
+
+@dataclass(frozen=True)
+class MfaRateLimits:
+    """Aggregate of MFA endpoint rate-limit strings (slowapi format).
+
+    Defaults assume the rate-limit ``key_func`` is ``mfa_rate_limit_key``
+    (user-aware + ingress-aware), NOT raw IP. Per-user buckets are safe to
+    keep generous because a misbehaving user only burns their own quota and
+    cannot starve the rest of the org. If you re-key on raw IP, retighten
+    these values (see README + ``MFA_RATE_LIMIT_<NAME>`` env overrides).
+    """
+
+    # Enrollment lifecycle — per-user buckets, large enough for org-wide rollout
+    enroll_start: str = _env("ENROLL_START", "10/minute")
+    enroll_confirm: str = _env("ENROLL_CONFIRM", "10/minute")
+
+    # Self-service maintenance
+    disable: str = _env("DISABLE", "10/minute")
+    regenerate_backup_codes: str = _env("REGENERATE_BACKUP_CODES", "5/hour")
+
+    # Admin operations (per-admin bucket; bulk reset stays tight because
+    # it is an emergency-only flow).
+    admin_reset: str = _env("ADMIN_RESET", "60/hour")
+    admin_reset_all: str = _env("ADMIN_RESET_ALL", "1/day")
+
+
+# Module-level singleton — import this from routers/mfa.py.
+MFA_LIMITS = MfaRateLimits()

backend/middleware/rate_limiter.py

@@ -2,6 +2,7 @@
 Rate Limiting Middleware for Production Security
 Protects API endpoints from abuse and DDoS attacks
 """
+import os
 import time
 import logging
 from typing import Callable
@@ -16,10 +17,15 @@ from database.connection import redis_client
 
 logger = logging.getLogger(__name__)
 
-# Rate limiter instance using Redis backend  
+# Resolve the Redis storage URI from REDIS_URL (set by docker-compose / k8s
+# ConfigMap). Falls back to the compose service hostname for backward
+# compatibility when REDIS_URL is unset.
+_REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379").rstrip("/")
+_LIMITER_STORAGE_URI = f"{_REDIS_URL}/0" if "/" not in _REDIS_URL.split("//", 1)[-1] else _REDIS_URL
+
 limiter = Limiter(
     key_func=get_remote_address,
-    storage_uri="redis://redis:6379/0",
+    storage_uri=_LIMITER_STORAGE_URI,
     default_limits=["1000/hour"],  # Default global limit
     retry_after=lambda name, t: int(t) + 10
 )

backend/models/cluster.py

@@ -1,6 +1,22 @@
-from pydantic import BaseModel
+from pydantic import BaseModel, field_validator
 from typing import Optional, List
 
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
+
+
+def _validated_acme_backend_url(value: Optional[str]) -> Optional[str]:
+    """Shared field validator body for `acme_backend_url`.
+
+    Pydantic turns the raised ValueError into a 422 with this message attached, so
+    the operator sees why the value was refused instead of discovering months later
+    that HTTP-01 never worked. Returns the normalised value — callers persist THIS,
+    not the raw input, so surrounding whitespace never reaches haproxy.cfg.
+    """
+    try:
+        return validate_acme_backend_url(value)
+    except AcmeBackendUrlError as exc:
+        raise ValueError(str(exc)) from None
+
 class HAProxyClusterCreate(BaseModel):
     name: str
     description: Optional[str] = None
@@ -8,7 +24,18 @@ class HAProxyClusterCreate(BaseModel):
     stats_socket_path: str = "/run/haproxy/admin.sock"
     haproxy_config_path: str = "/etc/haproxy/haproxy.cfg"
     haproxy_bin_path: str = "/usr/sbin/haproxy"  # HAProxy binary path
+    keepalived_config_path: str = "/etc/keepalived/keepalived.conf"  # HA/VIP: keepalived.conf path (Issue #27)
     pool_id: Optional[int] = None  # Which pool this cluster belongs to
+    # The create form submits both of these. Until they were declared here pydantic
+    # dropped them and the INSERT never carried them, so a cluster created with ACME
+    # switched on came back switched off with no error shown — the same silent-success
+    # failure this work exists to remove.
+    acme_enabled: Optional[bool] = None
+    acme_backend_url: Optional[str] = None
+
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
 
 class HAProxyClusterUpdate(BaseModel):
     name: Optional[str] = None
@@ -17,11 +44,16 @@ class HAProxyClusterUpdate(BaseModel):
     stats_socket_path: Optional[str] = None
     haproxy_config_path: Optional[str] = None
     haproxy_bin_path: Optional[str] = None
+    keepalived_config_path: Optional[str] = None
     pool_id: Optional[int] = None
     is_active: Optional[bool] = None
     acme_enabled: Optional[bool] = None
     acme_backend_url: Optional[str] = None
 
+    _validate_acme_backend_url = field_validator("acme_backend_url")(
+        _validated_acme_backend_url
+    )
+
 class HAProxyClusterResponse(BaseModel):
     id: int
     name: str

backend/models/csr.py

@@ -0,0 +1,251 @@
+"""
+Pydantic models for the CSR (Certificate Signing Request) feature (v1.9.0).
+
+A CSR row is the precursor of an ssl_certificates row: the backend generates
+the private key + CSR locally, the operator has the CSR signed by an external
+CA and then imports the signed certificate. The CSR `name` therefore obeys the
+exact same path-traversal contract as the SSL certificate name (Bulgu #21) —
+at import time it becomes /etc/ssl/haproxy/{name}.pem on every agent and is
+shell-processed by the agent script as root.
+
+The import model deliberately has NO private key field: the key never leaves
+the server. It is stored on the ssl_csrs row at generation time and paired
+with the signed certificate server-side.
+"""
+
+import re
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+KEY_ALGORITHMS = ('rsa-2048', 'rsa-4096', 'ecdsa-p256', 'ecdsa-p384')
+
+# RFC 1035 LDH hostname, lowercase, optional single leftmost wildcard label.
+# Single-label names are allowed (internal CAs routinely sign bare hostnames).
+_DNS_NAME_PATTERN = re.compile(
+    r'^(\*\.)?[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?'
+    r'(\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)*$'
+)
+
+# Reject control characters in free-text subject fields: they would be
+# persisted, echoed into the UI / issuer column, and printed into agent logs
+# via `openssl -subject` output.
+_CONTROL_CHARS_PATTERN = re.compile(r'[\x00-\x1f\x7f]')
+
+_MAX_SANS = 100
+_MAX_CERT_PEM_BYTES = 64 * 1024       # a leaf certificate is ~2 KB; 64 KB is generous
+_MAX_CHAIN_PEM_BYTES = 256 * 1024     # agents re-download all cert content every poll
+
+
+def _validate_dns_name(value: str, field_label: str) -> str:
+    v = (value or '').strip().lower()
+    if not v:
+        raise ValueError(f'{field_label} must not be empty')
+    if len(v) > 253:
+        raise ValueError(f'{field_label} must be 253 characters or fewer')
+    if not _DNS_NAME_PATTERN.match(v):
+        raise ValueError(
+            f'{field_label} {value!r} is not a valid DNS name — lowercase '
+            'letters, digits, hyphens and dots only; a wildcard is allowed '
+            'only as the leftmost label (e.g. *.example.com).'
+        )
+    return v
+
+
+def _validate_subject_text(value: Optional[str], field_label: str, max_len: int = 64) -> Optional[str]:
+    if value is None:
+        return None
+    v = value.strip()
+    if not v:
+        return None
+    if len(v) > max_len:
+        raise ValueError(f'{field_label} must be {max_len} characters or fewer')
+    if _CONTROL_CHARS_PATTERN.search(v):
+        raise ValueError(f'{field_label} must not contain control characters')
+    return v
+
+
+def _validate_csr_name(v: str) -> str:
+    """Mirror of SSLCertificateCreate.validate_name_no_path_traversal (Bulgu #21)
+    with one deliberate tightening: max length 100, matching the
+    ssl_certificates.name VARCHAR(100) column (the historical 200-char limit
+    overflows the column and 500s — not replicated here)."""
+    if v is None:
+        raise ValueError('CSR name is required')
+    stripped = v.strip()
+    if not stripped:
+        raise ValueError('CSR name must not be empty')
+    if stripped != v:
+        raise ValueError('CSR name must not contain leading/trailing whitespace')
+    if len(stripped) > 100:
+        raise ValueError('CSR name must be 100 characters or fewer')
+    if not re.match(r'^[A-Za-z0-9_.-]+$', stripped):
+        raise ValueError(
+            f'CSR name={v!r} contains forbidden characters — only letters, '
+            'digits, underscore, hyphen, and dot are allowed (the name becomes '
+            'a filename component under /etc/ssl/haproxy/ at import).'
+        )
+    if '..' in stripped:
+        raise ValueError(f'CSR name={v!r} must not contain ".." (path traversal)')
+    if stripped.startswith('.'):
+        raise ValueError(f'CSR name={v!r} must not start with "." (hidden filename)')
+    if stripped.startswith('-'):
+        raise ValueError(f'CSR name={v!r} must not start with "-" (CLI flag confusion)')
+    return stripped
+
+
+class SSLCSRCreate(BaseModel):
+    name: str                                   # becomes the certificate name at import
+    common_name: str
+    organization: Optional[str] = None          # O
+    organizational_unit: Optional[str] = None   # OU
+    locality: Optional[str] = None              # L
+    state: Optional[str] = None                 # ST
+    country: Optional[str] = None               # C — exactly 2 letters
+    email: Optional[str] = None                 # emailAddress
+    sans: List[str] = []                        # DNS names; CN is auto-added server-side
+    key_algorithm: str = 'rsa-2048'
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        return _validate_csr_name(v)
+
+    @field_validator('common_name')
+    @classmethod
+    def validate_common_name(cls, v):
+        v = _validate_dns_name(v, 'Common Name')
+        # RFC 5280 ub-common-name — many CAs reject CNs longer than 64 chars.
+        if len(v) > 64:
+            raise ValueError(
+                'Common Name must be 64 characters or fewer (RFC 5280 upper '
+                'bound) — put longer names in the SAN list instead.'
+            )
+        return v
+
+    @field_validator('sans')
+    @classmethod
+    def validate_sans(cls, v):
+        if not v:
+            return []
+        if len(v) > _MAX_SANS:
+            raise ValueError(f'At most {_MAX_SANS} SAN entries are allowed')
+        seen = set()
+        result = []
+        for entry in v:
+            normalised = _validate_dns_name(entry, 'SAN entry')
+            if normalised not in seen:
+                seen.add(normalised)
+                result.append(normalised)
+        return result
+
+    @field_validator('organization')
+    @classmethod
+    def validate_organization(cls, v):
+        return _validate_subject_text(v, 'Organization (O)')
+
+    @field_validator('organizational_unit')
+    @classmethod
+    def validate_organizational_unit(cls, v):
+        return _validate_subject_text(v, 'Organizational Unit (OU)')
+
+    @field_validator('locality')
+    @classmethod
+    def validate_locality(cls, v):
+        return _validate_subject_text(v, 'Locality (L)')
+
+    @field_validator('state')
+    @classmethod
+    def validate_state(cls, v):
+        return _validate_subject_text(v, 'State/Province (ST)')
+
+    @field_validator('country')
+    @classmethod
+    def validate_country(cls, v):
+        # cryptography raises a bare ValueError for a non-2-char COUNTRY_NAME;
+        # pre-validate so the operator gets a friendly 422 instead of a 500.
+        if v is None:
+            return None
+        v = v.strip()
+        if not v:
+            return None
+        if not re.match(r'^[A-Za-z]{2}$', v):
+            raise ValueError('Country (C) must be exactly 2 letters (ISO 3166-1 alpha-2, e.g. TR, US)')
+        return v.upper()
+
+    @field_validator('email')
+    @classmethod
+    def validate_email(cls, v):
+        v = _validate_subject_text(v, 'Email', max_len=254)
+        if v is not None and ('@' not in v or v.startswith('@') or v.endswith('@')):
+            raise ValueError('Email must be a valid address (missing or misplaced "@")')
+        return v
+
+    @field_validator('key_algorithm')
+    @classmethod
+    def validate_key_algorithm(cls, v):
+        if v not in KEY_ALGORITHMS:
+            raise ValueError(
+                f'key_algorithm must be one of: {", ".join(KEY_ALGORITHMS)}'
+            )
+        return v
+
+
+class SSLCSRImport(BaseModel):
+    """Import the CA-signed certificate for a pending CSR. The private key is
+    NOT part of the request — it is already stored on the CSR row."""
+    certificate_content: str                    # PEM
+    chain_content: Optional[str] = None         # PEM, optional
+    usage_type: str = 'frontend'                # "frontend" or "server"
+    is_global: bool = False
+    cluster_ids: Optional[List[int]] = None
+    # Escape hatch for name collisions that appeared AFTER the CSR was
+    # created: overrides the CSR's reserved name for the certificate row.
+    name: Optional[str] = None
+
+    @field_validator('certificate_content')
+    @classmethod
+    def validate_certificate(cls, v):
+        if not v or not v.strip():
+            raise ValueError('Certificate content is required')
+        v = v.strip()
+        if len(v.encode('utf-8', errors='ignore')) > _MAX_CERT_PEM_BYTES:
+            raise ValueError('Certificate content exceeds the 64 KB limit')
+        if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+            raise ValueError('Certificate must be in PEM format')
+        return v
+
+    @field_validator('chain_content')
+    @classmethod
+    def validate_chain(cls, v):
+        if v and v.strip():
+            v = v.strip()
+            if len(v.encode('utf-8', errors='ignore')) > _MAX_CHAIN_PEM_BYTES:
+                raise ValueError('Certificate chain exceeds the 256 KB limit')
+            if '-----BEGIN CERTIFICATE-----' not in v or '-----END CERTIFICATE-----' not in v:
+                raise ValueError('Certificate chain must be in PEM format')
+            return v
+        return None
+
+    @field_validator('usage_type')
+    @classmethod
+    def validate_usage_type(cls, v):
+        if v not in ['frontend', 'server']:
+            raise ValueError('usage_type must be either "frontend" or "server"')
+        return v
+
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        if v is None or not str(v).strip():
+            return None
+        return _validate_csr_name(v)
+
+    @model_validator(mode='after')
+    def validate_cluster_selection(self):
+        if not self.is_global and not self.cluster_ids:
+            raise ValueError(
+                'cluster_ids is required when is_global is false — pick at '
+                'least one cluster or import the certificate as global.'
+            )
+        return self

backend/models/frontend.py

@@ -85,6 +85,11 @@ class FrontendConfig(BaseModel):
     response_headers: Optional[str] = None
     options: Optional[str] = None
     tcp_request_rules: Optional[str] = None
+    # Issue #38: SPOE filter directives (Coraza WAF etc.) + frontend log-format.
+    # Passthrough TEXT (no validator) — SPOE `filter ... config <path>` legitimately
+    # references an operator-managed file, so the ACL `-f` guard must NOT apply here.
+    log_format: Optional[str] = None
+    filters: Optional[str] = None
     timeout_client: Optional[int] = None
     timeout_http_request: Optional[int] = None
     rate_limit: Optional[int] = None
@@ -451,26 +456,17 @@ class FrontendConfig(BaseModel):
             if any(dangerous in rule.lower() for dangerous in ['$(', '`']):
                 raise ValueError(f'ACL rule contains potentially dangerous content: "{rule}"')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) — reject
-            # the HAProxy `-f <file>` pattern-file flag here too so the
-            # manual Frontend API mirrors the wizard's parity rule.
-            # HAProxy OpenManager does not provision pattern files
-            # onto the HAProxy node filesystem, so any `-f /path/...`
-            # reference will fail HAProxy's `-c` parse at apply time
-            # with "failed to open pattern file". Reject up-front so
-            # operators get the same actionable error from both the
-            # manual page and the wizard.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'ACL rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not "
-                    "provision pattern files onto the HAProxy node "
-                    "filesystem, so the reference would fail at "
-                    "reload time. Use inline values instead "
-                    "(e.g. `src 10.0.0.0/24` rather than "
-                    "`src -f /etc/haproxy/admins.lst`)."
-                )
+            # Issue #38 follow-up — the `-f <file>` pattern-file flag
+            # is ACCEPTED here (the Bulgu #12 hard reject was removed).
+            # Pattern files are operator-managed host files, exactly
+            # like the SPOE `filter ... config <path>` reference this
+            # release started preserving: bulk import always accepted
+            # `-f`, the free-form fields (request_headers,
+            # tcp_request_rules) always accepted it, and the agent
+            # runs `haproxy -c` before every reload so a missing file
+            # fails safely (previous config keeps running). The route
+            # handlers surface a non-blocking warning listing the
+            # referenced pattern files instead.
 
             validated_rules.append(rule)
 
@@ -510,20 +506,12 @@ class FrontendConfig(BaseModel):
             if not any(rule.startswith(redirect_type) for redirect_type in valid_redirects):
                 raise ValueError(f'Invalid redirect rule: "{rule}". Must start with: location, prefix, or scheme.')
 
-            # Phase K Phase D follow-up (Bulgu #12 round 3) —
-            # mirror the wizard's `-f <file>` guard here. The
-            # `X !X` contradiction check used to live alongside
-            # this guard, but Bulgu #62 (round-22 audit) moved
-            # it into the route handler so updates can grandfather
-            # legacy rules created before the contradiction guard
-            # landed. See `routers/frontend.py::_collect_routing_rule_contradictions`.
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'Redirect rule "{rule}" uses the HAProxy `-f <file>` '
-                    "pattern-file flag, which is not supported in "
-                    "HAProxy OpenManager: the product does not provision "
-                    "pattern files onto the HAProxy node filesystem."
-                )
+            # Issue #38 follow-up — `-f <file>` pattern-file references
+            # are ACCEPTED (Bulgu #12 hard reject removed; see
+            # validate_acl_rules for the full rationale). The `X !X`
+            # contradiction check lives in the route handler
+            # (`routers/frontend.py::_collect_routing_rule_contradictions`,
+            # Bulgu #62) and is unchanged.
 
             validated_rules.append(rule)
 
@@ -531,9 +519,12 @@ class FrontendConfig(BaseModel):
 
     @validator('use_backend_rules')
     def validate_use_backend_rules_syntax(cls, v):
-        """Phase K Phase D follow-up (Bulgu #12 round 3) — manual
-        Frontend API parity guard: reject `-f <file>` references
-        and dangerous shell patterns.
+        """Manual Frontend API guard for dangerous shell patterns.
+
+        Issue #38 follow-up — the Bulgu #12 `-f <file>` hard reject
+        was removed (see validate_acl_rules for the rationale);
+        pattern-file references are operator-managed host files and
+        are surfaced as non-blocking warnings by the route handlers.
 
         Bulgu #62 (round-22 audit) — the `X !X` contradiction check
         previously lived here but moved into the route handler so
@@ -563,13 +554,5 @@ class FrontendConfig(BaseModel):
                     f'use_backend rule contains potentially dangerous '
                     f'content: "{rule}"'
                 )
-            if re.search(r"(^|\s)-f(\s|$)", rule):
-                raise ValueError(
-                    f'use_backend rule "{rule}" uses the HAProxy '
-                    "`-f <file>` pattern-file flag, which is not "
-                    "supported in HAProxy OpenManager: the product "
-                    "does not provision pattern files onto the HAProxy "
-                    "node filesystem."
-                )
             validated_rules.append(rule)
         return validated_rules

backend/models/mfa.py

@@ -0,0 +1,73 @@
+"""MFA-specific Pydantic models — Issue #18, v1.6.0.
+
+Kept in a separate module so the existing User / UserUpdate contracts in
+``backend/models/user.py`` stay byte-identical for backwards compatibility.
+"""
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+class MfaVerifyRequest(BaseModel):
+    """Body of POST /api/auth/login/mfa-verify (pre-auth — no JWT)."""
+
+    mfa_token: str = Field(..., min_length=64, max_length=64)
+    # 6 digits for TOTP or 8 alphanumerics (with optional dash) for backup codes
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaEnrollStartResponse(BaseModel):
+    """Returned by POST /api/mfa/enroll/start."""
+
+    secret: str
+    otpauth_uri: str
+    expires_in: int  # pending enrollment TTL in seconds
+
+
+class MfaEnrollConfirmRequest(BaseModel):
+    """Body of POST /api/mfa/enroll/confirm (TOTP only — backup codes not yet issued)."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaEnrollConfirmResponse(BaseModel):
+    enabled: bool
+    backup_codes: List[str]
+    method: Literal["totp"] = "totp"
+
+
+class MfaDisableRequest(BaseModel):
+    """Body of POST /api/mfa/disable — TOTP or backup."""
+
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaRegenerateBackupRequest(BaseModel):
+    """Body of POST /api/mfa/backup-codes/regenerate — TOTP only."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaRegenerateBackupResponse(BaseModel):
+    backup_codes: List[str]
+
+
+class MfaAdminResetRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset/{user_id}."""
+
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaAdminResetAllRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset-all (emergency)."""
+
+    confirm: Literal["RESET ALL MFA"]
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaStatusResponse(BaseModel):
+    enabled: bool
+    method: Optional[str] = None
+    enrolled_at: Optional[str] = None
+    last_used_at: Optional[str] = None
+    backup_codes_remaining: int = 0

backend/models/site_wizard.py

@@ -179,35 +179,17 @@ _MAX_RULE_STRING_LEN = 4096
 # attempts.
 _DANGEROUS_RULE_PATTERNS = ("$(", "`")
 
-# Phase K Phase D follow-up (Bulgu #12 round 3) — the HAProxy `-f
-# <file>` ACL/condition flag instructs HAProxy to load match patterns
-# from a server-side file at parse time. HAProxy OpenManager is a
-# fully-managed product: we do NOT provision pattern files onto the
-# HAProxy node's filesystem, and operators have no UI to upload one.
-# A `-f /some/path` reference therefore ALWAYS resolves to
-# "file not found" when HAProxy's real `-c` parse runs at apply
-# time, producing exactly the operator-reported failure mode:
-#   [ALERT] parsing ACL 'acl1' : failed to open pattern file </path>.
-#   [ALERT] parsing switching rule : no such ACL : 'acl1'.
-#
-# Surface this BEFORE persist by rejecting `-f` in any rule string
-# that comes through the wizard / manual frontend API. Reject ALL
-# variants (` -f `, leading `-f `, trailing `... -f`) defensively so
-# operators cannot slip the flag through with creative spacing.
-# The check is anchored to ACL/condition rule strings only; raw
-# HAProxy snippet fields (tcp_request_rules, request_headers, ...)
-# are NOT touched because those are inherently free-form and
-# advanced operators may legitimately reference pre-provisioned
-# pattern files there.
-_ACL_FILE_FLAG_PATTERN = re.compile(r"(^|\s)-f(\s|$)")
-_ACL_FILE_FLAG_MESSAGE = (
-    "pattern-file references with '-f <file>' are not supported in ACL / "
-    "use_backend / redirect rules: HAProxy OpenManager does not provision "
-    "pattern files onto the HAProxy node's filesystem, so the reference "
-    "would always fail at HAProxy reload time. Use inline values "
-    "instead (e.g. `acl is_admin src 10.0.0.0/24` rather than "
-    "`acl is_admin src -f /etc/haproxy/admins.lst`)."
-)
+# Issue #38 follow-up — the HAProxy `-f <file>` ACL/condition flag
+# loads match patterns from a file on the HAProxy host. The Bulgu #12
+# hard reject (`_ACL_FILE_FLAG_PATTERN`/`_ACL_FILE_FLAG_MESSAGE`) was
+# removed: pattern files are operator-managed host files (exactly like
+# the SPOE `filter ... config <path>` reference preserved since
+# v1.8.8), bulk import and the free-form fields (tcp_request_rules,
+# request_headers) always accepted them, and the agent runs
+# `haproxy -c` before every reload so a missing file fails safely
+# (the previous config keeps running). The manual frontend route
+# handlers emit a non-blocking warning listing referenced pattern
+# files (`routers/frontend.py::_pattern_file_warnings`).
 
 # Phase K Phase D follow-up (Bulgu #13) — detect a routing /
 # redirect rule whose condition references the SAME ACL in both
@@ -305,13 +287,10 @@ def _validate_haproxy_directive_string(
                 f"{field_label} entry contains potentially dangerous content: "
                 f"{pattern!r}"
             )
-    # Phase K Phase D follow-up (Bulgu #12 round 3) — reject the
-    # HAProxy `-f <file>` pattern-file flag because OpenManager does
-    # not manage the HAProxy node filesystem. See the module-level
-    # `_ACL_FILE_FLAG_PATTERN` docstring for the full operator-
-    # reported failure mode this guards against.
-    if _ACL_FILE_FLAG_PATTERN.search(stripped):
-        raise ValueError(f"{field_label}: {_ACL_FILE_FLAG_MESSAGE}")
+    # Issue #38 follow-up — `-f <file>` pattern-file references are
+    # ACCEPTED (Bulgu #12 hard reject removed; see the module-level
+    # `_ACL_FILE_FLAG_PATTERN` comment). The route handlers surface
+    # a non-blocking pattern-file warning instead.
     # Phase K Phase D follow-up (Bulgu #13) — for routing /
     # redirect rules (not ACL definitions themselves), reject a
     # condition that contains the same ACL in both positive and
@@ -488,6 +467,32 @@ class ServerStep(BaseModel):
                     "splits at the first space, so a space inside the "
                     "SNI value would corrupt the rendered config."
                 )
+        # Bulgu #90 (round-24 audit) — `cookie_value` is interpolated
+        # into the rendered `server <name> <addr>:<port> ... cookie
+        # <value> ...` line. HAProxy tokenises the line by whitespace
+        # and treats `;` as an INLINE COMMENT — and RFC 6265 cookie-
+        # value grammar separately disallows whitespace, `;`, `,`,
+        # `\\` and `"`. Pre-fix the validator only rejected newlines,
+        # so an operator could set `cookie_value="srv1; secure"` and
+        # see HAProxy silently truncate the server line at `;`
+        # (everything after becomes a comment). The Set-Cookie header
+        # rendered to clients would also fail the RFC's cookie-value
+        # grammar and most browsers drop the cookie, breaking session
+        # affinity without any error surface. Restrict to the
+        # conservative intersection — alphanumeric plus `_.-` —
+        # identical to backend.cookie_name. Legitimate session
+        # identifiers all fit in this set.
+        if info and info.field_name == "cookie_value":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.\-]+", v):
+                raise ValueError(
+                    "server.cookie_value must contain only alphanumerics, "
+                    "'.', '_' or '-' (HAProxy tokenises the server line "
+                    "by whitespace and treats ';' as an inline comment; "
+                    "RFC 6265 separately disallows whitespace, ';', ',' "
+                    "and backslash in cookie values). Got "
+                    f"{v!r}."
+                )
         return v
     # R17 (label corrected R18): CA bundle used by HAProxy to VERIFY the
     # upstream server's TLS certificate. Maps to the `ca-file` directive
@@ -658,6 +663,32 @@ class BackendStep(BaseModel):
         fields like `request_headers` / `response_headers` /
         `tcp_request_rules` are intentionally line-oriented and ARE
         NOT touched here.
+
+        Bulgu #90 (round-24 audit) — extend the validator beyond
+        newlines. Pre-fix `cookie_name` accepted strings like
+        `SESS'; DROP TABLE backends; --` (the SQL substring is
+        harmless thanks to parameterised queries, but the `;` is
+        HAProxy's INLINE COMMENT character: the renderer emits
+        `cookie SESS'; DROP TABLE backends; -- insert indirect
+        nocache`, which HAProxy parses as `cookie SESS'` followed by
+        an inline comment that swallows the persistence options the
+        operator typed). Result: session affinity silently broken
+        and the operator has zero diagnostic signal — the wizard
+        accepted the input, the apply succeeded, but cookies never
+        get re-emitted with the expected name. The same trap exists
+        for any HAProxy-section-keyword or whitespace token because
+        HAProxy tokenises by space.
+
+        `cookie_name` MUST be a single token. We use the conservative
+        intersection of RFC 6265 cookie-token chars and HAProxy
+        directive-name chars: alphanumeric plus `_.-`. Operators
+        with legitimate session cookies all live inside this set.
+
+        `cookie_options` is a space-separated keyword list
+        (`insert indirect nocache` etc., optionally `domain example.
+        com`, `attr SameSite=Lax`). Allow letters/digits/space/dot/
+        hyphen/underscore/equals. Reject `;` (comment), backslash,
+        quotes, and other shell metacharacters.
         """
         if v is None:
             return v
@@ -669,6 +700,28 @@ class BackendStep(BaseModel):
                 "newlines would smuggle additional directives into "
                 "the rendered config)"
             )
+        field_name = info.field_name if info else "cookie field"
+        if field_name == "cookie_name":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.\-]+", v):
+                raise ValueError(
+                    "backend.cookie_name must contain only alphanumerics, "
+                    "'.', '_' or '-' (HAProxy tokenises the `cookie` "
+                    "directive by whitespace and treats ';' as an inline "
+                    "comment — anything else silently truncates the "
+                    "rendered persistence options). Got "
+                    f"{v!r}."
+                )
+        elif field_name == "cookie_options":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.=\- ]*", v):
+                raise ValueError(
+                    "backend.cookie_options must contain only "
+                    "alphanumerics, spaces, '=', '.', '_' or '-' "
+                    "(HAProxy parses `;` as an inline comment, and "
+                    "quoting / backslash metacharacters are not "
+                    f"part of the `cookie` keyword grammar). Got {v!r}."
+                )
         return v
 
     @field_validator("health_check_uri")
@@ -1009,21 +1062,12 @@ class FrontendStep(BaseModel):
         normalised: List[Union[str, dict]] = []
         for el in v:
             if isinstance(el, dict):
-                # Phase K Phase D follow-up (Bulgu #12 round 3
-                # extension) — dict-shaped redirect rules emit their
-                # `condition` / `target` fields VERBATIM into the
-                # rendered HAProxy directive. A dict with
-                # `condition: "if { src -f /etc/haproxy/x.lst }"`
-                # would slip past the string-only validator above
-                # and trigger the same operator-reported "failed to
-                # open pattern file" rejection at apply time. Reject
-                # `-f` in any string-shaped value the dict carries.
-                for field_name in ("condition", "target", "type"):
-                    val = el.get(field_name)
-                    if isinstance(val, str) and _ACL_FILE_FLAG_PATTERN.search(val):
-                        raise ValueError(
-                            f"redirect_rules.{field_name}: {_ACL_FILE_FLAG_MESSAGE}"
-                        )
+                # Issue #38 follow-up — dict-shaped redirect rules may
+                # carry `-f <file>` pattern-file references in their
+                # `condition`/`target` values; these are ACCEPTED now
+                # (Bulgu #12 hard reject removed — operator-managed
+                # host files, fail-safe apply; see module-level
+                # `_ACL_FILE_FLAG_PATTERN` comment).
                 # Bulgu #13 extension — same contradiction guard
                 # for dict-shaped redirect conditions.
                 cond_val = el.get("condition")

backend/models/vip.py

@@ -0,0 +1,245 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Pydantic request/response models for the VIP management API. Field validation
+is strict because several values flow into a generated keepalived.conf and into
+root-run agent commands — we reuse the same FORBIDDEN-metacharacter discipline as
+models/agent.py (Bulgu #81) and validate the VIP as a real IPv4 address.
+
+Pydantic idiom: the project runs pydantic>=2.5; this module uses the v2-native
+@field_validator/@model_validator style (matching models/ssl.py).
+"""
+import ipaddress
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+# Shell/keepalived.conf metacharacters that must never appear in a value that
+# reaches the generated config or a root-run agent command (mirrors
+# models/agent.py:202, the Bulgu #81 convention).
+_FORBIDDEN = set('$`;&|<>"\'\\\n\r\x00*?')
+
+
+def _validate_iface(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('network_interface must be a non-empty string')
+    s = v.strip()
+    if len(s) > 64:
+        raise ValueError('network_interface too long (max 64 chars)')
+    if any(c in _FORBIDDEN for c in s):
+        raise ValueError('network_interface contains a forbidden character')
+    # Linux iface names: letters, digits, and . _ - : @ (vlans/aliases/altnames)
+    import re as _re
+    if not _re.match(r'^[A-Za-z0-9][A-Za-z0-9._:@-]{0,63}$', s):
+        raise ValueError(
+            'network_interface must start alphanumeric and contain only '
+            'letters, digits, and . _ - : @'
+        )
+    return s
+
+
+def _validate_ipv4(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('virtual_ip must be a non-empty string')
+    s = v.strip()
+    try:
+        addr = ipaddress.ip_address(s)
+    except ValueError:
+        raise ValueError(f'virtual_ip={v!r} is not a valid IP address')
+    if addr.version != 4:
+        raise ValueError('virtual_ip must be IPv4 — IPv6 VIPs are not supported yet')
+    return s
+
+
+class VIPMemberIn(BaseModel):
+    agent_id: int
+    network_interface: str
+    role: str = 'BACKUP'
+    priority: int = 100
+
+    @field_validator('network_interface')
+    @classmethod
+    def _iface(cls, v):
+        return _validate_iface(v)
+
+    @field_validator('role')
+    @classmethod
+    def _role(cls, v):
+        u = (v or '').strip().upper()
+        if u not in ('MASTER', 'BACKUP'):
+            raise ValueError("role must be 'MASTER' or 'BACKUP'")
+        return u
+
+    @field_validator('priority')
+    @classmethod
+    def _priority(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 254):
+            raise ValueError('priority must be an integer between 1 and 254')
+        return v
+
+
+def _validate_members(members: List['VIPMemberIn']) -> List['VIPMemberIn']:
+    # >=1 node: a single-node VIP is a keepalived-managed floating IP without failover
+    # (valid, e.g. a one-box HAProxy that wants a stable VIP, or before a 2nd node is added).
+    # Two or more nodes give actual VRRP failover. The UI flags the single-node case.
+    if not members:
+        raise ValueError('a VIP needs at least 1 member node')
+    agent_ids = [m.agent_id for m in members]
+    if len(set(agent_ids)) != len(agent_ids):
+        raise ValueError('each node may appear at most once in a VIP')
+    masters = [m for m in members if m.role == 'MASTER']
+    if len(masters) != 1:
+        raise ValueError('exactly one member must be MASTER')
+    master_prio = masters[0].priority
+    if any(m.role == 'BACKUP' and m.priority >= master_prio for m in members):
+        raise ValueError('the MASTER must have a strictly higher priority than every BACKUP')
+    return members
+
+
+def _validate_auth_pass(v: Optional[str]) -> Optional[str]:
+    if v is None or v == '':
+        return None
+    # keepalived PASS auth_pass is silently truncated to 8 chars (B-3) — reject longer
+    # so MASTER/BACKUP never silently disagree.
+    if not (1 <= len(v) <= 8):
+        raise ValueError('auth_pass must be 1-8 characters (keepalived PASS limit)')
+    if any(c in _FORBIDDEN for c in v):
+        raise ValueError('auth_pass contains a forbidden character')
+    # No whitespace: keepalived PASS auth_pass is a single token, and a whitespace-containing
+    # secret would only partially redact in the masked config diff (review HIGH-2).
+    if any(c.isspace() for c in v):
+        raise ValueError('auth_pass must not contain whitespace')
+    return v
+
+
+class VIPCreate(BaseModel):
+    name: str
+    description: Optional[str] = None
+    pool_id: int
+    virtual_ip: str
+    prefix_length: int = 24
+    virtual_router_id: Optional[int] = None   # auto-allocated within the pool when omitted
+    advert_int: int = 1
+    auth_pass: Optional[str] = None           # plaintext on the wire; stored Fernet-encrypted
+    use_unicast: bool = True
+    track_haproxy: bool = True
+    members: List[VIPMemberIn]
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if not isinstance(v, str) or not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255:
+            raise ValueError('name too long (max 255 chars)')
+        if any(c in _FORBIDDEN for c in s):
+            raise ValueError('name contains a forbidden character')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v)
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 32):
+            raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be an integer between 1 and 255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('advert_int must be an integer between 1 and 255 (seconds)')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        _validate_members(self.members)
+        return self
+
+
+class VIPUpdate(BaseModel):
+    """All fields optional — only provided fields are changed. Any change sets the
+    VIP back to last_config_status='PENDING' (router-side)."""
+    name: Optional[str] = None
+    description: Optional[str] = None
+    virtual_ip: Optional[str] = None
+    prefix_length: Optional[int] = None
+    virtual_router_id: Optional[int] = None
+    advert_int: Optional[int] = None
+    auth_pass: Optional[str] = None           # provide only to rotate; omit to keep existing
+    use_unicast: Optional[bool] = None
+    track_haproxy: Optional[bool] = None
+    members: Optional[List[VIPMemberIn]] = None
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if v is None:
+            return v
+        if not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255 or any(c in _FORBIDDEN for c in s):
+            raise ValueError('name invalid (too long or forbidden character)')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v) if v is not None else v
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 32):
+            raise ValueError('prefix_length must be 1-32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be 1-255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('advert_int must be 1-255 seconds')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        if self.members is not None:
+            _validate_members(self.members)
+        return self

backend/requirements.txt

@@ -15,4 +15,5 @@ bcrypt>=4.0.1
 slowapi>=0.1.9
 psutil>=5.9.8
 pytz>=2023.3
-josepy>=1.14.0
+josepy>=1.14.0
+pyotp>=2.9.0

backend/routers/acme_diagnostics.py

@@ -16,9 +16,11 @@ Per-user 5/min rate-limit via user_activity_logs SQL count (M18 / R50).
 
 import json
 import logging
+import uuid
 from datetime import datetime
 from typing import List, Optional
 
+import asyncpg
 from fastapi import APIRouter, Header, HTTPException
 
 from auth_middleware import check_user_permission, get_current_user_from_token
@@ -58,16 +60,39 @@ async def _enforce_rate_limit(conn, user_id: int, action: str) -> None:
 
 
 async def _load_order(conn, order_id: int) -> dict:
-    row = await conn.fetchrow(
-        """
-        SELECT id, account_id, status, domains, cluster_ids, error_detail,
-               post_completion_actions, pending_apply_version_name,
-               wizard_staged_until, created_by
-        FROM letsencrypt_orders
-        WHERE id = $1
-        """,
-        order_id,
-    )
+    """Fetch the order row, or raise a clean 404.
+
+    Bulgu #96 (prod-canary audit): `letsencrypt_orders.id` is a Postgres
+    int4 column. A path-param `order_id` outside the int4 range
+    (e.g. > 2_147_483_647) used to bubble up as
+    `asyncpg.exceptions.DataError: invalid input for query argument $1:
+    ... (value out of int32 range)` — which the diagnostics endpoint
+    then surfaced in a `diagnostics_unavailable` envelope, leaking the
+    raw Postgres / asyncpg error string ("query argument $1",
+    "int32 range") into the operator-visible response body.
+    Semantically an out-of-range ID can never reference a real order,
+    so we treat it identically to "row not found" and return a clean
+    404 — same shape as the not-found path, no SQL detail leakage.
+    """
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT id, account_id, status, domains, cluster_ids, error_detail,
+                   post_completion_actions, pending_apply_version_name,
+                   wizard_staged_until, created_by, challenge_type
+            FROM letsencrypt_orders
+            WHERE id = $1
+            """,
+            order_id,
+        )
+    except asyncpg.exceptions.DataError as exc:
+        logger.info(
+            "ACME order lookup rejected by Postgres (out-of-range / "
+            "uncastable id): order_id=%s exc=%s",
+            order_id,
+            exc,
+        )
+        raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     if not row:
         raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     return dict(row)
@@ -86,35 +111,154 @@ def _parse_jsonb_list(raw, default):
     return default
 
 
+def _diagnostic_failure_envelope(
+    order_id: int,
+    correlation_id: str,
+    exc: Exception,
+    *,
+    stage: str,
+) -> dict:
+    """Build a structured response when the diagnostic suite itself
+    cannot run. Bulgu #94 (Round-25): we return HTTP 200 with this
+    envelope rather than 500 so the UI can still SHOW the operator
+    what happened — the panel's whole purpose is to surface failure
+    causes, and the panel itself silently 500-ing is the worst-case
+    UX. The server-side log carries the full traceback keyed by
+    correlation_id for operator follow-up.
+    """
+    return {
+        "order_id": order_id,
+        "status": "diagnostics_unavailable",
+        "checks": [
+            {
+                "id": "diagnostics_runner",
+                "label": "Diagnostic runner",
+                "status": "fail",
+                "severity": "error",
+                "message": (
+                    f"Diagnostics could not run ({stage}): "
+                    f"{exc.__class__.__name__}: {exc}"
+                ),
+                "details": {
+                    "stage": stage,
+                    "exception_type": exc.__class__.__name__,
+                    "exception_message": str(exc),
+                    "correlation_id": correlation_id,
+                    "hint": (
+                        "Check the backend log for correlation_id "
+                        f"{correlation_id} for the full traceback."
+                    ),
+                },
+                "duration_ms": None,
+            }
+        ],
+        "humanized_error": {
+            "title": "Diagnostic panel could not run",
+            "message": (
+                "The diagnostic runner itself crashed before any check "
+                "could complete. This is independent of whether the ACME "
+                "provider is reachable from this cluster."
+            ),
+            "hint": (
+                "Share the correlation_id below with the platform team; "
+                "they can grep the API log for the full traceback."
+            ),
+            "correlation_id": correlation_id,
+        },
+        "meta": {
+            "correlation_id": correlation_id,
+            "error_stage": stage,
+            "error_type": exc.__class__.__name__,
+            "error_message": str(exc),
+        },
+        "generated_at": datetime.utcnow().isoformat() + "Z",
+    }
+
+
 @router.post("/orders/{order_id}/diagnostics")
 async def run_diagnostics(order_id: int, authorization: str = Header(None)):
-    """Run the full pre-flight + post-failure diagnostic suite."""
+    """Run the full pre-flight + post-failure diagnostic suite.
+
+    Bulgu #94 (Round-25 audit): this endpoint must NEVER return HTTP 500
+    for an in-suite failure. The diagnostic panel exists precisely to
+    explain what is broken; producing an opaque 500 defeats the entire
+    feature. Authentication / authorisation / rate-limit / not-found
+    errors still raise the appropriate 4xx, but any unexpected
+    exception from check execution is converted to a 200 response with
+    a structured failure envelope so the UI can display the cause.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
         await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostics_run")
         order = await _load_order(conn, order_id)
+    except HTTPException:
+        await close_database_connection(conn)
+        raise
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        logger.exception(
+            "ACME diagnostics setup failed for order=%s correlation_id=%s",
+            order_id,
+            correlation_id,
+        )
+        try:
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="load_order")
+        finally:
+            await close_database_connection(conn)
 
+    try:
         domains = _parse_jsonb_list(order["domains"], [])
         cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
 
-        results = await run_checks(
-            conn,
-            domains=domains,
-            cluster_ids=cluster_ids,
-            account_id=order["account_id"],
-        )
+        try:
+            results = await run_checks(
+                conn,
+                domains=domains,
+                cluster_ids=cluster_ids,
+                account_id=order["account_id"],
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            # run_checks now wraps individual checks, but a top-level
+            # crash (e.g. lost DB connection) still needs to be visible.
+            logger.exception(
+                "ACME diagnostics top-level failure for order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
+            )
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="run_checks")
 
-        humanized_error = humanize_error_detail(order["error_detail"])
+        try:
+            humanized_error = humanize_error_detail(order["error_detail"])
+        except Exception as exc:  # noqa: BLE001 — defensive
+            logger.warning(
+                "humanize_error_detail failed for order=%s correlation_id=%s: %s",
+                order_id,
+                correlation_id,
+                exc,
+            )
+            humanized_error = {
+                "title": "ACME error (raw)",
+                "message": str(order["error_detail"]) if order["error_detail"] else "",
+                "hint": "",
+                "parse_error": exc.__class__.__name__,
+            }
 
         return {
             "order_id": order_id,
             "status": order["status"],
             "checks": results,
             "humanized_error": humanized_error,
+            "meta": {
+                "correlation_id": correlation_id,
+                "checks_total": len(results),
+                "checks_failed": sum(1 for r in results if r.get("status") == "fail"),
+                "checks_warn": sum(1 for r in results if r.get("status") == "warn"),
+            },
             "generated_at": datetime.utcnow().isoformat() + "Z",
         }
     finally:
@@ -127,7 +271,15 @@ async def rerun_diagnostic_check(
     check_id: str,
     authorization: str = Header(None),
 ):
-    """Re-run a single check (DNS / port80 / routing / account / agents)."""
+    """Re-run a single check (DNS / port80 / routing / account / agents).
+
+    Bulgu #94 follow-up (Round-25 audit): the rerun path is just as
+    sensitive to opaque 500s as the full-suite POST. If
+    `_enforce_rate_limit` / `_load_order` / `run_checks` raises an
+    unexpected exception, we surface a structured `fail` row in the
+    same shape the table already renders — so the operator clicking
+    "Re-run" never sees an opaque toast and the row updates in place.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
@@ -138,30 +290,120 @@ async def rerun_diagnostic_check(
             detail=f"Unknown check_id '{check_id}'. Valid: {', '.join(CHECK_IDS)}",
         )
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
-        await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
-        order = await _load_order(conn, order_id)
+        try:
+            await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
+            order = await _load_order(conn, order_id)
+        except HTTPException:
+            raise
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun setup failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run setup failed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "setup",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "setup"},
+            }
 
-        domains = _parse_jsonb_list(order["domains"], [])
-        cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
-
-        results = await run_checks(
-            conn,
-            domains=domains,
-            cluster_ids=cluster_ids,
-            account_id=order["account_id"],
-            only=[check_id],
-        )
+        try:
+            domains = _parse_jsonb_list(order["domains"], [])
+            cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
+            results = await run_checks(
+                conn,
+                domains=domains,
+                cluster_ids=cluster_ids,
+                account_id=order["account_id"],
+                only=[check_id],
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun run_checks failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run crashed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "run_checks",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "run_checks"},
+            }
 
         return {
             "order_id": order_id,
             "check": results[0] if results else None,
+            "meta": {"correlation_id": correlation_id},
         }
     finally:
         await close_database_connection(conn)
 
 
+async def _user_activity_columns(conn) -> set:
+    """Return the set of column names actually present on user_activity_logs.
+
+    Bulgu #95 (Round-25 audit) — the canonical migration for
+    `user_activity_logs` defines `id, user_id, action, resource_type,
+    resource_id, details, ip_address, user_agent, created_at, timestamp`.
+    There is NO `status` column. The original `/events` SELECT pulled
+    `status` directly, so every diagnostic-panel open against an order
+    that had any user-activity-log correlation raised
+    `UndefinedColumnError: column "status" does not exist` and the API
+    returned HTTP 500. We now introspect the schema and only project
+    the columns that exist, so deployments at any migration level keep
+    rendering the diagnostic panel.
+    """
+    try:
+        rows = await conn.fetch(
+            """
+            SELECT column_name
+            FROM information_schema.columns
+            WHERE table_name = 'user_activity_logs'
+            """
+        )
+        return {r["column_name"] for r in rows}
+    except Exception as exc:  # noqa: BLE001 — schema introspection is best-effort
+        logger.warning("user_activity_logs schema introspection failed: %s", exc)
+        return set()
+
+
 @router.get("/orders/{order_id}/events")
 async def get_order_events(
     order_id: int,
@@ -174,6 +416,12 @@ async def get_order_events(
       resource_id=order_id) for context.
 
     Sorted by created_at ASC (oldest first) so the timeline reads naturally.
+
+    Bulgu #94/#95 (Round-25 audit): every sub-query is wrapped so that a
+    partial failure (missing column, missing table, malformed JSONB) is
+    surfaced via `meta.errors[]` rather than collapsing the whole panel
+    to HTTP 500. The diagnostic UI is a debugging surface — it must not
+    itself become opaque when one of its data sources is degraded.
     """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
@@ -182,95 +430,168 @@ async def get_order_events(
     if limit <= 0 or limit > 500:
         limit = 100
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
+    section_errors: List[dict] = []
     try:
-        # Existence check
-        await _load_order(conn, order_id)
-
-        # Detect whether acme_order_events exists (zero-impact for envs that
-        # have not yet run the v1.5.0 migration). Returns empty event_log when
-        # not yet present rather than 500-ing.
-        events_table_exists = await conn.fetchval(
-            """
-            SELECT EXISTS (
-                SELECT 1 FROM information_schema.tables WHERE table_name = 'acme_order_events'
+        try:
+            await _load_order(conn, order_id)
+        except HTTPException:
+            raise
+        except Exception as exc:  # noqa: BLE001 — surface, don't 500
+            logger.exception(
+                "ACME events load_order failed order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
             )
-            """
-        )
+            return {
+                "order_id": order_id,
+                "events": [],
+                "count": 0,
+                "meta": {
+                    "correlation_id": correlation_id,
+                    "errors": [
+                        {
+                            "section": "load_order",
+                            "exception_type": exc.__class__.__name__,
+                            "message": str(exc),
+                        }
+                    ],
+                },
+            }
 
         events: List[dict] = []
-        if events_table_exists:
-            event_rows = await conn.fetch(
+
+        # --- Section 1: acme_order_events ---
+        try:
+            events_table_exists = await conn.fetchval(
                 """
-                SELECT id, event_type, severity, message, details, correlation_id, created_at
-                FROM acme_order_events
-                WHERE order_id = $1
-                ORDER BY created_at ASC, id ASC
-                LIMIT $2
-                """,
+                SELECT EXISTS (
+                    SELECT 1 FROM information_schema.tables
+                    WHERE table_name = 'acme_order_events'
+                )
+                """
+            )
+            if events_table_exists:
+                event_rows = await conn.fetch(
+                    """
+                    SELECT id, event_type, severity, message, details, correlation_id, created_at
+                    FROM acme_order_events
+                    WHERE order_id = $1
+                    ORDER BY created_at ASC, id ASC
+                    LIMIT $2
+                    """,
+                    order_id,
+                    limit,
+                )
+                for r in event_rows:
+                    # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
+                    # raw JSON strings (no codec on the pool). For the FE
+                    # contract the `details` field MUST be either a dict or
+                    # null — otherwise the React renderer ends up trying to
+                    # access `details.foo` on a plain string and silently
+                    # gets undefined.
+                    _det = r["details"]
+                    if isinstance(_det, str):
+                        try:
+                            _det = json.loads(_det)
+                        except Exception:
+                            _det = {}
+                    if not isinstance(_det, (dict, list)):
+                        _det = {} if _det is None else {"raw": str(_det)}
+                    events.append({
+                        "source": "acme_order_event",
+                        "id": r["id"],
+                        "event_type": r["event_type"],
+                        "severity": r["severity"],
+                        "message": r["message"],
+                        "details": _det,
+                        "correlation_id": r["correlation_id"],
+                        "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
+                            if r["created_at"] else None,
+                    })
+        except Exception as exc:  # noqa: BLE001 — surface and continue
+            logger.exception(
+                "ACME events acme_order_events query failed order=%s correlation_id=%s",
                 order_id,
-                limit,
+                correlation_id,
             )
-            for r in event_rows:
-                # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
-                # raw JSON strings (no codec on the pool). For the FE
-                # contract the `details` field MUST be either a dict or
-                # null — otherwise the React renderer ends up trying to
-                # access `details.foo` on a plain string and silently
-                # gets undefined.
-                _det = r["details"]
-                if isinstance(_det, str):
-                    try:
-                        _det = json.loads(_det)
-                    except Exception:
-                        _det = {}
-                if not isinstance(_det, (dict, list)):
-                    _det = {} if _det is None else {"raw": str(_det)}
-                events.append({
-                    "source": "acme_order_event",
-                    "id": r["id"],
-                    "event_type": r["event_type"],
-                    "severity": r["severity"],
-                    "message": r["message"],
-                    "details": _det,
-                    "correlation_id": r["correlation_id"],
-                    "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
-                        if r["created_at"] else None,
+            section_errors.append({
+                "section": "acme_order_events",
+                "exception_type": exc.__class__.__name__,
+                "message": str(exc),
+            })
+
+        # --- Section 2: user_activity_logs (best-effort, schema-aware) ---
+        try:
+            ua_columns = await _user_activity_columns(conn)
+            if "resource_id" in ua_columns and "resource_type" in ua_columns:
+                # Project only columns we know exist. `status` is NOT in
+                # the canonical schema and was the original 500 cause.
+                projection_candidates = [
+                    "id", "action", "resource_type", "resource_id",
+                    "details", "created_at", "user_id", "status",
+                ]
+                projection = [c for c in projection_candidates if c in ua_columns]
+                if "id" not in projection or "created_at" not in projection:
+                    raise RuntimeError(
+                        "user_activity_logs is missing required columns "
+                        "(id / created_at) — skipping correlation"
+                    )
+                sql = (
+                    f"SELECT {', '.join(projection)} "
+                    "FROM user_activity_logs "
+                    "WHERE resource_type = 'letsencrypt_order' AND resource_id = $1 "
+                    "ORDER BY created_at ASC, id ASC LIMIT $2"
+                )
+                ua_rows = await conn.fetch(sql, str(order_id), limit)
+                for r in ua_rows:
+                    rd = dict(r)
+                    raw_details = rd.get("details")
+                    if isinstance(raw_details, str):
+                        msg = raw_details[:500]
+                        details_obj = {}
+                        try:
+                            parsed = json.loads(raw_details)
+                            if isinstance(parsed, (dict, list)):
+                                details_obj = parsed
+                        except Exception:
+                            details_obj = {}
+                    elif isinstance(raw_details, (dict, list)):
+                        msg = ""
+                        details_obj = raw_details
+                    else:
+                        msg = ""
+                        details_obj = {}
+                    raw_status = rd.get("status") or ""
+                    severity = "info" if str(raw_status).lower() in ("success", "ok", "") else "warn"
+                    events.append({
+                        "source": "user_activity_log",
+                        "id": rd.get("id"),
+                        "event_type": rd.get("action"),
+                        "severity": severity,
+                        "message": msg,
+                        "details": details_obj,
+                        "correlation_id": None,
+                        "created_at": rd["created_at"].isoformat().replace("+00:00", "Z")
+                            if rd.get("created_at") else None,
+                    })
+            else:
+                section_errors.append({
+                    "section": "user_activity_logs",
+                    "exception_type": "SchemaMissing",
+                    "message": "user_activity_logs lacks resource_type/resource_id columns",
                 })
-
-        # User activity rows correlated by resource — schema is permissive
-        # (`resource_type`/`resource_id` may not always be populated for older
-        # rows) so this query stays best-effort.
-        ua_rows = await conn.fetch(
-            """
-            SELECT id, action, resource_type, resource_id, status, details, created_at, user_id
-            FROM user_activity_logs
-            WHERE resource_type = 'letsencrypt_order' AND resource_id = $1
-            ORDER BY created_at ASC, id ASC
-            LIMIT $2
-            """,
-            str(order_id),
-            limit,
-        ) if await conn.fetchval(
-            """
-            SELECT EXISTS (
-                SELECT 1 FROM information_schema.columns
-                WHERE table_name = 'user_activity_logs' AND column_name = 'resource_id'
+        except Exception as exc:  # noqa: BLE001 — surface and continue
+            logger.exception(
+                "ACME events user_activity_logs query failed order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
             )
-            """
-        ) else []
-
-        for r in ua_rows:
-            events.append({
-                "source": "user_activity_log",
-                "id": r["id"],
-                "event_type": r["action"],
-                "severity": "info" if (r["status"] or "").lower() in ("success", "ok", "") else "warn",
-                "message": (r["details"] or "")[:500] if isinstance(r["details"], str) else "",
-                "details": r["details"] if not isinstance(r["details"], (str, type(None))) else {},
-                "correlation_id": None,
-                "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
-                    if r["created_at"] else None,
+            section_errors.append({
+                "section": "user_activity_logs",
+                "exception_type": exc.__class__.__name__,
+                "message": str(exc),
             })
 
         events.sort(key=lambda e: (e["created_at"] or "", e.get("id") or 0))
@@ -279,6 +600,10 @@ async def get_order_events(
             "order_id": order_id,
             "events": events,
             "count": len(events),
+            "meta": {
+                "correlation_id": correlation_id,
+                "errors": section_errors,
+            },
         }
     finally:
         await close_database_connection(conn)

backend/routers/agent.py

@@ -9,8 +9,15 @@ import os
 import json
 import ipaddress
 import hashlib
+import re
 # Pipeline trigger - force backend redeploy v2
 
+# v1.10.4 — a discovered keepalived.conf is stored and served to the UI, so the VRRP password is
+# masked out of the stored copy (the real value lives Fernet-encrypted in its own column). Mask
+# the WHOLE remainder of the line, mirroring vip.py's version-diff masking, so a password
+# containing whitespace cannot partially leak.
+_AUTH_PASS_MASK_RE = re.compile(r"(auth_pass\s+).*")
+
 from models import AgentCreate
 from models.agent import AgentToggle, AgentHeartbeat, AgentScriptRequest, AgentUpgradeRequest
 from database.connection import get_database_connection, close_database_connection
@@ -26,9 +33,43 @@ logger = logging.getLogger(__name__)
 # Global version storage (acts as in-memory database)
 AGENT_VERSIONS = {
     "macos": "2.1.0",  # Updated via endpoint
-    "linux": "2.0.0"
+    "linux": "2.1.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -217,7 +258,7 @@ def calculate_agent_health(status, last_seen):
         return "offline"
 
 @router.get("", summary="Get All Agents", response_description="List of all agents")
-async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None)):
+async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All Agents
     
@@ -269,9 +310,22 @@ async def get_agents(pool_id: Optional[int] = None, authorization: str = Header(
     - **haproxy_status**: Status of HAProxy service on agent's server
     - **last_seen**: Last heartbeat timestamp
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): the agent inventory (names, hostnames, IPs,
+    # pools, OS) is operator data and was previously served unauthenticated — it is
+    # also the read-back channel used in the RCE exfil PoC. Require EITHER a valid
+    # operator JWT OR a valid agent X-API-Key: deployed agents poll this endpoint
+    # (with their key, not a JWT) to read their own applied_config_version and avoid
+    # re-applying config on restart, so a JWT-only gate would break them. Checked
+    # before the try so the 401 is not swallowed by the generic handler.
+    if authorization:
+        current_user = await get_current_user_from_token(authorization)  # raises 401 on invalid JWT
+    else:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         try:
             if pool_id:
                 agents = await conn.fetch("""
@@ -729,15 +783,24 @@ async def generate_uninstall_script(platform: str, authorization: str = Header(N
     sudo ./uninstall-agent.sh
     ```
     """
+    # SECURITY (GHSA-3p5c-m5m4-mjpx): require authentication (operator JWT or agent
+    # key), consistent with generate-install-script. The uninstall script itself is
+    # generic (no secrets/topology), but an agent-management endpoint should not be
+    # anonymously reachable. Checked before the try so the 401 is not swallowed.
+    if authorization:
+        await get_current_user_from_token(authorization)
+    elif not await validate_agent_api_key(x_api_key):
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
-        # Validate platform
-        platform_lower = platform.lower()
-        if platform_lower not in ['linux', 'macos']:
-            raise HTTPException(
-                status_code=400, 
-                detail=f"Invalid platform: {platform}. Must be 'linux' or 'macos'"
-            )
-        
+        # Normalize platform to a canonical key (always 'linux' or 'macos').
+        # macOS agents register with platform 'darwin' (from `uname -s`), so the
+        # strict ['linux','macos'] check used to 400 on the UI's uninstall flow
+        # (GET /generate-uninstall-script/darwin). Reuse the same get_platform_key()
+        # helper the install-script generator uses, so darwin/osx/mac and the
+        # linux distro variants all resolve correctly. Backward-compatible:
+        # 'linux'/'macos' still map to themselves.
+        platform_lower = get_platform_key(platform)
+
         # Read uninstall script from agent_scripts directory (same as install scripts)
         import os
         script_filename = f"uninstall-agent-{platform_lower}.sh"
@@ -861,7 +924,25 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
         # Validate cluster access if agent belongs to a cluster
         if agent['cluster_id']:
             await validate_user_cluster_access(current_user['id'], agent['cluster_id'], conn)
-        
+
+        # HA/VIP (Issue #27): block deleting a node that's still a member of an active VIP.
+        # Otherwise the CASCADE would silently drop it from the VIP (breaking the one-MASTER
+        # topology with no signal) and the still-running node would keep advertising the VIP
+        # with no way to be told to tear down (review MED-3). Make the operator remove it
+        # from the VIP first — that stages a clean PENDING change they can apply.
+        try:
+            vip_member = await conn.fetchrow(
+                "SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+                "WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
+        except Exception:  # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
+            vip_member = None
+        if vip_member:
+            await close_database_connection(conn)
+            raise HTTPException(
+                status_code=409,
+                detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
+                        f"VIP on the HA / VIP page (and apply) before deleting the agent."))
+
         await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
         
         await close_database_connection(conn)
@@ -905,14 +986,24 @@ def _extract_agent_ip(heartbeat_data: AgentHeartbeat) -> Optional[str]:
     return None
 
 @router.post("/{agent_id}/heartbeat")
-async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat):
+async def agent_heartbeat(agent_id: int, heartbeat_data: AgentHeartbeat, x_api_key: Optional[str] = Header(None)):
     """Receive agent heartbeat and update status."""
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). This legacy by-ID
+    # heartbeat previously had NO auth, allowing unauthenticated state spoofing of
+    # any agent row. Deployed agents use the by-name heartbeat; a valid global
+    # agent token is now required here too. NOTE: raised BEFORE the try below so
+    # the 401 is not swallowed by the generic `except Exception` handler.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on by-id heartbeat for agent ID {agent_id}")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         await conn.execute("""
-            UPDATE agents 
-            SET status = 'online', 
+            UPDATE agents
+            SET status = 'online',
                 last_seen = CURRENT_TIMESTAMP,
                 hostname = COALESCE($2, hostname),
                 haproxy_status = COALESCE($3, haproxy_status),
@@ -1035,6 +1126,8 @@ async def agent_config_applied_notification(agent_name: str, notification_data:
         await close_database_connection(conn)
         return {"status": "ok", "message": "Config applied notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config applied notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1130,6 +1223,8 @@ async def agent_config_validation_failed(agent_name: str, notification_data: dic
         
         return {"status": "ok", "message": "Validation error notification received"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process validation error notification from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1419,6 +1514,8 @@ async def agent_config_sync(agent_name: str, sync_data: dict, x_api_key: Optiona
         logger.info(f"CONFIG SYNC: Agent '{agent_name}' synced {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers with database")
         return {"status": "ok", "message": f"Config synced - {len(active_backends)} backends, {len(active_frontends)} frontends, {len(active_servers)} servers processed"}
         
+    except HTTPException:
+        raise  # let auth 401/403 propagate (do not turn it into a 200 error body)
     except Exception as e:
         logger.error(f"Failed to process config sync from agent '{agent_name}': {e}")
         return {"status": "error", "message": str(e)}
@@ -1436,47 +1533,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1493,21 +1576,25 @@ async def agent_heartbeat_by_name(
         logger.error(f"Unexpected error processing heartbeat: {e}")
         raise HTTPException(status_code=500, detail="Internal server error")
     
+    # Agent authentication is MANDATORY (GHSA-3p5c-m5m4-mjpx). A valid global agent
+    # token is required to heartbeat OR auto-register. Deployed agents always send
+    # X-API-Key; an absent/invalid key is an unauthenticated caller. This is done
+    # OUTSIDE the processing try below (whose generic `except Exception` would
+    # otherwise convert the 401 into a 500), and before opening a DB connection
+    # (validate_agent_api_key(None) needs no DB). Closes keyless heartbeat spoofing
+    # and keyless rogue-agent auto-registration (the `elif not agent` keyless path
+    # below is now unreachable, since agent_auth is guaranteed truthy past here).
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key on heartbeat for agent '{heartbeat_data.name}'")
+        raise HTTPException(status_code=401, detail="Authentication required")
+
     # Continue with normal heartbeat processing
     try:
-        # Validate agent API key for security
-        from auth_middleware import validate_agent_api_key
-        agent_auth = await validate_agent_api_key(x_api_key)
-        
         conn = await get_database_connection()
         agent_name = heartbeat_data.name
 
-        # If API key provided, validate it exists but allow placeholder agent updates
-        if x_api_key and not agent_auth:
-            await close_database_connection(conn)
-            logger.warning(f"Invalid API key provided by agent '{agent_name}'")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-
         agent = await conn.fetchrow("SELECT id, pool_id, api_key FROM agents WHERE name = $1", agent_name)
         
         # If agent exists and is using a different API key, update the token association
@@ -1656,9 +1743,13 @@ async def agent_heartbeat_by_name(
             """, x_api_key)
 
         # Check if agent was in upgrading status and version has changed
-        current_agent_status = await conn.fetchval("SELECT status FROM agents WHERE id = $1", agent_id)
-        current_agent_version = await conn.fetchval("SELECT version FROM agents WHERE id = $1", agent_id)
-        current_upgrade_status = await conn.fetchval("SELECT upgrade_status FROM agents WHERE id = $1", agent_id)
+        # (v1.8.6: one round-trip instead of three; row is None exactly when the
+        # per-column fetchvals would each have returned None)
+        current_agent_row = await conn.fetchrow(
+            "SELECT status, version, upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_status = current_agent_row['status'] if current_agent_row else None
+        current_agent_version = current_agent_row['version'] if current_agent_row else None
+        current_upgrade_status = current_agent_row['upgrade_status'] if current_agent_row else None
         
         # Determine new status - preserve upgrading status unless version actually changed
         new_status = current_agent_status or 'online'
@@ -1749,9 +1840,14 @@ async def agent_heartbeat_by_name(
             heartbeat_data.operating_system, heartbeat_data.kernel_version,
             heartbeat_data.uptime, heartbeat_data.cpu_count, heartbeat_data.memory_total,
             heartbeat_data.disk_space, 
-            # Convert lists to JSON for JSONB columns
-            heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str) else json.dumps(heartbeat_data.network_interfaces or []),
-            heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str) else json.dumps(heartbeat_data.capabilities or []),
+            # Convert lists to JSON for JSONB columns. Don't WIPE network_interfaces/capabilities
+            # when a heartbeat omits them: send NULL so COALESCE keeps the existing value (a bare
+            # `or []` would store "[]" and erase the reported NICs / keepalived_management on every
+            # daemon heartbeat that doesn't include them — issue #27 corporate test).
+            (heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str)
+             else (json.dumps(heartbeat_data.network_interfaces) if heartbeat_data.network_interfaces else None)),
+            (heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str)
+             else (json.dumps(heartbeat_data.capabilities) if heartbeat_data.capabilities else None)),
             agent_ip, heartbeat_data.haproxy_status, heartbeat_data.haproxy_version,
             heartbeat_data.applied_config_version, new_status, update_applied_version,
             heartbeat_data.keepalive_state, heartbeat_data.keepalive_ip)
@@ -1907,9 +2003,19 @@ async def agent_heartbeat_by_name(
 @router.get("/{agent_name}/config")
 async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get HAProxy configuration for specific agent"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked BEFORE any
+    # DB work and before the existence check, so an unauthenticated caller learns
+    # neither the full haproxy.cfg nor whether the agent exists. Raised before the
+    # try so it is not swallowed by the generic handler; validate_agent_api_key(None)
+    # returns None without touching the DB.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' config fetch")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent info first to check pool
         # CRITICAL: Include cluster's haproxy_bin_path, haproxy_config_path, stats_socket_path
         # These are needed for dynamic validation - cluster admin can change paths without reinstalling agent
@@ -1921,30 +2027,19 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' config fetch")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching config using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Config fetch authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching config using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Config fetch authorized for agent '{agent_name}'")
+
         if not agent_info['enabled']:
             await close_database_connection(conn)
             return {
@@ -2035,9 +2130,18 @@ async def get_agent_config(agent_name: str, x_api_key: Optional[str] = Header(No
 @router.get("/{agent_name}/ssl-certificates")
 async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = None, x_api_key: Optional[str] = Header(None)):
     """Get SSL certificates for specific agent's cluster"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). This response
+    # returns SSL private_key_content, so authentication is checked BEFORE any DB
+    # work and before the existence check. Raised before the try so the 401 is not
+    # swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' SSL certificates")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Get agent and cluster info first
         agent_info = await conn.fetchrow("""
             SELECT a.id, a.name, a.pool_id, hc.id as cluster_id, hc.name as cluster_name,
@@ -2046,29 +2150,18 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
         """, agent_name)
-        
+
         if not agent_info:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' SSL certificates")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching SSL certificates using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"SSL fetch authorized for agent '{agent_name}'")
         
         if not agent_info['enabled']:
             await close_database_connection(conn)
@@ -2180,13 +2273,287 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             logger.info(f"SSL INCREMENTAL: No certificates updated since {since}")
             
         return response_data
-        
+
     except HTTPException:
         raise
     except Exception as e:
         logger.error(f"SSL certificates retrieval failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
+
+@router.get("/{agent_name}/keepalived-config")
+async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
+    teardown / no-op signal.
+
+    Snapshot-based (T-1): keys off vip_instances.is_active + the member's
+    applied_config_content, NOT the live last_config_status — so a PENDING edit never
+    flips a running member to not_configured (no mid-edit teardown). Auth is MANDATORY (a
+    valid agent key is required), but — like the /config and /ssl-certificates endpoints —
+    the agent API key is a SHARED/global install token, so the config is resolved by the
+    requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
+    (a hard 403 would break every VIP member whose name isn't the one row the shared token
+    resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
+    the agent stays inert; a node with no membership row always gets not_configured.
+    """
+    conn = None
+    try:
+        # Auth FIRST and MANDATORY: a valid agent key is REQUIRED (the response carries the
+        # VRRP secret). The token is shared/global, so resolve by agent_name and only LOG a
+        # name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
+        # unauthenticated caller can't probe which agent names exist.
+        from auth_middleware import validate_agent_api_key
+        if not x_api_key:
+            raise HTTPException(status_code=401, detail="Agent API key required")
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        # The agent API key is a SHARED/global install token (many agent rows per token),
+        # so validate_agent_api_key resolves it to one arbitrary agent for that token. Resolve
+        # the keepalived config strictly by the requested agent_name and treat a token/name
+        # mismatch as an advisory audit log — exactly like the /config and /ssl-certificates
+        # endpoints. (A hard 403 here would reject every VIP member whose name isn't the one
+        # row the shared token happens to return, so the VIP could never converge — review HIGH-1.)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        # Resolve the agent + its cluster's keepalived.conf path (cluster-driven, like the
+        # HAProxy paths). config_path is returned in EVERY response so the agent knows where
+        # to write/own-marker-check even on not_configured/teardown.
+        agent = await conn.fetchrow("""
+            SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
+                   hc.keepalived_config_path
+            FROM agents a
+            LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
+            WHERE a.name = $1
+        """, agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+        config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
+        if not agent['enabled']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        row = await conn.fetchrow("""
+            SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
+                   v.purge_on_teardown,
+                   m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
+            FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
+            WHERE m.agent_id = $1
+            -- Active VIP first (an agent has at most one). With NO active VIP, pick the most
+            -- RECENTLY updated inactive membership so a teardown reflects the latest delete
+            -- (incl. its purge flag) — not a stale older VIP the node was once part of.
+            ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
+            LIMIT 1
+        """, agent['id'])
+
+        if not row:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+        if not row['is_active']:
+            # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
+            # the agent still only purges on nodes where IT installed keepalived (install marker).
+            return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
+                    "config_path": config_path, "keepalived": None,
+                    "purge": bool(row['purge_on_teardown'])}
+        if not row['applied_config_content']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        from services.keepalived_config import build_haproxy_check_script
+        check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
+        return {
+            "agent_name": agent_name,
+            "status": "available",
+            "config_path": config_path,
+            "keepalived": {
+                "desired_state": "enabled",
+                "install_if_missing": True,
+                "vip_id": row['vip_id'],
+                "vip_name": row['vip_name'],
+                "config_content": row['applied_config_content'],
+                "config_hash": row['applied_config_hash'],
+                "check_script": check_script,
+                # v1.10.4 adoption handoff. The agent refuses to overwrite a keepalived.conf
+                # without our ownership marker — the guard that protects a hand-maintained
+                # setup. Adoption does not weaken it: it authorises exactly ONE takeover, of
+                # exactly the file we analysed, by pinning its hash. If the file changed since
+                # adoption the hashes differ and the agent keeps refusing, so an edit made
+                # between adoption and Apply can never be silently overwritten.
+                "allow_takeover": bool(row['takeover_expected_hash']),
+                "takeover_expected_hash": row['takeover_expected_hash'],
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
+        # Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
+        return {"agent_name": agent_name, "status": "not_configured",
+                "config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-status")
+async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — agent reports the outcome of a keepalived deploy/teardown.
+
+    Auth mirrors config-applied's post-Bulgu-#75 guard (reject a MISSING key — never the
+    `and` short-circuit that accepted no-key requests). The token is a shared/global install
+    token, so the status is recorded strictly for the requested agent_name and a token/name
+    mismatch is an advisory audit log (like /config-applied), not a 403 — review HIGH-1.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived status using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        vip_id = status_data.get("vip_id")
+        state = (status_data.get("state") or "").strip()[:24]
+        config_hash = (status_data.get("config_hash") or "")[:64]
+        message = status_data.get("message")
+        # v1.10.9 — retire the adoption takeover authorisation once the node CONFIRMS it is
+        # running our rendered config. `takeover_expected_hash` is the permission to overwrite a
+        # keepalived.conf that lacks our ownership marker; it was written at adoption and never
+        # cleared, so it stayed valid indefinitely and "one-shot" was only true in the sense of
+        # "for exactly that file content". Clearing it the moment the member acks OUR hash makes
+        # the claim real: if the file is replaced by hand afterwards the agent refuses and reports
+        # "externally managed", which is the visible behaviour an operator should get.
+        #
+        # Gated on the acked hash MATCHING applied_config_hash, so a partial or failed deploy
+        # never drops the authorisation and leaves the VIP unable to converge.
+        _retire_takeover = ("takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL "
+                            "AND {p} = applied_config_hash THEN NULL ELSE takeover_expected_hash END")
+        if vip_id is None:
+            # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
+            await conn.execute(f"""
+                UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
+                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$4")}
+                WHERE agent_id=$1
+            """, agent['id'], state, message, config_hash)
+        else:
+            await conn.execute(f"""
+                UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
+                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
+                    {_retire_takeover.format(p="$5")}
+                WHERE agent_id=$1 AND vip_id=$2
+            """, agent['id'], int(vip_id), state, message, config_hash)
+        return {"status": "ok"}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-status update failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-status update failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-discovery")
+async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key: Optional[str] = Header(None)):
+    """v1.10.4 — the agent reports a keepalived.conf it found on the node but does NOT own.
+
+    This is what makes adopting a hand-maintained VIP possible: the heartbeat only carries the
+    VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+    fields, so the file itself has to be read. Read-only on the agent side — reporting never
+    changes anything on the node.
+
+    Auth mirrors /keepalived-status: a MISSING key is rejected outright, and because the token
+    is a shared install token a name mismatch is an advisory audit log rather than a 403.
+
+    SECRETS: the reported content may contain the VRRP `auth_pass`. It is split immediately —
+    the password is Fernet-encrypted into its own column and the stored copy of the file has it
+    masked, so nothing readable through the API or a DB dump carries it in cleartext. The
+    parse result is never logged.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived discovery using API key "
+                        f"from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        config_path = (payload.get("config_path") or "/etc/keepalived/keepalived.conf")[:500]
+        exists = bool(payload.get("exists"))
+        if not exists:
+            # The file is gone (keepalived removed, or we adopted and now own it) — drop the row
+            # so the UI stops offering a stale candidate.
+            await conn.execute("DELETE FROM vip_discoveries WHERE agent_id = $1", agent['id'])
+            return {"status": "cleared"}
+
+        content = payload.get("config_content") or ""
+        if len(content) > 256_000:
+            raise HTTPException(status_code=413, detail="keepalived.conf too large to analyse")
+        is_managed = bool(payload.get("is_managed"))
+        config_hash = hashlib.md5(content.encode("utf-8", "replace")).hexdigest()
+
+        from services.keepalived_parser import analyse_keepalived_conf, KeepalivedParseError
+        from services.keepalived_config import encrypt_vrrp_secret
+
+        parse_error = None
+        analysis = None
+        auth_enc = None
+        try:
+            analysis = analyse_keepalived_conf(content)
+            # Split the secret out of everything we persist or serve.
+            for cand in analysis.get("candidates", []):
+                secret = (cand.get("vip") or {}).pop("auth_pass", None)
+                cand["vip"]["has_auth_pass"] = bool(secret)
+                if secret and auth_enc is None:
+                    auth_enc = encrypt_vrrp_secret(secret)
+        except KeepalivedParseError as exc:
+            parse_error = str(exc)[:500]
+        except Exception as exc:  # noqa: BLE001 — a malformed file must not 500 the agent loop
+            parse_error = f"could not analyse the config ({type(exc).__name__})"
+
+        masked = _AUTH_PASS_MASK_RE.sub(r"\1********", content)
+        await conn.execute("""
+            INSERT INTO vip_discoveries
+                (agent_id, config_path, config_hash, is_managed, raw_config_masked,
+                 auth_pass_encrypted, analysis, parse_error, reported_at)
+            VALUES ($1,$2,$3,$4,$5,$6,$7,$8,CURRENT_TIMESTAMP)
+            ON CONFLICT (agent_id) DO UPDATE SET
+                config_path = EXCLUDED.config_path,
+                config_hash = EXCLUDED.config_hash,
+                is_managed = EXCLUDED.is_managed,
+                raw_config_masked = EXCLUDED.raw_config_masked,
+                auth_pass_encrypted = EXCLUDED.auth_pass_encrypted,
+                analysis = EXCLUDED.analysis,
+                parse_error = EXCLUDED.parse_error,
+                reported_at = CURRENT_TIMESTAMP
+        """, agent['id'], config_path, config_hash, is_managed, masked, auth_enc,
+            json.dumps(analysis) if analysis is not None else None, parse_error)
+        return {"status": "recorded", "config_hash": config_hash}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-discovery failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-discovery failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
 @router.get("/script-version")
 async def get_latest_script_version(platform: str = "macos"):
     """Get the latest available agent script version for specified platform"""
@@ -2234,16 +2601,24 @@ async def get_latest_script_version(platform: str = "macos"):
 @router.get("/{agent_name}/upgrade-status")
 async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = Header(None)):
     """Get agent upgrade status - used by agents to check if they should upgrade"""
+    # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Checked before any
+    # DB work; deployed agents always send X-API-Key. Raised before the try so the
+    # 401 is not swallowed; validate_agent_api_key(None) returns None without a DB hit.
+    from auth_middleware import validate_agent_api_key
+    agent_auth = await validate_agent_api_key(x_api_key)
+    if not agent_auth:
+        logger.warning(f"Missing/invalid API key for agent '{agent_name}' upgrade status")
+        raise HTTPException(status_code=401, detail="Authentication required")
     try:
         conn = await get_database_connection()
-        
+
         # Check if agent has upgrade pending (include platform and pool for validation)
         agent = await conn.fetchrow("""
             SELECT status, version as current_version, platform, pool_id
-            FROM agents 
+            FROM agents
             WHERE name = $1
         """, agent_name)
-        
+
         if not agent:
             await close_database_connection(conn)
             return {
@@ -2251,26 +2626,15 @@ async def get_agent_upgrade_status(agent_name: str, x_api_key: Optional[str] = H
                 "target_version": "",
                 "message": "Agent not found"
             }
-        
-        # Validate agent API key
-        # API key is global - can be used for multiple agents
-        if x_api_key:
-            from auth_middleware import validate_agent_api_key
-            agent_auth = await validate_agent_api_key(x_api_key)
-            
-            if not agent_auth:
-                await close_database_connection(conn)
-                logger.warning(f"Invalid API key provided for agent '{agent_name}' upgrade status")
-                raise HTTPException(status_code=401, detail="Invalid API key")
-            
-            # Log which agent's API key was used (for audit trail)
-            if agent_auth['name'] == agent_name:
-                logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
-            else:
-                logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
-            
-            logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
-        
+
+        # Log which agent's API key was used (for audit trail)
+        if agent_auth['name'] == agent_name:
+            logger.debug(f"Agent '{agent_name}' checking upgrade status using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' checking upgrade status using API key from agent '{agent_auth['name']}'")
+
+        logger.debug(f"Upgrade status check authorized for agent '{agent_name}'")
+
         await close_database_connection(conn)
         
         # Agent should upgrade if status is 'upgrading'
@@ -2704,7 +3068,17 @@ async def get_agent_script_template(platform: str, authorization: str = Header(N
     """Get the latest script template for specified platform from database"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): the raw install/upgrade script is a
+        # version-management surface. Gate reads with agents.version too, matching
+        # the write path above (operator/security_admin/super_admin retain access).
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         conn = await get_database_connection()
         
         # Get latest script template for platform
@@ -2755,7 +3129,19 @@ async def save_agent_script_template(platform: str, template_data: dict, authori
     """Save updated script template to database using shared helper function"""
     try:
         current_user = await get_current_user_from_token(authorization)
-        
+
+        # SECURITY (GHSA-7rhv-c5pc-69r8): agent script templates become the
+        # install/self-upgrade script executed as root on HAProxy nodes. A poisoned
+        # template is RCE. Authentication alone is NOT enough — require the same
+        # agents.version permission as POST /versions; otherwise any JWT holder
+        # (including viewer) could overwrite the active script.
+        has_permission = await check_user_permission(current_user["id"], "agents", "version")
+        if not has_permission:
+            raise HTTPException(
+                status_code=403,
+                detail="Insufficient permissions: agents.version required"
+            )
+
         script_content = template_data.get('script_content', '')
         version = template_data.get('version', '')
         

backend/routers/auth.py

@@ -9,14 +9,48 @@ from datetime import datetime, timedelta
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
 from models.user import LoginRequest, User, UserCreate, UserUpdate, UserPasswordUpdate
+from models.mfa import MfaVerifyRequest
 from utils.activity_log import log_user_activity
 from auth_middleware import get_current_user_from_token
+from services import mfa_service
 
 # Rate limiting temporarily disabled
 
 router = APIRouter(prefix="/api/auth", tags=["Authentication"])
 logger = logging.getLogger(__name__)
 
+MFA_PENDING_TTL_SECONDS = 300  # 5 minutes — pre-verification challenge lifetime
+MFA_PENDING_MAX_ATTEMPTS = 5  # invalidate token after this many wrong codes
+
+
+async def _fetch_mfa_state(conn, user_id: int):
+    """Return (mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step) or None
+    when the MFA columns aren't yet present (pre-migration deploys).
+    """
+    try:
+        return await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+    except Exception as exc:
+        logger.warning(f"MFA columns not available (assuming disabled): {exc}")
+        return None
+
+
+async def _cleanup_expired_pending_logins(conn, user_id: int) -> None:
+    """Lazy cleanup of expired pending MFA challenges for this user."""
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_logins WHERE user_id = $1 AND expires_at < NOW()",
+            user_id,
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-login cleanup skipped: {exc}")
+
 # Security scheme
 security = HTTPBearer()
 
@@ -96,7 +130,7 @@ async def login(login_request: LoginRequest, request: Request):
                 SELECT id, username, email, password_hash, is_active, role, 
                        created_at, updated_at, last_login_at
                 FROM users 
-                WHERE username = $1
+                WHERE username = $1 AND is_active = TRUE
             """, login_request.username)
         except Exception as schema_error:
             logger.warning(f"Schema error, trying fallback query: {schema_error}")
@@ -106,7 +140,7 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login_at
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
             except Exception as column_error:
                 logger.warning(f"last_login_at column error, trying last_login: {column_error}")
@@ -115,26 +149,67 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
         
         if not user:
+            # Covers both "no such user" and "soft-deleted (is_active=FALSE)".
+            # We deliberately return the same generic 401 in either case to
+            # avoid leaking whether an account exists (account enumeration
+            # prevention). Soft-deleted rows are filtered out by the
+            # `AND is_active = TRUE` predicate above.
             await close_database_connection(conn)
             logger.warning(f"Failed login attempt for username: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
-        if not user['is_active']:
-            await close_database_connection(conn)
-            logger.warning(f"Login attempt for inactive account: {login_request.username}")
-            raise HTTPException(status_code=401, detail="Account is deactivated")
-        
+
         # Verify password
         import bcrypt
         if not bcrypt.checkpw(login_request.password.encode('utf-8'), user['password_hash'].encode('utf-8')):
             await close_database_connection(conn)
             logger.warning(f"Wrong password for user: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
+
+        # Issue #18 — MFA branch (v1.6.0): if the user opted in, defer JWT mint and
+        # last_login update until /api/auth/login/mfa-verify completes.
+        mfa_state = await _fetch_mfa_state(conn, user['id'])
+        if mfa_state and mfa_state.get('mfa_enabled'):
+            await _cleanup_expired_pending_logins(conn, user['id'])
+            challenge_token = mfa_service.generate_challenge_token()
+            try:
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_logins (user_id, challenge_token, expires_at, ip_address)
+                    VALUES ($1, $2, NOW() + ($3 || ' seconds')::interval, $4)
+                    """,
+                    user['id'],
+                    challenge_token,
+                    str(MFA_PENDING_TTL_SECONDS),
+                    str(request.client.host) if request.client else None,
+                )
+            except Exception as exc:
+                await close_database_connection(conn)
+                logger.error(f"Failed to create MFA pending login: {exc}")
+                raise HTTPException(status_code=500, detail="MFA challenge creation failed")
+
+            await close_database_connection(conn)
+
+            await log_user_activity(
+                user_id=user['id'],
+                action='mfa.login.challenge_issued',
+                resource_type='mfa',
+                resource_id=str(user['id']),
+                details={'login_method': 'username_password'},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get('user-agent'),
+            )
+
+            return {
+                "mfa_required": True,
+                "mfa_token": challenge_token,
+                "methods": ["totp", "backup"],
+                "expires_in": MFA_PENDING_TTL_SECONDS,
+            }
+
         # Update last login (try different column names)
         try:
             await conn.execute("""
@@ -242,6 +317,348 @@ async def login(login_request: LoginRequest, request: Request):
         logger.error(f"Login error: {e}")
         raise HTTPException(status_code=500, detail="Login failed")
 
+@router.post(
+    "/login/mfa-verify",
+    summary="MFA Verification (Step 2 of Login)",
+    response_description="JWT access token after successful TOTP/backup verification",
+)
+async def login_mfa_verify(payload: MfaVerifyRequest, request: Request):
+    """
+    # MFA Verification — Step 2 of the two-step login flow
+
+    Submit a 6-digit TOTP code OR an 8-character backup code (with optional dash)
+    together with the ``mfa_token`` returned by ``POST /api/auth/login`` for an
+    MFA-enabled account. On success, returns the same response shape as a
+    non-MFA login (Branch A).
+
+    ## Request Body
+    - **mfa_token**: 64-char challenge token from /login response
+    - **code**: 6 digits (TOTP) or `XXXX-YYYY` (backup)
+
+    ## Error Responses
+    - **401**: Invalid code (attempts counter increments)
+    - **410**: Challenge expired or invalidated (too many wrong attempts)
+    """
+    ip_address = str(request.client.host) if request.client else None
+    user_agent = request.headers.get('user-agent')
+
+    # Outcome captured from the transactional block so we can do JWT mint /
+    # activity logging AFTER commit (no side effects on rollback).
+    success_payload: Optional[dict] = None
+    failure: Optional[dict] = None  # { user_id, attempts, invalidated, reason, http_status, detail }
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Round 1 audit fix — wrap the whole verify+update in a single
+        # transaction with row-level locks (FOR UPDATE) so two concurrent
+        # /mfa-verify calls cannot both consume the same TOTP step or the
+        # same pending challenge. We never raise inside the transaction once
+        # we've started mutating the pending row (would rollback the mark);
+        # instead we capture `failure` and raise after commit.
+        async with conn.transaction():
+            pending = await conn.fetchrow(
+                """
+                SELECT id, user_id, attempts, expires_at, used_at
+                  FROM mfa_pending_logins
+                 WHERE challenge_token = $1
+                 FOR UPDATE
+                """,
+                payload.mfa_token,
+            )
+
+            if not pending:
+                failure = {
+                    'user_id': None,
+                    'attempts': 0,
+                    'invalidated': True,
+                    'reason': 'challenge_not_found',
+                    'http_status': 410,
+                    'detail': 'MFA challenge not found or expired',
+                }
+            elif pending['used_at'] is not None:
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_already_used',
+                    'http_status': 410,
+                    'detail': 'MFA challenge already used',
+                }
+            elif pending['expires_at'] and pending['expires_at'] < datetime.utcnow():
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_expired',
+                    'http_status': 410,
+                    'detail': 'MFA challenge expired',
+                }
+            elif pending['attempts'] >= MFA_PENDING_MAX_ATTEMPTS:
+                await conn.execute(
+                    "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                    pending['id'],
+                )
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'too_many_attempts_pre_check',
+                    'http_status': 410,
+                    'detail': 'MFA challenge invalidated (too many attempts)',
+                }
+
+            if failure is None:
+                # Lock the user row so the atomic TOTP-step bump cannot race a
+                # parallel verify on a different pending challenge for the
+                # same account.
+                user_row = await conn.fetchrow(
+                    """
+                    SELECT id, username, email, role, is_active,
+                           created_at, updated_at, last_login_at,
+                           mfa_secret_encrypted, mfa_last_used_totp_step
+                      FROM users
+                     WHERE id = $1
+                     FOR UPDATE
+                    """,
+                    pending['user_id'],
+                )
+                if not user_row or not user_row['is_active']:
+                    failure = {
+                        'user_id': pending['user_id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': False,
+                        'reason': 'user_unavailable',
+                        'http_status': 401,
+                        'detail': 'User not available',
+                    }
+                elif not user_row['mfa_secret_encrypted']:
+                    failure = {
+                        'user_id': user_row['id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': True,
+                        'reason': 'mfa_not_configured',
+                        'http_status': 410,
+                        'detail': 'MFA not configured for this user',
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(user_row['mfa_secret_encrypted'])
+                    verified_method: Optional[str] = None
+                    codes_remaining: Optional[int] = None
+
+                    if secret_plain:
+                        ok, step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, user_row['mfa_last_used_totp_step']
+                        )
+                        if ok:
+                            # Atomic step bump — refuse if another request already
+                            # consumed this (or a newer) TOTP step.
+                            bumped = await conn.fetchval(
+                                """
+                                UPDATE users
+                                   SET mfa_last_used_totp_step = $1,
+                                       mfa_last_used_at = NOW()
+                                 WHERE id = $2
+                                   AND (mfa_last_used_totp_step IS NULL
+                                        OR mfa_last_used_totp_step < $1)
+                                 RETURNING id
+                                """,
+                                step,
+                                user_row['id'],
+                            )
+                            if bumped:
+                                verified_method = 'totp'
+
+                    if verified_method is None:
+                        # Backup codes — atomic single-use consumption.
+                        rows = await conn.fetch(
+                            """
+                            SELECT id, code_hash FROM mfa_backup_codes
+                             WHERE user_id = $1 AND used_at IS NULL
+                            """,
+                            user_row['id'],
+                        )
+                        for row in rows:
+                            if await mfa_service.check_backup_code(payload.code, row['code_hash']):
+                                consumed_id = await conn.fetchval(
+                                    """
+                                    UPDATE mfa_backup_codes
+                                       SET used_at = NOW()
+                                     WHERE id = $1 AND used_at IS NULL
+                                     RETURNING id
+                                    """,
+                                    row['id'],
+                                )
+                                if consumed_id:
+                                    verified_method = 'backup'
+                                    codes_remaining = await conn.fetchval(
+                                        "SELECT COUNT(*) FROM mfa_backup_codes WHERE user_id = $1 AND used_at IS NULL",
+                                        user_row['id'],
+                                    )
+                                break
+
+                    if verified_method is None:
+                        new_attempts = pending['attempts'] + 1
+                        invalidated = new_attempts >= MFA_PENDING_MAX_ATTEMPTS
+                        await conn.execute(
+                            """
+                            UPDATE mfa_pending_logins
+                               SET attempts = $1,
+                                   used_at = CASE WHEN $2 THEN NOW() ELSE used_at END
+                             WHERE id = $3
+                            """,
+                            new_attempts,
+                            invalidated,
+                            pending['id'],
+                        )
+                        failure = {
+                            'user_id': user_row['id'],
+                            'attempts': new_attempts,
+                            'invalidated': invalidated,
+                            'reason': 'invalid_code',
+                            'http_status': 410 if invalidated else 401,
+                            'detail': 'MFA challenge invalidated (too many attempts)'
+                            if invalidated else 'Invalid MFA code',
+                        }
+                    else:
+                        # Verified — finalize state inside the transaction so a
+                        # concurrent verify sees used_at on retry.
+                        await conn.execute(
+                            "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                            pending['id'],
+                        )
+                        if verified_method != 'totp':
+                            await conn.execute(
+                                "UPDATE users SET mfa_last_used_at = NOW() WHERE id = $1",
+                                user_row['id'],
+                            )
+                        try:
+                            await conn.execute(
+                                "UPDATE users SET last_login_at = CURRENT_TIMESTAMP WHERE id = $1",
+                                user_row['id'],
+                            )
+                        except Exception as exc:
+                            logger.warning(f"last_login_at update failed (continuing): {exc}")
+
+                        user_roles = await conn.fetch(
+                            """
+                            SELECT r.id, r.name, r.display_name, r.permissions
+                              FROM user_roles ur
+                              JOIN roles r ON ur.role_id = r.id
+                             WHERE ur.user_id = $1 AND ur.is_active = TRUE AND r.is_active = TRUE
+                            """,
+                            user_row['id'],
+                        )
+
+                        permissions: dict = {}
+                        roles_list: list = []
+                        for role_row in user_roles:
+                            roles_list.append({
+                                'id': role_row['id'],
+                                'name': role_row['name'],
+                                'display_name': role_row['display_name'],
+                            })
+                            role_permissions = role_row['permissions']
+                            if isinstance(role_permissions, str):
+                                import json
+                                role_permissions = json.loads(role_permissions)
+                            if role_permissions:
+                                for perm in role_permissions:
+                                    if '.' in perm:
+                                        resource, action = perm.split('.', 1)
+                                        if resource not in permissions:
+                                            permissions[resource] = {}
+                                        permissions[resource][action] = True
+
+                        success_payload = {
+                            'user': dict(user_row),
+                            'roles_list': roles_list,
+                            'permissions': permissions,
+                            'method': verified_method,
+                            'codes_remaining': codes_remaining,
+                        }
+
+        # ------------------------------------------------------------------
+        # Transaction has committed. Side-effects (JWT mint, audit log) below.
+        # ------------------------------------------------------------------
+        await close_database_connection(conn)
+        conn = None
+
+        if failure is not None:
+            if failure['user_id'] is not None:
+                await log_user_activity(
+                    user_id=failure['user_id'],
+                    action='mfa.login.failed',
+                    resource_type='mfa',
+                    resource_id=str(failure['user_id']),
+                    details={
+                        'reason': failure['reason'],
+                        'attempts': failure['attempts'],
+                        'invalidated': failure['invalidated'],
+                    },
+                    ip_address=ip_address,
+                    user_agent=user_agent,
+                )
+            raise HTTPException(status_code=failure['http_status'], detail=failure['detail'])
+
+        # Success path
+        assert success_payload is not None  # for type checkers; transaction guarantees this
+        user_row = success_payload['user']
+
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        token_payload = {
+            "user_id": user_row['id'],
+            "username": user_row['username'],
+            "email": user_row['email'],
+            "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+            "exp": datetime.utcnow() + timedelta(hours=24),
+        }
+        token = jwt.encode(token_payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+        await log_user_activity(
+            user_id=user_row['id'],
+            action='mfa.login.success',
+            resource_type='mfa',
+            resource_id=str(user_row['id']),
+            details={
+                'method': success_payload['method'],
+                'codes_remaining': success_payload['codes_remaining']
+                if success_payload['method'] == 'backup' else None,
+            },
+            ip_address=ip_address,
+            user_agent=user_agent,
+        )
+
+        return {
+            "access_token": token,
+            "token_type": "bearer",
+            "expires_in": 86400,
+            "user": {
+                "id": user_row['id'],
+                "username": user_row['username'],
+                "email": user_row['email'],
+                "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+                "is_active": user_row['is_active'],
+                "created_at": user_row['created_at'].isoformat() if user_row.get('created_at') else None,
+                "last_login_at": datetime.utcnow().isoformat(),
+            },
+            "roles": success_payload['roles_list'],
+            "permissions": success_payload['permissions'],
+        }
+
+    except HTTPException:
+        raise
+    except Exception as exc:
+        logger.error(f"MFA verify error: {exc}")
+        raise HTTPException(status_code=500, detail="MFA verification failed")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
 @router.post("/logout", summary="User Logout", response_description="Logout confirmation")
 async def logout(request: Request, authorization: str = Header(None)):
     """

backend/routers/backend.py

@@ -333,31 +333,38 @@ async def get_backends(
                 """)
         
         result = []
+        # Issue #24: servers must honor include_inactive exactly like the backend
+        # queries above. Pre-fix these sub-queries hardcoded `is_active = TRUE`
+        # (added in f34a6ee to hide soft-deleted entities), so a server toggled
+        # OFF (is_active=false) vanished from the UI with no way to reactivate it.
+        # Default callers (include_inactive=false) keep the is_active filter →
+        # byte-identical behavior; include_inactive=true now also returns inactive
+        # (disabled / soft-deleted) servers. last_config_status is selected so the
+        # frontend can tell DISABLED (re-enableable) from DELETION (pending delete).
+        server_active_filter = "" if include_inactive else "AND is_active = TRUE"
         for backend in backends:
-            # Get servers for this backend with cluster_id (ONLY show active servers)
-            # CRITICAL FIX: Add is_active = TRUE filter to prevent soft-deleted servers from appearing
             if cluster_id:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND cluster_id = $2 AND is_active = TRUE ORDER BY server_name
+                    FROM backend_servers
+                    WHERE backend_name = $1 AND cluster_id = $2 {server_active_filter} ORDER BY server_name
                 """, backend["name"], cluster_id)
             else:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND is_active = TRUE ORDER BY server_name
-                """, backend["name"]) 
+                    FROM backend_servers
+                    WHERE backend_name = $1 {server_active_filter} ORDER BY server_name
+                """, backend["name"])
             
             # Prepare server list with real-time HAProxy status from agents
             server_list = []
@@ -413,6 +420,7 @@ async def get_backends(
                     "fall": s.get("fall"),
                     "rise": s.get("rise"),
                     "is_active": s["is_active"],
+                    "last_config_status": s.get("last_config_status") or "APPLIED",  # Issue #24: lets UI distinguish DISABLED (re-enableable) from DELETION
                     "status": server_status,
                     "status_age_minutes": status_age_minutes,
                     "last_status_update": s.get("haproxy_status_updated_at").isoformat().replace('+00:00', 'Z') if s.get("haproxy_status_updated_at") else None,
@@ -1909,12 +1917,12 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
         conn = await get_database_connection()
         
-        # Get server info
+        # Get server info. Issue #24: fetch the FULL row (not just 5 columns) so we
+        # can snapshot the pre-toggle state for reject-rollback (see config version below).
         server = await conn.fetchrow("""
-            SELECT id, server_name, backend_name, is_active, cluster_id
-            FROM backend_servers WHERE id = $1
+            SELECT * FROM backend_servers WHERE id = $1
         """, server_id)
-        
+
         if not server:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail="Server not found")
@@ -1975,22 +1983,45 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
                 
                 # Generate new HAProxy config (after database commit)
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
-                
+
                 # Create new config version
                 config_hash = hashlib.sha256(config_content.encode()).hexdigest()
                 version_name = f"server-{server_id}-toggle-{int(time.time())}"
-                
+
                 # Get system admin user ID for created_by
                 conn2 = await get_database_connection()
                 admin_user_id = await conn2.fetchval("SELECT id FROM users WHERE username = 'admin' LIMIT 1") or 1
-                
+
+                # Issue #24: persist an entity snapshot so a Reject of this toggle
+                # rolls back is_active to its pre-toggle value. Pre-fix the toggle's
+                # config version carried NO metadata, so reject only reset
+                # last_config_status and the server stayed disabled (out of sync with
+                # the still-active live config). Mirrors the server-edit snapshot path;
+                # reject's rollback_entity_from_snapshot('server') restores is_active.
+                import json
+                from utils.entity_snapshot import save_entity_snapshot
+                entity_snapshot_metadata = await save_entity_snapshot(
+                    conn=conn2,
+                    entity_type="server",
+                    entity_id=server_id,
+                    old_values=dict(server),            # full pre-toggle row
+                    new_values={"is_active": new_status},
+                    operation="UPDATE",
+                )
+                old_config = await conn2.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                metadata = {"pre_apply_snapshot": old_config or "", **entity_snapshot_metadata}
+
                 # Create PENDING config version
                 config_version_id = await conn2.fetchval("""
-                    INSERT INTO config_versions 
-                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
-                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                    INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING', $6)
                     RETURNING id
-                """, cluster_id, version_name, config_content, config_hash, admin_user_id)
+                """, cluster_id, version_name, config_content, config_hash, admin_user_id, json.dumps(metadata))
                 
                 logger.error(f"SERVER TOGGLE DEBUG: Created PENDING config version {version_name} for cluster {cluster_id}")
                 

backend/routers/cluster.py

@@ -111,6 +111,14 @@ router = APIRouter(prefix="/api/clusters", tags=["clusters"])
 logger = logging.getLogger(__name__)
 
 
+class _AcmeNoConfigChange(Exception):
+    """Internal signal: an ACME edit renders the same config, so mint nothing.
+
+    Control flow, not an error — it unwinds out of the version-minting block without
+    tripping the generic `except Exception` handler that would log it as a failure.
+    """
+
+
 class _ConcurrentlyDrained(Exception):
     """Sentinel raised inside ``apply_pending_changes`` when the
     advisory-lock-protected re-fetch shows that another caller
@@ -292,12 +300,16 @@ async def create_cluster(cluster: HAProxyClusterCreate, authorization: str = Hea
         
         # Create cluster
         cluster_id = await conn.fetchval("""
-            INSERT INTO haproxy_clusters (name, description, connection_type, is_active, 
-                                        stats_socket_path, haproxy_config_path, haproxy_bin_path, pool_id)
-            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7)
+            INSERT INTO haproxy_clusters (name, description, connection_type, is_active,
+                                        stats_socket_path, haproxy_config_path, haproxy_bin_path,
+                                        keepalived_config_path, pool_id,
+                                        acme_enabled, acme_backend_url)
+            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8, COALESCE($9, FALSE), $10)
             RETURNING id
         """, cluster.name, cluster.description, cluster.connection_type,
-            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path, cluster.pool_id)
+            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path,
+            cluster.keepalived_config_path, cluster.pool_id,
+            cluster.acme_enabled, cluster.acme_backend_url)
         
         await close_database_connection(conn)
         
@@ -387,7 +399,8 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
         # Check if cluster exists and get current values
         existing_cluster = await conn.fetchrow("""
             SELECT name, description, connection_type, is_active, stats_socket_path,
-                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled
+                   haproxy_config_path, haproxy_bin_path, pool_id, acme_enabled,
+                   acme_backend_url
             FROM haproxy_clusters WHERE id = $1
         """, cluster_id)
         if not existing_cluster:
@@ -436,7 +449,12 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"haproxy_bin_path = ${param_counter}")
             update_values.append(cluster.haproxy_bin_path)
             param_counter += 1
-            
+
+        if cluster.keepalived_config_path is not None:
+            update_fields.append(f"keepalived_config_path = ${param_counter}")
+            update_values.append(cluster.keepalived_config_path)
+            param_counter += 1
+
         if cluster.pool_id is not None:
             update_fields.append(f"pool_id = ${param_counter}")
             update_values.append(cluster.pool_id)
@@ -446,7 +464,13 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"acme_enabled = ${param_counter}")
             update_values.append(cluster.acme_enabled)
             param_counter += 1
-        if cluster.acme_backend_url is not None:
+        # Keyed on "was the field submitted?", not "is it non-None". With a plain
+        # `is not None` test there is no way to CLEAR the value: the validator maps an
+        # empty box to None, which is indistinguishable from "not supplied", so once an
+        # operator set a per-cluster URL they could never revert to the global setting —
+        # the field would accept the edit and silently keep the old value.
+        _acme_url_submitted = 'acme_backend_url' in cluster.model_fields_set
+        if _acme_url_submitted:
             update_fields.append(f"acme_backend_url = ${param_counter}")
             update_values.append(cluster.acme_backend_url)
             param_counter += 1
@@ -456,14 +480,75 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_query = f"UPDATE haproxy_clusters SET {', '.join(update_fields)} WHERE id = $1"
             await conn.execute(update_query, *update_values)
 
-        # If acme_enabled actually changed, create a PENDING config version with entity snapshot
-        if cluster.acme_enabled is not None and cluster.acme_enabled != existing_cluster.get('acme_enabled', False):
+        # Create a PENDING config version when an ACME edit would change what the
+        # HAProxy nodes actually run.
+        #
+        # This used to trigger only on `acme_enabled` flipping. `acme_backend_url` is
+        # written to the DB a few lines above but minted nothing, so correcting a wrong
+        # challenge backend from the panel was a silent no-op: the value changed, no
+        # pending version existed, Apply answered "No pending changes to apply", and the
+        # nodes kept the old address indefinitely. That made the one field an operator
+        # needs to fix HTTP-01 impossible to actually apply.
+        _acme_toggled = (
+            cluster.acme_enabled is not None
+            and cluster.acme_enabled != existing_cluster.get('acme_enabled', False)
+        )
+        if _acme_toggled or _acme_url_submitted:
             try:
-                from services.haproxy_config import generate_haproxy_config_for_cluster
+                from services.haproxy_config import (
+                    generate_haproxy_config_for_cluster,
+                    is_config_generation_error,
+                    extract_acme_backend_target,
+                )
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
+                if is_config_generation_error(config_content):
+                    # Same sentinel-instead-of-exception contract as the apply path. A
+                    # PENDING version holding the sentinel is a landmine: the operator
+                    # sees a pending change and applies it, replacing the whole config.
+                    logger.error(
+                        f"ACME TOGGLE: config generation for cluster {cluster_id} returned an "
+                        f"error sentinel; no PENDING version created: {config_content!r}"
+                    )
+                    raise HTTPException(
+                        status_code=422,
+                        detail=(
+                            "ACME setting was saved, but a configuration could not be generated "
+                            "for this cluster, so no pending change was created. "
+                            f"Generator reported: {config_content.strip()[:300]}"
+                        ),
+                    )
+                # A URL edit that renders the same `server _acme_mgmt` line changes
+                # nothing on the nodes, so minting a version would put a no-op pending
+                # change in front of the operator. Compare that one line rather than the
+                # whole config: the generator also renders unrelated PENDING entities,
+                # so a full-text diff reports a change on every edit anyone has queued.
+                _active_config = await conn.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                _new_target = extract_acme_backend_target(config_content)
+                _old_target = extract_acme_backend_target(_active_config)
+                if not _acme_toggled and _new_target == _old_target:
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} URL updated but the rendered "
+                        f"challenge backend is unchanged ({_new_target!r}); no config "
+                        f"version created."
+                    )
+                    raise _AcmeNoConfigChange()
+
                 import time as _time
                 import json as _json
-                version_name = f"cluster-{cluster_id}-acme-{'enable' if cluster.acme_enabled else 'disable'}-{int(_time.time())}"
+                if _acme_toggled:
+                    _acme_kind = 'enable' if cluster.acme_enabled else 'disable'
+                else:
+                    _acme_kind = 'backend'
+                version_name = f"cluster-{cluster_id}-acme-{_acme_kind}-{int(_time.time())}"
+                logger.info(
+                    f"ACME-BACKEND: cluster {cluster_id} pending config version "
+                    f"'{version_name}' created — challenge backend {_old_target!r} -> "
+                    f"{_new_target!r}. Apply the cluster for the nodes to pick it up."
+                )
 
                 from utils.entity_snapshot import save_entity_snapshot
                 snapshot_metadata = await save_entity_snapshot(
@@ -475,7 +560,14 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                         "acme_backend_url": existing_cluster.get('acme_backend_url'),
                     },
                     new_values={
-                        "acme_enabled": cluster.acme_enabled,
+                        # `acme_enabled` is None when only the URL was submitted; the
+                        # snapshot must record the value that is actually in force, or a
+                        # rollback would write NULL over a working flag.
+                        "acme_enabled": (
+                            cluster.acme_enabled
+                            if cluster.acme_enabled is not None
+                            else existing_cluster.get('acme_enabled', False)
+                        ),
                         "acme_backend_url": getattr(cluster, 'acme_backend_url', None) or existing_cluster.get('acme_backend_url'),
                     },
                     operation="UPDATE"
@@ -490,9 +582,20 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
                     """, cluster_id, version_name, config_content, current_user.get('id', 1), metadata_json)
                 finally:
                     await close_database_connection(conn2)
+            except _AcmeNoConfigChange:
+                # Not an error: the edit was accepted and simply renders the same
+                # address, so there is nothing for the operator to apply.
+                pass
+            except HTTPException:
+                # The sentinel guard above deliberately fails the request. Without this
+                # clause the generic handler below would swallow it and report success
+                # while no PENDING version exists — the exact silent-success failure
+                # mode this change exists to remove.
+                await close_database_connection(conn)
+                raise
             except Exception as acme_err:
                 logger.error(f"Failed to create ACME config version for cluster {cluster_id}: {acme_err}")
-        
+
         await close_database_connection(conn)
         
         # Log activity
@@ -517,7 +620,7 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
 
 
 @router.get("/{cluster_id}", summary="Get Cluster by ID", response_description="Cluster details")
-async def get_cluster(cluster_id: int, authorization: str = Header(None)):
+async def get_cluster(cluster_id: int, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get Specific HAProxy Cluster
     
@@ -528,8 +631,12 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters/1" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters/1" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -554,20 +661,30 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     - **404**: Cluster not found
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 final convergence): authenticate
-        # the caller before fetching cluster topology by ID. Pre-fix
-        # this sibling of GET /api/clusters was anonymous, so an
-        # attacker could iterate cluster IDs to enumerate the same
-        # info (stats socket, paths, ACME flags, pool identity) the
-        # list endpoint just locked down. Closes the asymmetry.
+    # R18c audit fix (round 6 final convergence): authenticate the caller
+    # before fetching cluster topology by ID. Pre-fix this sibling of
+    # GET /api/clusters was anonymous, so an attacker could iterate cluster
+    # IDs to enumerate the same info (stats socket, paths, ACME flags, pool
+    # identity) the list endpoint just locked down.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         cluster = await conn.fetchrow("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name
             FROM haproxy_clusters c
@@ -591,6 +708,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
             "stats_socket_path": cluster["stats_socket_path"],
             "haproxy_config_path": cluster["haproxy_config_path"],
             "haproxy_bin_path": cluster["haproxy_bin_path"],
+            "keepalived_config_path": cluster.get("keepalived_config_path"),
             "pool_id": cluster["pool_id"],
             "pool_name": cluster["pool_name"],
             "acme_enabled": cluster.get("acme_enabled", False),
@@ -602,7 +720,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
         raise HTTPException(status_code=500, detail=str(e))
 
 @router.get("", summary="Get All Clusters", response_description="List of all clusters")
-async def get_clusters(authorization: str = Header(None)):
+async def get_clusters(authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All HAProxy Clusters
     
@@ -610,8 +728,12 @@ async def get_clusters(authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -662,23 +784,34 @@ async def get_clusters(authorization: str = Header(None)):
     ## Error Responses
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 #3 — KRITIK info leak): require an
-        # authenticated caller. Pre-fix the endpoint accepted
-        # anonymous GETs and returned cluster topology including
-        # internal HAProxy paths (stats socket, config path, bin
-        # path), pool ids, ACME flags, and agent counts. This is
-        # both reconnaissance for an attacker and the spine of the
-        # cluster-scoped RBAC the rest of the platform builds on,
-        # so guarding it at the read layer is essential after R18c
-        # round 5's roster + role guards.
+    # R18c audit fix (round 6 #3 — KRITIK info leak): require an authenticated
+    # caller. Pre-fix the endpoint accepted anonymous GETs and returned cluster
+    # topology including internal HAProxy paths (stats socket, config path, bin
+    # path), pool ids, ACME flags, and agent counts. This is both reconnaissance
+    # for an attacker and the spine of the cluster-scoped RBAC the rest of the
+    # platform builds on, so guarding it at the read layer is essential.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    # (Guard kept OUTSIDE the try below: get_clusters' broad `except Exception`
+    # re-wraps raised HTTPExceptions into 500, which produced the "500 - 401"
+    # in the issue log; raising here yields a clean 401.)
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         clusters = await conn.fetch("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name,
                    COALESCE(agent_counts.total_agents, 0) as total_agents,
@@ -737,6 +870,7 @@ async def get_clusters(authorization: str = Header(None)):
                 "stats_socket_path": cluster["stats_socket_path"],
                 "haproxy_config_path": cluster["haproxy_config_path"],
                 "haproxy_bin_path": cluster["haproxy_bin_path"],
+                "keepalived_config_path": cluster.get("keepalived_config_path"),
                 "pool_id": cluster["pool_id"],
                 "pool_name": cluster["pool_name"],
                 "acme_enabled": cluster.get("acme_enabled", False),
@@ -1313,6 +1447,8 @@ async def list_cluster_config_versions(cluster_id: int, authorization: str = Hea
                 version_type = "WAF Rule"
             elif "ssl-" in version['version_name']:
                 version_type = "SSL Certificate"
+            elif "vip-" in version['version_name']:
+                version_type = "HA / VIP"
             
             # Parse validation error if present
             validation_error = version.get("validation_error")
@@ -1417,10 +1553,14 @@ async def apply_pending_changes(
         # Users will handle configuration completeness through the centralized Apply Management page
         
         # Get all pending config versions for this cluster
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP apply/reject endpoints
+        # (keepalived is not part of haproxy.cfg). Exclude them so a generic cluster apply
+        # from any entity page never marks a VIP version APPLIED without enacting it. This
+        # is a no-op for every non-VIP cluster (no vip-* rows exist).
         pending_versions = await conn.fetch("""
             SELECT id, version_name, created_at, config_content, checksum, metadata
-            FROM config_versions 
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
             ORDER BY created_at ASC
         """, cluster_id)
         
@@ -1870,6 +2010,28 @@ defaults
                 logger.info(f"🧩 APPLY: Generating fresh configuration from database for cluster {cluster_id}")
                 fresh_config_content = await generate_haproxy_config_for_cluster(cluster_id, conn)
             
+            # `generate_haproxy_config_for_cluster` reports failure by RETURNING a
+            # one-line comment instead of raising (haproxy_config.py outer `except`).
+            # Without this guard that sentinel is hashed, stored as an APPLIED version
+            # and shipped to every agent — silently replacing the cluster's entire
+            # configuration with a comment. Any generator exception (an out-of-range
+            # port in acme_backend_url is enough) triggers it. Refuse the apply instead;
+            # the previous APPLIED version stays in force.
+            from services.haproxy_config import is_config_generation_error
+            if is_config_generation_error(fresh_config_content):
+                logger.error(
+                    f"APPLY ABORTED: config generation for cluster {cluster_id} returned an "
+                    f"error sentinel instead of a configuration: {fresh_config_content!r}"
+                )
+                raise HTTPException(
+                    status_code=422,
+                    detail=(
+                        "Configuration could not be generated for this cluster, so nothing "
+                        "was applied and the running configuration is unchanged. "
+                        f"Generator reported: {fresh_config_content.strip()[:300]}"
+                    ),
+                )
+
             # Create a new consolidated config version with fresh content
             import hashlib
             import time
@@ -2539,6 +2701,96 @@ async def get_config_version_diff(cluster_id: int, version_id: int, authorizatio
                     }
                 }
         
+        # HA/VIP (Issue #27): vip-{id}-{action} versions show the generated keepalived.conf
+        # each member node will deploy as the change content (VRRP secret masked). Mirrors
+        # the ssl-* special case above so VIP uses the STANDARD View Change diff modal.
+        # `adopt` MUST stay in this alternation. A vip-* action missing here does not degrade
+        # gracefully: the version falls through to the generic HAProxy diff, which compares this
+        # row's keepalived.conf against the cluster's previous haproxy.cfg and shows the whole
+        # HAProxy config as removed. v1.10.4 added `adopt` without it (fixed in v1.10.8).
+        vip_match = re.search(r'vip-(\d+)-(create|update|delete|adopt)', current_version['version_name'])
+        if vip_match:
+            vip_id = int(vip_match.group(1))
+            vip_action = vip_match.group(2)
+            rendered, vip_meta = None, None
+            old_content = ""
+            try:
+                from routers.vip import render_vip_config_masked
+                rendered, vip_meta = await render_vip_config_masked(conn, vip_id)
+                # For a create/update diff, fetch the PREVIOUS applied vip-* config for this VIP
+                # (the last-deployed keepalived.conf) so an EDIT shows ONLY the changed lines
+                # instead of the whole config as "added". Both sides are already secret-masked.
+                if vip_action != "delete":
+                    prev_applied = await conn.fetchrow(
+                        "SELECT config_content FROM config_versions WHERE version_name LIKE $1 "
+                        "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1",
+                        f"vip-{vip_id}-%", cluster_id, current_version['id'])
+                    if prev_applied and prev_applied['config_content']:
+                        old_content = prev_applied['config_content']
+            except Exception as vip_err:
+                logger.warning(f"VIP DIFF: render failed for vip {vip_id}: {vip_err}")
+            await close_database_connection(conn)
+
+            changes = []
+            line_number = 1
+            if vip_action == "delete" or rendered is None:
+                title = (vip_meta or {}).get("name") or f"VIP {vip_id}"
+                for line in [f"# HA/VIP change: {title}",
+                             "# keepalived will be stopped and the virtual IP released on each member node."
+                             if vip_action == "delete" else
+                             "# (configuration is not available to render yet)"]:
+                    changes.append({"type": "context", "line": line, "line_number": line_number})
+                    line_number += 1
+                summary = {"added": 0, "removed": 1 if vip_action == "delete" else 0,
+                           "total_changes": 1 if vip_action == "delete" else 0}
+            else:
+                # Real line diff. Match the STANDARD haproxy diff format: the line is stored
+                # WITHOUT a +/- prefix (the UI adds it from `type` — the old `+ {line}` here
+                # caused the doubled "+ +"). A CREATE (no previous applied config) shows
+                # everything as added; an UPDATE shows ONLY the lines that actually changed.
+                import difflib
+                new_content = current_version['config_content'] or rendered or ""
+                added_count = 0
+                removed_count = 0
+                line_number = 0
+                if not old_content:
+                    for i, l in enumerate(new_content.split('\n')):
+                        changes.append({"type": "added", "line": l, "line_number": i + 1})
+                        added_count += 1
+                else:
+                    for dl in difflib.unified_diff(old_content.split('\n'), new_content.split('\n'),
+                                                   lineterm='', n=3):
+                        if dl.startswith('@@'):
+                            mm = re.search(r'@@ -(\d+),?\d* \+(\d+),?\d* @@', dl)
+                            if mm:
+                                line_number = int(mm.group(2))
+                            continue
+                        if dl.startswith('---') or dl.startswith('+++'):
+                            continue
+                        if dl.startswith('+'):
+                            changes.append({"type": "added", "line": dl[1:], "line_number": line_number})
+                            added_count += 1
+                            line_number += 1
+                        elif dl.startswith('-'):
+                            changes.append({"type": "removed", "line": dl[1:], "line_number": line_number})
+                            removed_count += 1
+                        elif dl.startswith(' '):
+                            changes.append({"type": "context", "line": dl[1:], "line_number": line_number})
+                            line_number += 1
+                summary = {"added": added_count, "removed": removed_count,
+                           "total_changes": added_count + removed_count}
+
+            return {
+                "current_version": {
+                    "id": current_version['id'],
+                    "version_name": current_version['version_name'],
+                    "created_at": current_version['created_at'].isoformat().replace('+00:00', 'Z')
+                },
+                "previous_version": None,
+                "changes": changes,
+                "summary": summary,
+            }
+
         # Check if current version has config content
         if not current_version['config_content']:
             # Special handling for restore versions - they might not have content yet
@@ -3566,17 +3818,19 @@ async def confirm_restore_config_version(
                     # UPDATE existing frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        UPDATE frontends 
-                        SET bind_address = $1, bind_port = $2, default_backend = $3, 
+                        UPDATE frontends
+                        SET bind_address = $1, bind_port = $2, default_backend = $3,
                             mode = $4, ssl_enabled = $5, ssl_port = $6,
                             maxconn = $7, timeout_client = $8,
+                            log_format = $11, filters = $12,
                             updated_at = CURRENT_TIMESTAMP, last_config_status = 'PENDING'
                         WHERE id = $9 AND cluster_id = $10
-                    """, 
+                    """,
                         parsed_fe.bind_address, parsed_fe.bind_port, parsed_fe.default_backend,
                         parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        fe_id, cluster_id
+                        fe_id, cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_updated"] += 1
                     logger.info(f"RESTORE: Updated frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")
@@ -3584,16 +3838,17 @@ async def confirm_restore_config_version(
                     # CREATE new frontend (ALL 8 parsed fields)
                     # CRITICAL FIX: Include maxconn and timeout_client so UI shows restored values
                     await conn.execute("""
-                        INSERT INTO frontends 
+                        INSERT INTO frontends
                         (name, bind_address, bind_port, default_backend, mode, ssl_enabled, ssl_port,
                          maxconn, timeout_client,
-                         cluster_id, is_active, last_config_status, created_at, updated_at)
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
-                    """, 
+                         cluster_id, log_format, filters, is_active, last_config_status, created_at, updated_at)
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, TRUE, 'PENDING', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
+                    """,
                         parsed_fe.name, parsed_fe.bind_address, parsed_fe.bind_port,
                         parsed_fe.default_backend, parsed_fe.mode, parsed_fe.ssl_enabled, parsed_fe.ssl_port,
                         parsed_fe.maxconn, parsed_fe.timeout_client,
-                        cluster_id
+                        cluster_id,
+                        parsed_fe.log_format, parsed_fe.filters  # Issue #38
                     )
                     changes_summary["frontends_created"] += 1
                     logger.info(f"RESTORE: Created frontend '{parsed_fe.name}' (SSL: {parsed_fe.ssl_enabled}, maxconn: {parsed_fe.maxconn})")
@@ -4226,6 +4481,20 @@ async def undo_reject_config_version(
                 ),
             )
 
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP entity, so undo is handled
+        # by the VIP router — it re-stages the rejected change as PENDING from the version's
+        # captured pending_state (reactivating the VIP if a rejected create soft-deleted it,
+        # or re-applying a rejected edit). Returns an error string if it can't (e.g. the
+        # name/address/VRID was reused since reject) -> surface a clean 409.
+        if version_name.startswith('vip-'):
+            from routers.vip import restore_vip_from_rejected_version
+            err = await restore_vip_from_rejected_version(conn, version_id)
+            await close_database_connection(conn)
+            if err:
+                raise HTTPException(status_code=409, detail=f"Cannot undo this VIP change — {err}.")
+            return {"message": "VIP change restored to PENDING — review and Apply it from Apply Management",
+                    "version_name": version_name}
+
         async with conn.transaction():
             # Mark the version as PENDING again
             await conn.execute("""
@@ -4902,9 +5171,17 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         await validate_user_cluster_access(current_user['id'], cluster_id, conn)
 
         # Get all pending config versions for this cluster (CRITICAL: Include metadata for rollback!)
+        # HA/VIP (Issue #27): exclude vip-* versions — they are rejected/reverted by the
+        # VIP reject endpoint (which restores keepalived state), not the generic rollback.
+        # ORDER BY created_at ASC: the rollback loop dedups per entity and keeps the FIRST-processed
+        # snapshot, so the OLDEST snapshot must win — its old_values hold the true pre-change state.
+        # Critical when one entity has multiple pending versions (e.g. cluster ACME enable->disable->enable):
+        # rolling back to the oldest restores the original acme_enabled. (Matches the apply SELECT, which
+        # already orders created_at ASC.)
         pending_versions = await conn.fetch("""
             SELECT id, version_name, metadata FROM config_versions
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
+            ORDER BY created_at ASC
         """, cluster_id)
         
         # CRITICAL FIX: Detect and clean orphan config versions
@@ -5116,11 +5393,12 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         )
         
         # Mark all pending config versions as REJECTED (don't delete them)
+        # HA/VIP (Issue #27): leave vip-* versions to the VIP reject endpoint.
         rejected_count = len(pending_versions)
         await conn.execute("""
-            UPDATE config_versions 
+            UPDATE config_versions
             SET status = 'REJECTED'
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
         """, cluster_id)
 
         # Update WAF rules status to APPLIED (rolled back)

backend/routers/config.py

@@ -3,7 +3,7 @@ Configuration Management and Validation API
 Provides endpoints for HAProxy configuration validation, templates, and optimization
 """
 
-from fastapi import APIRouter, HTTPException, Header, Request
+from fastapi import APIRouter, HTTPException, Header, Request, Depends
 from pydantic import BaseModel
 from typing import Dict, List, Any, Optional
 import logging
@@ -18,7 +18,7 @@ from utils.config_templates import (
 )
 from utils.haproxy_config_parser import parse_haproxy_config
 from utils.logging_config import log_with_correlation, PerformanceLogger
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from database.connection import get_database_connection, close_database_connection
 
 router = APIRouter(prefix="/api/config", tags=["Configuration Management"])
@@ -62,7 +62,7 @@ class ConfigOptimizationRequest(BaseModel):
     optimization_level: str = "balanced"  # conservative, balanced, aggressive
     target_environment: str = "production"  # development, staging, production
 
-@router.post("/validate")
+@router.post("/validate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth; runs HAProxy validator on caller input
 async def validate_configuration(
     request: ConfigValidationRequest,
     current_user: dict = None,
@@ -203,7 +203,7 @@ async def get_template_details(template_id: str):
         )
         raise HTTPException(status_code=500, detail=f"Failed to get template: {str(e)}")
 
-@router.post("/templates/{template_id}/generate")
+@router.post("/templates/{template_id}/generate", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def generate_configuration(
     template_id: str,
     request: TemplateGenerationRequest,
@@ -271,7 +271,7 @@ async def generate_configuration(
                 detail=f"Configuration generation failed: {str(e)}"
             )
 
-@router.post("/optimize")
+@router.post("/optimize", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was optional-auth
 async def optimize_configuration(
     request: ConfigOptimizationRequest,
     current_user: dict = None,
@@ -855,6 +855,9 @@ async def parse_bulk_config(
                 "response_headers": frontend.response_headers,
                 "options": frontend.options,
                 "tcp_request_rules": frontend.tcp_request_rules,
+                # Issue #38: SPOE filters + frontend log-format
+                "log_format": frontend.log_format,
+                "filters": frontend.filters,
                 # CRITICAL: SSL Advanced Options (parsed from bind directive)
                 "ssl_alpn": frontend.ssl_alpn,
                 "ssl_npn": frontend.ssl_npn,
@@ -1089,7 +1092,69 @@ async def parse_bulk_config(
         
         # Add auto-assignment info at the beginning
         enhanced_warnings = ssl_auto_assign_info + enhanced_warnings
-        
+
+        # ─────────────────────────────────────────────────────────────────
+        # Issue #38: SPOE pre-flight advisories. Surface, at preview time, the
+        # SPOE configurations that would FAIL HAProxy's `haproxy -c` at apply so
+        # the operator sees them BEFORE importing. Cluster-aware: the referenced
+        # SPOE engine config (e.g. coraza.cfg) is a sibling of the cluster's
+        # haproxy_config_path, which HAProxy OpenManager does not provision.
+        # ─────────────────────────────────────────────────────────────────
+        try:
+            _cfg_path = await conn.fetchval(
+                "SELECT haproxy_config_path FROM haproxy_clusters WHERE id = $1",
+                request.cluster_id,
+            ) or "/etc/haproxy/haproxy.cfg"
+            _cfg_dir = _cfg_path.rsplit("/", 1)[0] or "/etc/haproxy"
+            for _fe in frontends_data:
+                _rh = _fe.get("request_headers") or ""
+                _filters = _fe.get("filters") or ""
+                # engines declared by `filter spoe engine <name> config <path>`
+                _declared_engines = set(re.findall(
+                    r"filter\s+spoe\s+engine\s+(\S+)", _filters, re.IGNORECASE))
+                # engines referenced by `... send-spoe-group <name> <group>`
+                _used_engines = set(re.findall(
+                    r"send-spoe-group\s+(\S+)", _rh, re.IGNORECASE))
+                _missing = _used_engines - _declared_engines
+                if _missing:
+                    enhanced_warnings.append(
+                        f"⚠️ Frontend '{_fe['name']}': 'send-spoe-group' references SPOE "
+                        f"engine(s) {', '.join(sorted(_missing))} but no matching "
+                        f"'filter spoe engine <name> ...' line was found. HAProxy will "
+                        f"reject this at apply with \"unable to find SPOE engine\". Add the "
+                        f"filter line to this frontend."
+                    )
+                for _path in re.findall(
+                        r"filter\s+spoe\s+engine\s+\S+\s+config\s+(\S+)",
+                        _filters, re.IGNORECASE):
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': SPOE engine config '{_path}' and its "
+                        f"SPOA backend must exist on the HAProxy host (cluster config dir: "
+                        f"{_cfg_dir}). HAProxy OpenManager preserves the filter directive but "
+                        f"does not provision these files; otherwise 'haproxy -c' fails at apply."
+                    )
+                # Issue #38 follow-up: ACL `-f <file>` pattern-file advisory.
+                # Scan only the structured rule fields (acl/use_backend) —
+                # request_headers/tcp_request_rules were always free-form and
+                # warning on them now would add new noise for existing users.
+                _pattern_paths = []
+                for _rule in (_fe.get("acl_rules") or []) + (_fe.get("use_backend_rules") or []):
+                    if isinstance(_rule, str):
+                        _pattern_paths.extend(
+                            re.findall(r"(?:^|\s)-f\s+(\S+)", _rule))
+                if _pattern_paths:
+                    _uniq = sorted(set(_pattern_paths))
+                    enhanced_warnings.append(
+                        f"ℹ️ Frontend '{_fe['name']}': ACL/routing rules reference pattern "
+                        f"file(s) {', '.join(_uniq)}. Each file must exist at that exact path "
+                        f"on every HAProxy host in the cluster (cluster config dir: {_cfg_dir}) "
+                        f"— HAProxy OpenManager does not create or distribute pattern files. "
+                        f"A missing file fails safely at 'haproxy -c' (previous config keeps "
+                        f"running)."
+                    )
+        except Exception as _spoe_adv_err:
+            logger.warning(f"SPOE advisory generation skipped: {_spoe_adv_err}")
+
         # BULK IMPORT MVP: Check existing entities for UPSERT detection
         # Mark each entity as new or update for UI display
         # CRITICAL: Only mark as UPDATE if there are actual field changes
@@ -1150,7 +1215,17 @@ async def parse_bulk_config(
                 if frontend.get("tcp_request_rules") and frontend["tcp_request_rules"] != existing["tcp_request_rules"]:
                     has_changes = True
                     changes["tcp_request_rules"] = {"old": existing["tcp_request_rules"], "new": frontend["tcp_request_rules"]}
-                
+                # Issue #38: SPOE filters + log-format change detection. REQUIRED for
+                # persistence (not just display): without it, an import that only adds
+                # a `filter`/`log-format` to an existing frontend would be flagged
+                # "no change" and the directive would never be written to the DB.
+                if frontend.get("log_format") and frontend["log_format"] != existing.get("log_format"):
+                    has_changes = True
+                    changes["log_format"] = {"old": existing.get("log_format"), "new": frontend["log_format"]}
+                if frontend.get("filters") and frontend["filters"] != existing.get("filters"):
+                    has_changes = True
+                    changes["filters"] = {"old": existing.get("filters"), "new": frontend["filters"]}
+
                 # CRITICAL: SSL Advanced Options change detection
                 if frontend.get("ssl_alpn") is not None and frontend.get("ssl_alpn") != existing.get("ssl_alpn"):
                     has_changes = True
@@ -2094,7 +2169,18 @@ async def bulk_create_entities(
                         update_fields.append(f"options = ${param_index}")
                         update_values.append(frontend_data["options"])
                         param_index += 1
-                    
+
+                    # Issue #38: SPOE filters + frontend log-format (merge strategy)
+                    if frontend_data.get("log_format") and frontend_data["log_format"] != existing_full.get("log_format"):
+                        update_fields.append(f"log_format = ${param_index}")
+                        update_values.append(frontend_data["log_format"])
+                        param_index += 1
+
+                    if frontend_data.get("filters") and frontend_data["filters"] != existing_full.get("filters"):
+                        update_fields.append(f"filters = ${param_index}")
+                        update_values.append(frontend_data["filters"])
+                        param_index += 1
+
                     # CRITICAL FIX: Update SSL advanced options (alpn, npn, ciphers, etc.)
                     # These are parsed from bind directive and should be preserved in database
                     if "ssl_alpn" in frontend_data and frontend_data.get("ssl_alpn") != existing_full.get("ssl_alpn"):
@@ -2214,9 +2300,10 @@ async def bulk_create_entities(
                             timeout_client, timeout_http_request, maxconn,
                             request_headers, response_headers, tcp_request_rules, options,
                             rate_limit, compression, log_separate, monitor_uri,
-                            cluster_id, acl_rules, use_backend_rules, redirect_rules, updated_at
-                        ) 
-                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                            cluster_id, acl_rules, use_backend_rules, redirect_rules,
+                            log_format, filters, updated_at
+                        )
+                        VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
                         RETURNING id
                     """, 
                         frontend_data["name"],
@@ -2257,7 +2344,9 @@ async def bulk_create_entities(
                         request.cluster_id,
                         json.dumps(frontend_data.get("acl_rules", [])),  # acl_rules
                         json.dumps(frontend_data.get("use_backend_rules", [])),  # use_backend_rules
-                        json.dumps([])   # redirect_rules
+                        json.dumps([]),   # redirect_rules
+                        frontend_data.get("log_format"),  # Issue #38
+                        frontend_data.get("filters")      # Issue #38
                     )
                     
                     created_entities["frontends"].append({

backend/routers/configuration.py

@@ -201,13 +201,15 @@ async def get_pending_config_requests(agent_name: str, x_api_key: Optional[str]
     Called during heartbeat.
     """
     try:
-        # Validate agent API key
+        # Validate agent API key — MANDATORY (GHSA-3p5c-m5m4-mjpx). Deployed agents
+        # always send X-API-Key; an absent/invalid key is unauthenticated. This
+        # endpoint also mutates state (marks requests 'processing'), so a keyless
+        # caller could otherwise starve the real agent.
         agent_auth = await validate_agent_api_key(x_api_key)
-        
-        if x_api_key and not agent_auth:
-            logger.warning(f"Invalid API key provided by agent '{agent_name}' for pending requests")
-            raise HTTPException(status_code=401, detail="Invalid API key")
-        
+        if not agent_auth:
+            logger.warning(f"Missing/invalid API key from '{agent_name}' for pending requests")
+            raise HTTPException(status_code=401, detail="Authentication required")
+
         conn = await get_database_connection()
         
         # Get pending requests

backend/routers/csr.py

@@ -0,0 +1,375 @@
+"""
+CSR (Certificate Signing Request) endpoints (v1.9.0).
+
+Generate a private key + CSR in-app, download the CSR PEM, have it signed by
+an external CA, then import the signed certificate — which creates a normal
+ssl_certificates row that flows through the existing pipeline
+(config version → Apply Management → agent pull).
+
+Security posture:
+- All endpoints enforce ssl.* permissions explicitly (including the read
+  endpoints — deliberately stricter than the legacy cert detail route).
+- The private key is NEVER returned by any endpoint here; after import it is
+  reachable only via the existing certificate detail route.
+- Key generation is offloaded to a thread (RSA-4096 takes seconds; the
+  backend runs a single-worker event loop by default) and rate-limited
+  per user via the user_activity_logs COUNT pattern (acme_diagnostics
+  precedent — slowapi is not registered on the app).
+"""
+
+import asyncio
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, HTTPException, Request, Header
+
+from database.connection import get_database_connection, close_database_connection
+from auth_middleware import get_current_user_from_token, check_user_permission
+from models.csr import SSLCSRCreate, SSLCSRImport
+from services import csr_service, ssl_service
+from routers.ssl import _assert_safe_cert_name, validate_user_cluster_access
+from utils.activity_log import log_user_activity
+
+router = APIRouter(prefix="/api/ssl/csrs", tags=["SSL CSRs"])
+logger = logging.getLogger(__name__)
+
+_RATE_LIMIT_CREATE_PER_MIN = 10
+
+# Columns exposed to the API — private_key_pem is deliberately absent so a
+# future `SELECT *` refactor cannot silently start leaking it.
+_CSR_LIST_COLUMNS = """
+    c.id, c.name, c.common_name, c.subject, c.sans, c.key_algorithm,
+    c.status, c.ssl_certificate_id, c.completed_at, c.created_at, c.updated_at,
+    s.name AS certificate_name, u.username AS created_by_username
+"""
+
+_INT32_MAX = 2_147_483_647
+
+
+def _client_ip(request: Optional[Request]) -> Optional[str]:
+    try:
+        return str(request.client.host) if request and request.client else None
+    except Exception:
+        return None
+
+
+def _user_agent(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.headers.get("user-agent") if request else None
+    except Exception:
+        return None
+
+
+async def _require(authorization: Optional[str], action: str):
+    """Authenticate + enforce ssl.<action>; returns current_user or raises 401/403."""
+    current_user = await get_current_user_from_token(authorization)
+    ok = await check_user_permission(current_user["id"], "ssl", action, current_user=current_user)
+    if not ok:
+        raise HTTPException(status_code=403, detail=f"Insufficient permissions: ssl.{action} required")
+    return current_user
+
+
+def _assert_int32_id(csr_id: int) -> None:
+    """ssl_csrs.id is int4 — an out-of-range path param would surface as an
+    asyncpg DataError 500 (Bulgu #96 precedent); return a clean 404 instead."""
+    if csr_id < 1 or csr_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="CSR not found")
+
+
+def _assert_valid_cluster_id(cluster_id: int) -> None:
+    """Same int4 guard for body-supplied cluster ids: haproxy_clusters.id is
+    SERIAL/int4, so an out-of-range value would raise asyncpg DataError inside
+    validate_user_cluster_access and surface as a 500 with the raw driver
+    error. Fail with the same clean 404 the cluster lookup itself produces."""
+    if not isinstance(cluster_id, int) or cluster_id < 1 or cluster_id > _INT32_MAX:
+        raise HTTPException(status_code=404, detail="Cluster not found")
+
+
+async def _enforce_create_rate_limit(conn, user_id: int) -> None:
+    """Per-user per-minute limit on key generation, counted against the
+    csr_create audit-log action (acme_diagnostics _enforce_rate_limit pattern,
+    backed by the (user_id, action, created_at DESC) composite index)."""
+    cnt = await conn.fetchval(
+        """
+        SELECT COUNT(*)
+        FROM user_activity_logs
+        WHERE user_id = $1
+          AND action = 'csr_create'
+          AND created_at >= NOW() - INTERVAL '60 seconds'
+        """,
+        user_id,
+    )
+    if cnt is not None and cnt >= _RATE_LIMIT_CREATE_PER_MIN:
+        raise HTTPException(
+            status_code=429,
+            detail=(
+                f"Rate limit exceeded: at most {_RATE_LIMIT_CREATE_PER_MIN} "
+                "CSRs may be created per minute"
+            ),
+        )
+
+
+@router.post("")
+async def create_csr(payload: SSLCSRCreate, request: Request, authorization: Optional[str] = Header(None)):
+    """Generate a private key + CSR. Returns the CSR PEM immediately (so the
+    UI can show copy/download in one round trip) — never the private key."""
+    current_user = await _require(authorization, "create")
+    conn = None
+    try:
+        # Belt and braces on top of the model validator — same duplication
+        # convention as the certificate create route.
+        _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+        await _enforce_create_rate_limit(conn, current_user["id"])
+
+        # Fail fast on a taken name BEFORE burning CPU on key generation;
+        # insert_csr_row re-checks and the partial unique index closes the race.
+        await csr_service.assert_csr_name_available(conn, payload.name)
+
+        bundle = await asyncio.to_thread(csr_service.generate_csr_bundle, payload)
+        csr_id = await csr_service.insert_csr_row(conn, payload, bundle, current_user["id"])
+
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_create',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'csr_name': payload.name,
+                'common_name': payload.common_name,
+                'sans': bundle['sans'],
+                'key_algorithm': payload.key_algorithm,
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": f"CSR '{payload.name}' created successfully",
+            "csr": csr_service.csr_row_to_dict(row, include_pem=True),
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error creating CSR: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("")
+async def list_csrs(authorization: Optional[str] = Header(None)):
+    """List CSRs (no PEM payloads — fetch the detail route for the CSR PEM).
+    Cluster-agnostic: a CSR binds to clusters only at import time."""
+    await _require(authorization, "read")
+    conn = None
+    try:
+        conn = await get_database_connection()
+        rows = await conn.fetch(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            ORDER BY c.created_at DESC
+            """
+        )
+        return [csr_service.csr_row_to_dict(r) for r in rows]
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing CSRs: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.get("/{csr_id}")
+async def get_csr(csr_id: int, authorization: Optional[str] = Header(None)):
+    """CSR detail including the CSR PEM. The private key is never included."""
+    await _require(authorization, "read")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            f"""
+            SELECT {_CSR_LIST_COLUMNS}, c.csr_pem
+            FROM ssl_csrs c
+            LEFT JOIN ssl_certificates s ON c.ssl_certificate_id = s.id
+            LEFT JOIN users u ON c.created_by = u.id
+            WHERE c.id = $1
+            """,
+            csr_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        return csr_service.csr_row_to_dict(row, include_pem=True)
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{csr_id}/import")
+async def import_csr_certificate(
+    csr_id: int,
+    payload: SSLCSRImport,
+    request: Request,
+    authorization: Optional[str] = Header(None),
+):
+    """Import the CA-signed certificate for a pending CSR. Creates an
+    ssl_certificates row (source='csr', PENDING) and stages one config
+    version per affected cluster — the operator applies manually."""
+    current_user = await _require(authorization, "create")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        if payload.name:
+            _assert_safe_cert_name(payload.name)
+
+        conn = await get_database_connection()
+
+        if not payload.is_global:
+            for cluster_id in payload.cluster_ids or []:
+                _assert_valid_cluster_id(cluster_id)
+                await validate_user_cluster_access(current_user["id"], cluster_id, conn)
+
+        result = await csr_service.import_signed_certificate(
+            conn, csr_id, payload, current_user["id"]
+        )
+        cert_id = result["certificate_id"]
+
+        if payload.is_global:
+            cluster_rows = await conn.fetch(
+                "SELECT id FROM haproxy_clusters WHERE is_active = TRUE"
+            )
+            affected_clusters = [r['id'] for r in cluster_rows]
+        else:
+            affected_clusters = payload.cluster_ids or []
+
+        # Post-commit staging — a config-generation failure never rolls back
+        # the certificate (same semantics as the manual create flow).
+        sync_results = await ssl_service.stage_ssl_config_versions(
+            conn, cert_id, affected_clusters, action='create',
+            created_by=current_user["id"],
+        )
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='create',
+            resource_type='ssl_certificate',
+            resource_id=str(cert_id),
+            details={
+                'certificate_name': result['certificate_name'],
+                'domain': result.get('primary_domain', 'unknown'),
+                'via': 'csr',
+                'csr_id': csr_id,
+                'usage_type': payload.usage_type,
+                'is_global': payload.is_global,
+                'cluster_ids': payload.cluster_ids,
+                'warnings': result['warnings'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='csr_import',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={
+                'certificate_id': cert_id,
+                'certificate_name': result['certificate_name'],
+            },
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        return {
+            "message": (
+                f"Certificate '{result['certificate_name']}' imported "
+                "successfully. Go to Apply Management to deploy."
+            ),
+            "certificate_id": cert_id,
+            "warnings": result["warnings"],
+            "sync_results": sync_results,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error importing signed certificate for CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.delete("/{csr_id}")
+async def delete_csr(csr_id: int, request: Request, authorization: Optional[str] = Header(None)):
+    """Hard delete. For a pending CSR this permanently destroys the private
+    key (any certificate later signed from that CSR becomes unusable); for a
+    completed CSR it only removes history — the imported certificate is not
+    affected (the FK points csr → cert)."""
+    current_user = await _require(authorization, "delete")
+    _assert_int32_id(csr_id)
+    conn = None
+    try:
+        conn = await get_database_connection()
+        async with conn.transaction():
+            # FOR UPDATE serialises against an in-flight import of the same CSR.
+            row = await conn.fetchrow(
+                "SELECT id, name, status FROM ssl_csrs WHERE id = $1 FOR UPDATE",
+                csr_id,
+            )
+            if not row:
+                raise HTTPException(status_code=404, detail="CSR not found")
+            await conn.execute("DELETE FROM ssl_csrs WHERE id = $1", csr_id)
+
+        await log_user_activity(
+            user_id=current_user["id"],
+            action='delete',
+            resource_type='ssl_csr',
+            resource_id=str(csr_id),
+            details={'csr_name': row['name'], 'status': row['status']},
+            ip_address=_client_ip(request),
+            user_agent=_user_agent(request),
+        )
+
+        if row['status'] == 'pending':
+            message = (
+                f"CSR '{row['name']}' deleted — its private key has been "
+                "permanently destroyed."
+            )
+        else:
+            message = (
+                f"CSR '{row['name']}' deleted (history only) — the imported "
+                "certificate is not affected."
+            )
+        return {"message": message}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error deleting CSR {csr_id}: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+    finally:
+        if conn:
+            await close_database_connection(conn)

backend/routers/dashboard.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Header
+from fastapi import APIRouter, HTTPException, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 from datetime import datetime
 import logging
@@ -11,7 +12,7 @@ from agent_notifications import get_cluster_agents_status
 router = APIRouter(prefix="/api", tags=["dashboard", "pools"])
 logger = logging.getLogger(__name__)
 
-@router.get("/dashboard/overview")
+@router.get("/dashboard/overview", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaked cluster/pool/agent stats, names, health & alerts anonymously (auth was optional)
 async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization: str = Header(None)):
     """Get dashboard overview with comprehensive statistics, optionally filtered by cluster"""
     try:
@@ -238,7 +239,7 @@ async def get_dashboard_overview(cluster_id: Optional[int] = None, authorization
         logger.error(f"Error fetching dashboard overview: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/dashboard/stats")
+@router.get("/dashboard/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): aggregate cluster/agent counts
 async def get_dashboard_stats():
     """Get dashboard statistics"""
     try:
@@ -279,7 +280,7 @@ async def get_dashboard_stats():
     except Exception as e:
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/pools")
+@router.get("/pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): pool names/env/counts
 async def get_pools():
     """Get all HAProxy cluster pools"""
     try:
@@ -350,7 +351,7 @@ async def get_pools():
         logger.error(f"Error fetching pools: {e}")
         return {"pools": []}
 
-@router.get("/haproxy-cluster-pools")
+@router.get("/haproxy-cluster-pools", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_cluster_pools():
     """Get all HAProxy cluster pools (legacy endpoint)"""
     # Just call the main pools endpoint
@@ -474,7 +475,7 @@ async def update_pool(pool_id: int, pool: PoolUpdate, authorization: str = Heade
         logger.error(f"Failed to update pool: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to update pool: {str(e)}")
 
-@router.get("/haproxy-cluster-pools/{pool_id}/agents")
+@router.get("/haproxy-cluster-pools/{pool_id}/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): full agent inventory — same class as GET /api/agents
 async def get_pool_agents(pool_id: int):
     """Get all agents for a specific pool"""
     try:
@@ -561,7 +562,7 @@ async def get_pool_agents(pool_id: int):
         logger.error(f"Error fetching pool agents: {e}")
         raise HTTPException(status_code=500, detail=f"Failed to fetch pool agents: {str(e)}")
 
-@router.get("/haproxy/stats")
+@router.get("/haproxy/stats", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c)
 async def get_haproxy_stats(cluster_id: Optional[int] = None):
     """Get HAProxy statistics"""
     try:

backend/routers/dashboard_stats.py

@@ -3,13 +3,22 @@ Dashboard Stats Router
 API endpoints for HAProxy statistics dashboard
 """
 
-from fastapi import APIRouter, HTTPException, Query
+from fastapi import APIRouter, HTTPException, Query, Depends
 from typing import Optional, List
 import logging
 
 from services.dashboard_stats_service import dashboard_stats_service
+from auth_middleware import require_authenticated_user
 
-router = APIRouter(prefix="/api/dashboard-stats", tags=["dashboard-stats"])
+# SECURITY (GHSA-3p5c-m5m4-mjpx): this entire router (traffic metrics, backend
+# health, cluster topology, agent status) was mounted without authentication.
+# Require a valid JWT on every route. The frontend Dashboard already sends the
+# operator JWT on these calls, so this is transparent to the UI.
+router = APIRouter(
+    prefix="/api/dashboard-stats",
+    tags=["dashboard-stats"],
+    dependencies=[Depends(require_authenticated_user)],
+)
 logger = logging.getLogger(__name__)
 
 

backend/routers/frontend.py

@@ -117,6 +117,41 @@ def _rule_contradiction_text(rule: Any) -> Optional[str]:
     return None
 
 
+def _pattern_file_warnings(
+    acl_rules: Optional[List[Any]] = None,
+    use_backend_rules: Optional[List[Any]] = None,
+    redirect_rules: Optional[List[Any]] = None,
+) -> List[str]:
+    """Issue #38 follow-up — non-blocking `-f <file>` pattern-file
+    advisory for the manual frontend API.
+
+    The Bulgu #12 hard reject was removed from the Pydantic models:
+    pattern files are operator-managed host files (same policy as the
+    SPOE `filter ... config <path>` reference preserved since v1.8.8)
+    and the agent's pre-reload `haproxy -c` makes a missing file fail
+    safely. This helper returns one warning listing the unique file
+    paths referenced across the rule fields, or [] when no rule uses
+    `-f` — operators who don't use pattern files see no change.
+    """
+    paths: List[str] = []
+    for rules in (acl_rules, use_backend_rules, redirect_rules):
+        for rule in rules or []:
+            text = rule if isinstance(rule, str) else (
+                rule.get("condition") if isinstance(rule, dict) else None)
+            if isinstance(text, str):
+                paths.extend(re.findall(r"(?:^|\s)-f\s+(\S+)", text))
+    if not paths:
+        return []
+    uniq = sorted(set(paths))
+    return [
+        f"ACL/routing rules reference pattern file(s) {', '.join(uniq)}. "
+        f"Each file must exist at that exact path on every HAProxy host "
+        f"in the cluster — HAProxy OpenManager does not create or "
+        f"distribute pattern files. A missing file fails safely at "
+        f"'haproxy -c' (the previous config keeps running)."
+    ]
+
+
 def _collect_routing_rule_contradictions(
     rules: List[Any], origin_label: str,
 ) -> List[Tuple[str, Any]]:
@@ -198,12 +233,28 @@ def _enforce_routing_rule_contradictions(
     for label, rule in conflicts:
         sig = _rule_to_signature(rule)
         if sig and sig in grandfathered_signatures:
+            # Bulgu #83 (round-23 audit) — re-word the operator-
+            # facing warning. The pre-fix message led with
+            # "Grandfathered <label> entry contains a self-
+            # contradictory X !X condition that pre-dated this
+            # validation", which (a) is internal jargon the
+            # operator does not parse, and (b) implies the rule
+            # is OLD when in fact the only thing this branch
+            # knows is that the rule was NOT changed by the
+            # current edit. The operator may well have authored
+            # the rule one minute earlier. State that explicitly
+            # and include the verbatim rule body so the operator
+            # does not have to hunt through the ACL Builder
+            # cards to find the offender.
+            rule_text = rule if isinstance(rule, str) else sig[:160]
             warnings.append(
-                f"Grandfathered {label} entry contains a "
-                f"self-contradictory `X !X` condition that pre-dated "
-                f"this validation. The rule never fires; fix it at "
-                f"your convenience. (rule: "
-                f"{rule if isinstance(rule, str) else sig[:160]})"
+                f"{label}: rule was not modified by this edit but "
+                f"contains a self-contradictory `X !X` condition "
+                f"(`X AND NOT X` is always false, so the rule never "
+                f"fires and traffic falls through to "
+                f"`default_backend`). Your current edit was saved; "
+                f"fix the rule at your convenience. "
+                f"(rule: {rule_text})"
             )
         else:
             blocking.append((label, rule))
@@ -392,6 +443,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -408,6 +460,7 @@ async def get_frontends(
                            ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                            acl_rules, redirect_rules, use_backend_rules,
                            request_headers, response_headers, options, tcp_request_rules,
+                           log_format, filters,
                            timeout_client, timeout_http_request,
                            rate_limit, compression, log_separate, monitor_uri,
                            maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -437,6 +490,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -449,6 +503,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -464,6 +519,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -476,6 +532,7 @@ async def get_frontends(
                                ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni,
                                acl_rules, redirect_rules, use_backend_rules,
                                request_headers, response_headers, options, tcp_request_rules,
+                               log_format, filters,
                                timeout_client, timeout_http_request,
                                rate_limit, compression, log_separate, monitor_uri,
                                maxconn, is_active, created_at, updated_at, cluster_id, last_config_status
@@ -585,6 +642,8 @@ async def get_frontends(
                     "response_headers": f.get("response_headers"),
                     "options": f.get("options"),
                     "tcp_request_rules": f.get("tcp_request_rules"),
+                    "log_format": f.get("log_format"),  # Issue #38
+                    "filters": f.get("filters"),        # Issue #38
                     "timeout_client": f.get("timeout_client"),
                     "timeout_http_request": f.get("timeout_http_request"),
                     "rate_limit": f.get("rate_limit"),
@@ -719,18 +778,18 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 acl_rules, redirect_rules, use_backend_rules,
                 request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
                 rate_limit, compression, log_separate, monitor_uri,
-                cluster_id, maxconn, updated_at
-            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, CURRENT_TIMESTAMP) 
+                cluster_id, maxconn, log_format, filters, updated_at
+            ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, CURRENT_TIMESTAMP)
             RETURNING id
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, frontend.ssl_enabled,
             frontend.ssl_certificate_id, ssl_cert_ids_json, frontend.ssl_port, frontend.ssl_cert_path, frontend.ssl_cert, frontend.ssl_verify,
-            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites, 
+            frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
             frontend.ssl_min_ver, frontend.ssl_max_ver, frontend.ssl_strict_sni,
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters)
         
         # If cluster_id provided, create new config version for agents
         sync_results = []
@@ -809,12 +868,19 @@ async def create_frontend(frontend: FrontendConfig, request: Request, authorizat
                 user_agent=request.headers.get('user-agent')
             )
         
-        return {
+        response: dict = {
             "message": f"Frontend '{frontend.name}' created successfully",
             "id": frontend_id,
             "frontend": frontend.dict(),
             "sync_results": sync_results
         }
+        # Issue #38 follow-up — non-blocking pattern-file advisory
+        # (additive field; absent when no rule references `-f`).
+        pattern_warnings = _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if pattern_warnings:
+            response["warnings"] = pattern_warnings
+        return response
     except HTTPException:
         raise
     except Exception as e:
@@ -1044,9 +1110,10 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                 acl_rules = $20, redirect_rules = $21, use_backend_rules = $22,
                 request_headers = $23, response_headers = $24, options = $25, tcp_request_rules = $26, timeout_client = $27, timeout_http_request = $28,
                 rate_limit = $29, compression = $30, log_separate = $31, monitor_uri = $32,
-                cluster_id = $33, maxconn = $34, updated_at = CURRENT_TIMESTAMP
-            WHERE id = $35
-        """, frontend.name, frontend.bind_address, frontend.bind_port, 
+                cluster_id = $33, maxconn = $34, log_format = $35, filters = $36,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $37
+        """, frontend.name, frontend.bind_address, frontend.bind_port,
             frontend.default_backend, frontend.mode, ssl_enabled,
             ssl_certificate_id, ssl_cert_ids_json, ssl_port, ssl_cert_path, ssl_cert, ssl_verify,
             frontend.ssl_alpn, frontend.ssl_npn, frontend.ssl_ciphers, frontend.ssl_ciphersuites,
@@ -1054,7 +1121,7 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
             json.dumps(frontend.acl_rules or []), json.dumps(frontend.redirect_rules or []), json.dumps(frontend.use_backend_rules or []),
             frontend.request_headers, frontend.response_headers, filtered_options, frontend.tcp_request_rules, frontend.timeout_client, frontend.timeout_http_request,
             frontend.rate_limit, frontend.compression, frontend.log_separate, frontend.monitor_uri,
-            frontend.cluster_id, frontend.maxconn, frontend_id)
+            frontend.cluster_id, frontend.maxconn, frontend.log_format, frontend.filters, frontend_id)
             
         # Debug: Check what was actually saved
         updated_frontend = await conn.fetchrow("""
@@ -1108,6 +1175,8 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
                     "response_headers": frontend.response_headers,
                     "options": filtered_options,
                     "tcp_request_rules": frontend.tcp_request_rules,
+                    "log_format": frontend.log_format,  # Issue #38
+                    "filters": frontend.filters,        # Issue #38
                     "timeout_client": frontend.timeout_client,
                     "timeout_http_request": frontend.timeout_http_request,
                     "rate_limit": frontend.rate_limit,
@@ -1219,8 +1288,12 @@ async def update_frontend(frontend_id: int, frontend: FrontendConfig, request: R
         # yellow toast on the next refresh. The save SUCCEEDED; the
         # warnings only flag latent legacy data the operator may
         # want to clean up at their convenience.
-        if contradiction_warnings:
-            response["warnings"] = contradiction_warnings
+        # Issue #38 follow-up — append the pattern-file advisory to
+        # the same list (additive; empty when no rule uses `-f`).
+        all_warnings = list(contradiction_warnings or []) + _pattern_file_warnings(
+            frontend.acl_rules, frontend.use_backend_rules, frontend.redirect_rules)
+        if all_warnings:
+            response["warnings"] = all_warnings
         return response
     except HTTPException:
         raise

backend/routers/health.py

@@ -3,8 +3,9 @@ Production-Ready Health Check and Monitoring Endpoints
 Provides comprehensive system health monitoring for Kubernetes and production environments
 """
 
-from fastapi import APIRouter, HTTPException
+from fastapi import APIRouter, HTTPException, Depends
 from fastapi.responses import JSONResponse
+from auth_middleware import require_authenticated_user
 import logging
 import asyncio
 import time
@@ -74,7 +75,7 @@ async def readiness_probe():
         logger.error(f"Readiness probe failed: {e}")
         raise HTTPException(status_code=503, detail=f"Not ready: {str(e)}")
 
-@router.get("/deep")
+@router.get("/deep", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks host CPU/mem/disk, DB/redis/python versions, PID
 async def deep_health_check():
     """Comprehensive health check with detailed system information"""
     global _health_cache
@@ -197,7 +198,7 @@ async def deep_health_check():
         
         raise HTTPException(status_code=503, detail=error_response)
 
-@router.get("/agents")
+@router.get("/agents", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks agent names/hostnames
 async def agents_health():
     """Monitor agent connectivity and health status"""
     try:
@@ -261,7 +262,7 @@ async def agents_health():
         logger.error(f"Agent health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Agent health check failed: {str(e)}")
 
-@router.get("/clusters")
+@router.get("/clusters", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): leaks cluster names, HAProxy versions, pending counts
 async def clusters_health():
     """Monitor HAProxy cluster health and configuration status"""
     try:
@@ -329,7 +330,7 @@ async def clusters_health():
         logger.error(f"Cluster health check failed: {e}")
         raise HTTPException(status_code=500, detail=f"Cluster health check failed: {str(e)}")
 
-@router.get("/errors")
+@router.get("/errors", dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): app error metrics, sibling of /deep,/agents,/clusters
 async def error_statistics():
     """Get application error statistics and metrics"""
     try:

backend/routers/letsencrypt.py

@@ -1,6 +1,7 @@
 from fastapi import APIRouter, HTTPException, Header
-from pydantic import BaseModel, Field, field_validator
-from typing import Optional, List
+from pydantic import BaseModel, Field, field_validator, model_validator
+from typing import Optional, List, Dict
+import base64
 import json
 import logging
 import re
@@ -10,6 +11,40 @@ from datetime import datetime
 from database.connection import get_database_connection, close_database_connection
 from services.acme_service import acme_service
 from services.haproxy_config import generate_haproxy_config_for_cluster
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import encrypt_dns_credentials, decrypt_dns_credentials
+
+# Issue #35: DNS-01 challenge methods.
+_CHALLENGE_TYPES = ("http-01", "dns-01")
+
+
+async def _dns01_enabled() -> bool:
+    """Global kill-switch (system_settings acme.dns01_enabled, default False). Read via the ACME
+    settings dict so non-admins never need the admin-only /api/settings/acme endpoint."""
+    try:
+        settings = await acme_service._get_settings()
+        val = settings.get('dns01_enabled')
+        if isinstance(val, str):
+            return val.strip().lower() in ('1', 'true', 'yes', 'on')
+        return bool(val)
+    except Exception:
+        return False
+
+
+# Per-user sliding-window rate limit for the manual dns-confirm action (soft anti-abuse so a user
+# can't spam the CA via the confirm button). Per-process; sufficient for a manual UI action.
+_DNS_CONFIRM_RL: Dict[int, list] = {}
+_DNS_CONFIRM_LIMIT = 5
+_DNS_CONFIRM_WINDOW = 60.0
+
+
+async def _enforce_dns_confirm_rate_limit(user_id: int) -> None:
+    now = time.time()
+    bucket = [t for t in _DNS_CONFIRM_RL.get(user_id, []) if now - t < _DNS_CONFIRM_WINDOW]
+    if len(bucket) >= _DNS_CONFIRM_LIMIT:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded: dns-confirm allowed 5 requests per minute")
+    bucket.append(now)
+    _DNS_CONFIRM_RL[user_id] = bucket
 
 logger = logging.getLogger(__name__)
 
@@ -25,8 +60,81 @@ class AccountCreate(BaseModel):
     email: str
     directory_url: Optional[str] = None
     tos_agreed: bool = True
-    eab_kid: Optional[str] = None
-    eab_hmac_key: Optional[str] = None
+    # EAB (External Account Binding) for CAs that require it (ZeroSSL, Google). The KID is opaque
+    # (bound only); the HMAC key must be base64url so newAccount's _b64url_decode won't raise a
+    # cryptic binascii error (a common copy mistake is standard-base64 '+'/'/' vs urlsafe '-'/'_').
+    eab_kid: Optional[str] = Field(default=None, max_length=256)
+    eab_hmac_key: Optional[str] = Field(default=None, max_length=512)
+    # Issue #35: per-account default challenge method + DNS provider (for dns-01).
+    challenge_type: str = "http-01"
+    dns_provider: Optional[str] = None
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @field_validator('eab_hmac_key')
+    @classmethod
+    def _validate_eab_hmac_key(cls, v):
+        if not v:
+            return v
+        try:
+            base64.urlsafe_b64decode(v + '=' * (-len(v) % 4))
+        except Exception:
+            raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
+        return v
+
+    @field_validator('directory_url')
+    @classmethod
+    def _validate_directory_url(cls, v):
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): reject non-https URLs and literal
+        # non-public IP hosts at the API boundary. The full DNS-based SSRF check
+        # runs at fetch time (acme_service.get_directory -> ssrf_guard).
+        if not v:
+            return v
+        from urllib.parse import urlparse
+        import ipaddress
+        from utils.ssrf_guard import is_public_ip
+        parsed = urlparse(v.strip())
+        if parsed.scheme.lower() != 'https':
+            raise ValueError("directory_url must be an https URL")
+        host = parsed.hostname
+        if not host:
+            raise ValueError("directory_url has no host")
+        try:
+            ipaddress.ip_address(host)
+            is_ip_literal = True
+        except ValueError:
+            is_ip_literal = False
+        if is_ip_literal and not is_public_ip(host):
+            raise ValueError("directory_url must not point to a private/loopback IP address")
+        return v
+
+    @model_validator(mode='after')
+    def _require_provider_for_dns01(self):
+        if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
+            raise ValueError("dns_provider is required when challenge_type is 'dns-01'")
+        return self
+
+
+class DnsCredentialsUpsert(BaseModel):
+    dns_provider: str = Field(..., min_length=1, max_length=50)
+    credentials: Dict[str, str] = Field(default_factory=dict)
+
+    @field_validator('credentials')
+    @classmethod
+    def _validate_credentials(cls, v):
+        if len(v) > 20:
+            raise ValueError("Too many credential fields")
+        for key, val in v.items():
+            if not isinstance(key, str) or not re.match(r'^[a-zA-Z0-9_]{1,50}$', key):
+                raise ValueError(f"Invalid credential field name: {key!r}")
+            if not isinstance(val, str) or len(val) > 4000:
+                raise ValueError(f"Credential value for {key!r} is missing or too long")
+        return v
 
 
 class CertificateRequest(BaseModel):
@@ -38,6 +146,8 @@ class CertificateRequest(BaseModel):
     account_id: Optional[int] = None
     cluster_ids: List[int] = Field(default_factory=list)
     auto_renew: bool = True
+    # Issue #35: optional override; when None the account's default method is used.
+    challenge_type: Optional[str] = None
 
     @field_validator('domains')
     @classmethod
@@ -54,7 +164,30 @@ class CertificateRequest(BaseModel):
             if not _DOMAIN_REGEX.match(d_norm):
                 raise ValueError(f"Invalid domain format: '{d}'")
             normalized.append(d_norm)
-        return normalized
+        # De-duplicate (case/whitespace variants normalize to the same value) while preserving order,
+        # so we don't submit a redundant SAN to the CA or render duplicate-keyed tags in the UI.
+        return list(dict.fromkeys(normalized))
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v is not None and v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @model_validator(mode='after')
+    def _wildcard_requires_dns01(self):
+        # Static cross-field guard: a wildcard SAN can ONLY be issued via dns-01 (the CA rejects
+        # wildcard over http-01). The runtime dns01_enabled gate + provider resolution happen in the
+        # endpoint (validators can't do async/DB). When challenge_type is None here, the effective
+        # method is resolved from the account in the endpoint, which re-checks this.
+        if any((d or '').startswith('*.') for d in (self.domains or [])):
+            # Only reject when the caller EXPLICITLY chose a non-dns-01 method. When challenge_type is
+            # None, the effective method is resolved from the account in the endpoint, which re-checks
+            # wildcard-requires-dns-01 — so account-default dns-01 inheritance still works for wildcards.
+            if self.challenge_type is not None and self.challenge_type != 'dns-01':
+                raise ValueError("Wildcard certificates require challenge_type 'dns-01'")
+        return self
 
 
 # --- Account management ---
@@ -77,7 +210,7 @@ async def list_accounts(authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         rows = await conn.fetch(
-            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at FROM letsencrypt_accounts ORDER BY id"
+            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at, challenge_type, dns_provider FROM letsencrypt_accounts ORDER BY id"
         )
         return [dict(r) for r in rows]
     finally:
@@ -107,16 +240,36 @@ async def create_account(body: AccountCreate, authorization: str = Header(None))
         eab_kid = body.eab_kid or settings.get('eab_kid', '') or None
         eab_hmac_key = body.eab_hmac_key or settings.get('eab_hmac_key', '') or None
 
+        # Issue #35: a dns-01 account must name a supported DNS provider.
+        if body.challenge_type == 'dns-01':
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((body.dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {body.dns_provider}")
+
         result = await acme_service.register_account(
             email=body.email,
             directory_url=directory_url,
             tos_agreed=body.tos_agreed,
             eab_kid=eab_kid,
             eab_hmac_key=eab_hmac_key,
+            challenge_type=body.challenge_type,
+            dns_provider=(body.dns_provider or None),
         )
         return result
+    except HTTPException:
+        # Preserve deliberate status codes (e.g. 409 DNS-01 disabled, 422 unsupported provider) —
+        # the broad except below would otherwise downgrade them all to 400.
+        raise
     except Exception as e:
         logger.error(f"ACME account registration failed: {e}")
+        # Humanize the common EAB-required failure (ZeroSSL/Google). The ACME error propagates as a
+        # string ("Account registration failed: {<dict>}"), so match the URN substring in str(e).
+        if 'externalaccountrequired' in str(e).lower():
+            raise HTTPException(status_code=400, detail=(
+                "This CA requires External Account Binding (EAB). Enter the EAB Key ID and HMAC Key "
+                "from your ZeroSSL/Google account and retry."
+            ))
         raise HTTPException(status_code=400, detail=str(e))
 
 
@@ -186,6 +339,147 @@ async def remove_account(account_id: int, authorization: str = Header(None)):
         await close_database_connection(conn)
 
 
+# --- Issue #35: DNS-01 providers + per-account DNS credentials ---
+
+@router.get("/dns-providers")
+async def get_dns_providers(authorization: str = Header(None)):
+    """List supported DNS providers + their credential-field schema (for the UI). Also returns the
+    global dns01_enabled gate so a non-admin cert UI can read it without the admin-only settings API.
+    Authenticated (any user); not admin-only."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    return {"dns01_enabled": await _dns01_enabled(), "providers": list_providers()}
+
+
+@router.get("/accounts/{account_id}/dns-credentials")
+async def get_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Masked metadata only — provider + which credential fields are set + updated_at. NEVER returns
+    the ciphertext or any plaintext token. Read-only for any authenticated user (matches list_accounts)."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            "SELECT dns_provider, credentials_encrypted, updated_at FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+            account_id,
+        )
+        if not row:
+            return {"configured": False, "dns_provider": None, "credential_fields_present": [], "updated_at": None}
+        present = []
+        decrypted = decrypt_dns_credentials(row["credentials_encrypted"])
+        if isinstance(decrypted, dict):
+            present = sorted(decrypted.keys())
+        return {
+            "configured": True,
+            "dns_provider": row["dns_provider"],
+            "credential_fields_present": present,
+            "updated_at": row["updated_at"],
+        }
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/accounts/{account_id}/dns-credentials")
+async def upsert_dns_credentials(account_id: int, body: DnsCredentialsUpsert, authorization: str = Header(None)):
+    """Store (encrypted) DNS provider credentials for an account. Admin-only. Verifies the
+    credentials against the provider BEFORE persisting; returns a sanitized result (never the token)."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    provider_name = body.dns_provider.strip()
+    if not is_supported(provider_name):
+        raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {provider_name}")
+
+    conn = await get_database_connection()
+    try:
+        exists = await conn.fetchval("SELECT 1 FROM letsencrypt_accounts WHERE id = $1", account_id)
+        if not exists:
+            raise HTTPException(status_code=404, detail="ACME account not found")
+
+        # Verify credentials synchronously; only persist on success. The detail is user-safe.
+        try:
+            provider = get_provider(provider_name, dict(body.credentials))
+            verify = await provider.verify_credentials()
+        except DnsProviderError as exc:
+            raise HTTPException(status_code=422, detail=str(exc))
+        except HTTPException:
+            raise
+        except Exception:
+            # Defensive: never let a provider-internal exception string (which could echo creds in a
+            # future provider) reach the client. Always a sanitized 422.
+            raise HTTPException(status_code=422, detail="DNS provider credential verification failed")
+        if not verify.get("ok"):
+            raise HTTPException(status_code=422, detail=verify.get("detail") or "DNS provider credential verification failed")
+
+        token = encrypt_dns_credentials(dict(body.credentials))
+        await conn.execute(
+            """INSERT INTO letsencrypt_account_dns_credentials (account_id, dns_provider, credentials_encrypted, updated_at)
+               VALUES ($1, $2, $3, NOW())
+               ON CONFLICT (account_id) DO UPDATE SET
+                   dns_provider = EXCLUDED.dns_provider,
+                   credentials_encrypted = EXCLUDED.credentials_encrypted,
+                   updated_at = NOW()""",
+            account_id, provider_name, token,
+        )
+        # Keep the account's provider selection in sync.
+        await conn.execute(
+            "UPDATE letsencrypt_accounts SET dns_provider = $1, updated_at = NOW() WHERE id = $2",
+            provider_name, account_id,
+        )
+        return {"ok": True, "dns_provider": provider_name, "detail": verify.get("detail", "Credentials stored.")}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/accounts/{account_id}/dns-credentials")
+async def delete_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Remove an account's stored DNS credentials. Admin-only."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    conn = await get_database_connection()
+    try:
+        await conn.execute("DELETE FROM letsencrypt_account_dns_credentials WHERE account_id = $1", account_id)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/orders/{order_id}/dns-confirm")
+async def confirm_dns_order(order_id: int, authorization: str = Header(None)):
+    """Manual DNS-01 only: the user asserts the TXT record is published; tell the CA to validate.
+    Requires ssl.create + a per-user rate limit; acts only on a dns-01 + manual + pending order."""
+    from auth_middleware import get_current_user_from_token, check_user_permission
+    current_user = await get_current_user_from_token(authorization)
+    has_perm = await check_user_permission(current_user['id'], 'ssl', 'create')
+    if not has_perm:
+        raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.create required")
+    await _enforce_dns_confirm_rate_limit(current_user['id'])
+
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+    finally:
+        await close_database_connection(conn)
+    if not order:
+        raise HTTPException(status_code=404, detail="Order not found")
+    if order['challenge_type'] != 'dns-01' or (order['dns_provider'] or 'manual') != 'manual':
+        raise HTTPException(status_code=409, detail="This order is not a manual DNS-01 order")
+    if order['status'] not in ('pending', 'processing'):
+        raise HTTPException(status_code=409, detail=f"Order is '{order['status']}' and cannot be confirmed")
+
+    from services.dns01_orchestrator import confirm_manual_dns01
+    await confirm_manual_dns01(order_id)
+    return {"ok": True, "message": "DNS-01 confirmation submitted; the CA will validate shortly."}
+
+
 # --- Certificate operations ---
 
 @router.post("/certificates")
@@ -222,32 +516,61 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
         logger.info(f"ACME: Using account_id={account_id} for certificate request")
 
         warnings = []
-        # Audit Tur 5 / Commit 8c: empty cluster_ids in UI means "global certificate".
-        # Resolve to all ACME-enabled active clusters; only fail if NONE exist.
+
+        # Issue #35: resolve the effective challenge method (request override, else account default).
+        conn_acct = await get_database_connection()
+        try:
+            acct = await conn_acct.fetchrow(
+                "SELECT challenge_type, dns_provider FROM letsencrypt_accounts WHERE id = $1", account_id
+            )
+        finally:
+            await close_database_connection(conn_acct)
+        effective_challenge = (body.challenge_type or (acct['challenge_type'] if acct else None) or 'http-01')
+        dns_provider = (acct['dns_provider'] if acct else None)
+        is_dns01 = (effective_challenge == 'dns-01')
+        has_wildcard = any(d.startswith('*.') for d in body.domains)
+
+        if is_dns01:
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail="The selected ACME account has no DNS provider configured for DNS-01.")
+            if (dns_provider or 'manual') == 'manual':
+                # Manual DNS-01 cannot be renewed unattended; auto-renew is forced off on the issued
+                # certificate (see _complete_certificate). Tell the requester so it isn't a surprise.
+                warnings.append(
+                    "Manual DNS-01 certificates cannot auto-renew unattended. Auto-renew will be disabled; "
+                    "re-publish the TXT record and request renewal before expiry."
+                )
+        elif has_wildcard:
+            raise HTTPException(status_code=422, detail="Wildcard certificates require a DNS-01 account.")
+
+        # Empty cluster_ids = "global certificate". For DNS-01 no ACME Challenge Routing / port 80 is
+        # needed, so resolve to ALL active clusters; http-01 still requires acme_enabled clusters.
         if not body.cluster_ids:
             conn_resolve = await get_database_connection()
             try:
-                acme_clusters_resolved = await conn_resolve.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
-                )
+                if is_dns01:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE")
             finally:
                 await close_database_connection(conn_resolve)
-            if not acme_clusters_resolved:
+            if not resolved:
+                if is_dns01:
+                    raise HTTPException(status_code=422, detail="Cannot issue certificate: no active clusters configured.")
                 raise HTTPException(
                     status_code=422,
                     detail="Cannot issue certificate: no ACME-enabled clusters configured. "
                            "Enable ACME Challenge Routing on at least one cluster in Cluster Management, "
                            "Apply the configuration change, then retry."
                 )
-            body.cluster_ids = [c['id'] for c in acme_clusters_resolved]
+            body.cluster_ids = [c['id'] for c in resolved]
             warnings.append(
-                f"No clusters specified — applied to all ACME-enabled cluster(s) ({len(body.cluster_ids)})"
+                f"No clusters specified — applied to all {'active' if is_dns01 else 'ACME-enabled'} cluster(s) ({len(body.cluster_ids)})"
             )
-            logger.warning(
-                f"ACME: Empty cluster_ids → global cert fallback to {len(body.cluster_ids)} ACME-enabled cluster(s)"
-            )
-        else:
-            # Validate that referenced clusters exist + are ACME-enabled (warn-only).
+        elif not is_dns01:
+            # http-01 only: warn if no cluster has ACME Challenge Routing enabled.
             try:
                 conn_warn = await get_database_connection()
                 try:
@@ -255,7 +578,6 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
                         "SELECT COUNT(*) FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
                     )
                     if acme_clusters == 0:
-                        logger.warning("ACME: No clusters with ACME Challenge Routing enabled - certificate validation will likely fail")
                         warnings.append(
                             "No clusters have ACME Challenge Routing enabled. "
                             "Certificate validation will fail. Enable it in Cluster Management and Apply Changes first."
@@ -269,14 +591,30 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             account_id=account_id,
             domains=body.domains,
             cluster_ids=body.cluster_ids,
+            challenge_type=effective_challenge,
+            created_by=current_user['id'],
         )
 
-        challenges = await acme_service.respond_to_challenges(order['order_id'])
+        # Audit trail: record who requested the certificate + the method (esp. for DNS-01/wildcard,
+        # which has a wider blast radius than http-01). Never raises into the request path.
+        try:
+            from utils.activity_log import record_event
+            await record_event(
+                order['order_id'], "acme.order.requested",
+                message=f"Certificate requested ({effective_challenge}) by user {current_user['id']}",
+                details={"user_id": current_user['id'], "challenge_type": effective_challenge,
+                         "dns_provider": dns_provider, "domains": body.domains},
+            )
+        except Exception:
+            pass
+
+        # http-01 posts the challenge response immediately (token served continuously). dns-01 is
+        # driven by the orchestrator AFTER the TXT is published (manual waits for dns-confirm), so we
+        # must NOT respond here.
+        challenges = []
+        if not is_dns01:
+            challenges = await acme_service.respond_to_challenges(order['order_id'])
 
-        # Commit 5b: re-fetch the order's *current* status from DB. The status
-        # returned by create_order() reflects the moment of creation; after
-        # respond_to_challenges() the CA may have already advanced it (e.g. to
-        # 'processing'). Surfacing stale status leads UI to under-poll.
         conn_status = await get_database_connection()
         try:
             fresh_status = await conn_status.fetchval(
@@ -287,14 +625,23 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             await close_database_connection(conn_status)
         effective_status = fresh_status or order['status']
 
-        logger.info(f"ACME: Order {order['order_id']} created, {len(challenges)} challenge(s) posted, status={effective_status}")
+        if is_dns01:
+            msg = ("Order created. Publish the DNS TXT record shown for each domain, then confirm."
+                   if dns_provider == 'manual'
+                   else "Order created. The DNS TXT record(s) will be published automatically; waiting for CA validation.")
+        else:
+            msg = "Order created. ACME challenges have been posted. Waiting for CA validation."
+
+        logger.info(f"ACME: Order {order['order_id']} created ({effective_challenge}), status={effective_status}")
 
         return {
             "order_id": order['order_id'],
             "status": effective_status,
             "domains": body.domains,
+            "challenge_type": effective_challenge,
+            "dns_provider": dns_provider,
             "challenges": challenges,
-            "message": "Order created. ACME challenges have been posted. Waiting for CA validation.",
+            "message": msg,
             "warnings": warnings,
         }
     except HTTPException:
@@ -316,7 +663,8 @@ async def list_orders(authorization: str = Header(None)):
         rows = await conn.fetch("""
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.ssl_certificate_id, o.cluster_ids, o.error_detail,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             ORDER BY o.created_at DESC
@@ -345,7 +693,8 @@ async def get_order(order_id: int, authorization: str = Header(None)):
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.certificate_url, o.finalize_url, o.expires_at,
                    o.error_detail, o.ssl_certificate_id, o.cluster_ids,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             WHERE o.id = $1
@@ -353,13 +702,23 @@ async def get_order(order_id: int, authorization: str = Header(None)):
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
 
+        # Issue #35: include challenge_type + dns_txt_value (PUBLIC DNS data — NOT key_authorization,
+        # NOT the API token) so the UI can render manual DNS-01 instructions. Explicit column list.
         challenges = await conn.fetch(
-            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
+            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at, "
+            "challenge_type, dns_txt_value FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
         )
         result = dict(order)
         result['domains'] = json.loads(result['domains']) if isinstance(result['domains'], str) else result['domains']
         result['cluster_ids'] = json.loads(result['cluster_ids']) if isinstance(result['cluster_ids'], str) else result['cluster_ids']
-        result['challenges'] = [dict(c) for c in challenges]
+        ch_list = []
+        for c in challenges:
+            cd = dict(c)
+            if cd.get('challenge_type') == 'dns-01':
+                # Server computes the record name (wildcard *.-stripping lives server-side).
+                cd['dns_record_name'] = acme_service._challenge_dns_name(cd['domain'])
+            ch_list.append(cd)
+        result['challenges'] = ch_list
         return result
     finally:
         await close_database_connection(conn)
@@ -470,18 +829,23 @@ async def renew_order(order_id: int, authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         order = await conn.fetchrow(
-            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1", order_id
+            "SELECT account_id, domains, cluster_ids, challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
         )
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
         domains = json.loads(order['domains']) if isinstance(order['domains'], str) else order['domains']
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
+        challenge_type = order['challenge_type'] or 'http-01'
 
         new_order = await acme_service.create_order(
-            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids
+            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids,
+            challenge_type=challenge_type, created_by=current_user['id'],
         )
-        challenges = await acme_service.respond_to_challenges(new_order['order_id'])
-        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenges": challenges}
+        # dns-01 is driven by the orchestrator after the TXT is published; only http-01 responds here.
+        challenges = []
+        if challenge_type != 'dns-01':
+            challenges = await acme_service.respond_to_challenges(new_order['order_id'])
+        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenge_type": challenge_type, "challenges": challenges}
     finally:
         await close_database_connection(conn)
 
@@ -611,11 +975,17 @@ async def get_renewal_schedule(authorization: str = Header(None)):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
     conn = await get_database_connection()
     try:
+        # Issue #35: expose the challenge method/provider (via the originating order/account) so the
+        # UI can distinguish manual DNS-01 certs, which cannot auto-renew unattended. LEFT JOINs keep
+        # legacy certs (no order link / pre-DNS-01 columns) rendering as http-01.
         certs = await conn.fetch("""
-            SELECT id, name, primary_domain, expiry_date, auto_renew, days_until_expiry
-            FROM ssl_certificates
-            WHERE source = 'letsencrypt' AND is_active = TRUE
-            ORDER BY expiry_date ASC NULLS LAST
+            SELECT c.id, c.name, c.primary_domain, c.expiry_date, c.auto_renew, c.days_until_expiry,
+                   o.challenge_type, a.dns_provider
+            FROM ssl_certificates c
+            LEFT JOIN letsencrypt_orders o ON o.id = c.letsencrypt_order_id
+            LEFT JOIN letsencrypt_accounts a ON a.id = o.account_id
+            WHERE c.source = 'letsencrypt' AND c.is_active = TRUE
+            ORDER BY c.expiry_date ASC NULLS LAST
         """)
         return [dict(c) for c in certs]
     finally:
@@ -663,6 +1033,18 @@ async def _complete_certificate(order_id: int) -> dict:
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
         primary_domain = domains[0] if domains else 'unknown'
 
+        # Issue #35: a manual DNS-01 certificate cannot be auto-renewed unattended (the renewal task
+        # skips it — see main.py), so persist auto_renew=FALSE rather than storing a misleading
+        # "Enabled" that the user trusts while the cert silently expires. http-01 and automated
+        # DNS-01 (e.g. Cloudflare) keep auto_renew=TRUE, preserving existing behaviour.
+        auto_renew_value = True
+        if order.get('challenge_type') == 'dns-01':
+            acct_provider = await conn.fetchval(
+                "SELECT dns_provider FROM letsencrypt_accounts WHERE id = $1", order['account_id']
+            )
+            if (acct_provider or 'manual') == 'manual':
+                auto_renew_value = False
+
         # Commit 5g: guard against empty cert_private_key.
         # Inserting an SSL certificate row with an empty private_key would silently
         # produce an unusable certificate (HAProxy would fail to load on Apply, or
@@ -749,12 +1131,13 @@ async def _complete_certificate(order_id: int) -> dict:
                     certificate_content = $1, private_key_content = $2, chain_content = $3,
                     all_domains = $4::jsonb, expiry_date = $5, days_until_expiry = $6,
                     issuer = $7, fingerprint = $8, letsencrypt_order_id = $9,
-                    auto_renew = TRUE, is_active = TRUE, last_config_status = 'PENDING',
+                    auto_renew = $11, is_active = TRUE, last_config_status = 'PENDING',
                     updated_at = NOW()
                 WHERE id = $10
             """, cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id,
+                auto_renew_value)
             logger.info(f"ACME RENEWAL: Updated certificate {cert_id} for {primary_domain}, status=PENDING")
         else:
             cert_row = await conn.fetchrow("""
@@ -764,11 +1147,12 @@ async def _complete_certificate(order_id: int) -> dict:
                      usage_type, source, letsencrypt_order_id, auto_renew, is_active,
                      last_config_status)
                 VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10,
-                        'frontend', 'letsencrypt', $11, TRUE, TRUE, 'PENDING')
+                        'frontend', 'letsencrypt', $11, $12, TRUE, 'PENDING')
                 RETURNING id
             """, f"le-{primary_domain}", cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), primary_domain, all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id,
+                auto_renew_value)
             cert_id = cert_row['id']
 
         await conn.execute(
@@ -815,12 +1199,18 @@ async def _complete_certificate(order_id: int) -> dict:
             if mapped:
                 effective_cluster_ids = [r['cluster_id'] for r in mapped]
             else:
-                # Audit Tur 6 / Commit 5j: only fall back to ACME-enabled clusters.
-                # Applying renewal to ACME-disabled clusters could re-introduce Issue #11
-                # patterns and risks deploying certs to clusters where they can't be renewed.
-                all_clusters = await conn.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                # Audit Tur 6 / Commit 5j: http-01 only falls back to ACME-enabled clusters.
+                # Issue #35: a dns-01 cert needs NO challenge routing / port 80, so it can renew on
+                # any active cluster — resolve to all active clusters for dns-01.
+                ch_type = await conn.fetchval(
+                    "SELECT challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
                 )
+                if ch_type == 'dns-01':
+                    all_clusters = await conn.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    all_clusters = await conn.fetch(
+                        "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                    )
                 effective_cluster_ids = [r['id'] for r in all_clusters]
             if effective_cluster_ids:
                 logger.info(f"ACME RENEWAL: Resolved {len(effective_cluster_ids)} cluster(s) for global cert {cert_id}")

backend/routers/mfa.py

@@ -0,0 +1,727 @@
+"""MFA router — Issue #18, v1.6.0.
+
+All endpoints are additive; nothing here breaks existing JWT or apply_service flows.
+Authentication is JWT-based (Bearer). Admin endpoints additionally require
+``users.is_admin == True`` (canonical super-admin flag).
+"""
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from middleware.mfa_rate_limit_key import mfa_rate_limit_key
+from middleware.mfa_rate_limits import MFA_LIMITS
+from middleware.rate_limiter import limiter
+from models.mfa import (
+    MfaAdminResetAllRequest,
+    MfaAdminResetRequest,
+    MfaDisableRequest,
+    MfaEnrollConfirmRequest,
+    MfaEnrollConfirmResponse,
+    MfaEnrollStartResponse,
+    MfaRegenerateBackupRequest,
+    MfaRegenerateBackupResponse,
+    MfaStatusResponse,
+)
+from services import mfa_service
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/api/mfa", tags=["MFA"])
+
+# Lifecycle constants (Plan section 5)
+PENDING_ENROLLMENT_TTL_SECONDS = 600  # 10 minutes — QR scan + verify window
+PENDING_ENROLLMENT_MAX_ATTEMPTS = 5
+
+
+# ---------------------------------------------------------------------------
+# Authentication helpers
+# ---------------------------------------------------------------------------
+
+
+async def _require_user(authorization: Optional[str]) -> dict:
+    user = await get_current_user_from_token(authorization)
+    if not user:
+        raise HTTPException(status_code=401, detail="Not authenticated")
+    return user
+
+
+async def _require_admin(authorization: Optional[str]) -> dict:
+    user = await _require_user(authorization)
+    if not user.get("is_admin"):
+        raise HTTPException(status_code=403, detail="Admin privileges required")
+    return user
+
+
+async def _cleanup_expired_pending_enrollments(conn) -> None:
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_enrollments WHERE expires_at < NOW()"
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-enrollment cleanup skipped: {exc}")
+
+
+# ---------------------------------------------------------------------------
+# Self status
+# ---------------------------------------------------------------------------
+
+
+@router.get(
+    "/status",
+    summary="MFA status for the authenticated user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_status(authorization: str = Header(None)):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            current["id"],
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            current["id"],
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+@router.get(
+    "/admin/status/{user_id}",
+    summary="Admin: MFA status of any user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_admin_status(user_id: int, authorization: str = Header(None)):
+    await _require_admin(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            user_id,
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Enrollment
+# ---------------------------------------------------------------------------
+
+
+@router.post(
+    "/enroll/start",
+    summary="Begin TOTP enrollment (returns secret + otpauth URI)",
+    response_model=MfaEnrollStartResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_start, key_func=mfa_rate_limit_key)
+async def mfa_enroll_start(request: Request, authorization: str = Header(None)):
+    current = await _require_user(authorization)
+
+    # Round 7 audit fix — REFUSE re-enrollment if the user is already MFA-on.
+    # Without this guard a stolen JWT could silently rotate the victim's TOTP
+    # secret + invalidate all their backup codes via /enroll/start ->
+    # /enroll/confirm (overwriting `users.mfa_secret_encrypted` and replacing
+    # `mfa_backup_codes`). To re-enroll, the user must first call /api/mfa/disable
+    # (which requires a fresh TOTP) or an admin must run /api/mfa/admin-reset.
+    secret_plain = mfa_service.generate_totp_secret()
+    secret_encrypted = mfa_service.encrypt_secret(secret_plain)
+
+    conn = await get_database_connection()
+    blocked = False
+    try:
+        # Single transaction with SELECT FOR UPDATE closes the TOCTOU window
+        # between the mfa_enabled check and the pending_enrollment upsert.
+        async with conn.transaction():
+            row = await conn.fetchrow(
+                "SELECT mfa_enabled FROM users WHERE id = $1 FOR UPDATE",
+                current["id"],
+            )
+            if row and row["mfa_enabled"]:
+                blocked = True
+            else:
+                await _cleanup_expired_pending_enrollments(conn)
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_enrollments
+                        (user_id, secret_encrypted, attempts, expires_at)
+                    VALUES ($1, $2, 0, NOW() + ($3 || ' seconds')::interval)
+                    ON CONFLICT (user_id) DO UPDATE
+                      SET secret_encrypted = EXCLUDED.secret_encrypted,
+                          attempts = 0,
+                          expires_at = EXCLUDED.expires_at,
+                          created_at = CURRENT_TIMESTAMP
+                    """,
+                    current["id"],
+                    secret_encrypted,
+                    str(PENDING_ENROLLMENT_TTL_SECONDS),
+                )
+    finally:
+        await close_database_connection(conn)
+
+    if blocked:
+        raise HTTPException(
+            status_code=400,
+            detail="MFA is already enabled. Disable it first (via /api/mfa/disable or admin reset) to re-enroll.",
+        )
+
+    hostname_hint = request.url.hostname if request.url else None
+    label = mfa_service.build_account_label(current["username"], hostname_hint)
+    otpauth_uri = mfa_service.build_otpauth_uri(label, secret_plain)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.started",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"secret_len": len(secret_plain)},  # NEVER log the secret itself
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+
+    return MfaEnrollStartResponse(
+        secret=secret_plain,
+        otpauth_uri=otpauth_uri,
+        expires_in=PENDING_ENROLLMENT_TTL_SECONDS,
+    )
+
+
+@router.post(
+    "/enroll/confirm",
+    summary="Confirm enrollment with a TOTP code; returns 10 backup codes once",
+    response_model=MfaEnrollConfirmResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_confirm, key_func=mfa_rate_limit_key)
+async def mfa_enroll_confirm(
+    payload: MfaEnrollConfirmRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # Pre-generate plain codes & hashes outside the DB transaction so we
+    # never hold a row lock for ~2-3s of bcrypt work.
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    # failure: { reason, attempts, http_status, detail }; success when None.
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            # Lock the pending row so concurrent /enroll/confirm calls for the
+            # same user can't both consume the same pending enrollment.
+            pending = await conn.fetchrow(
+                """
+                SELECT secret_encrypted, attempts, expires_at
+                  FROM mfa_pending_enrollments
+                 WHERE user_id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+
+            if not pending:
+                failure = {
+                    "reason": "no_pending",
+                    "http_status": 410,
+                    "detail": "No pending enrollment; start again",
+                }
+            else:
+                from datetime import datetime as _dt
+                if pending["expires_at"] and pending["expires_at"] < _dt.utcnow():
+                    await conn.execute(
+                        "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                        current["id"],
+                    )
+                    failure = {
+                        "reason": "expired",
+                        "http_status": 410,
+                        "detail": "Enrollment expired; start again",
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(pending["secret_encrypted"])
+                    if not secret_plain:
+                        await conn.execute(
+                            "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                            current["id"],
+                        )
+                        failure = {
+                            "reason": "unreadable",
+                            "http_status": 500,
+                            "detail": "Pending enrollment unreadable; start again",
+                        }
+                    else:
+                        ok, _step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, None
+                        )
+                        if not ok:
+                            new_attempts = (pending["attempts"] or 0) + 1
+                            if new_attempts >= PENDING_ENROLLMENT_MAX_ATTEMPTS:
+                                await conn.execute(
+                                    "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "too_many_attempts",
+                                    "attempts": new_attempts,
+                                    "http_status": 410,
+                                    "detail": "Enrollment invalidated; start again",
+                                }
+                            else:
+                                await conn.execute(
+                                    "UPDATE mfa_pending_enrollments SET attempts = $1 WHERE user_id = $2",
+                                    new_attempts,
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "invalid_code",
+                                    "attempts": new_attempts,
+                                    "http_status": 401,
+                                    "detail": "Invalid code",
+                                }
+                        else:
+                            # Verified — finalize state inside the transaction.
+                            await conn.execute(
+                                """
+                                UPDATE users
+                                   SET mfa_enabled = TRUE,
+                                       mfa_method = 'totp',
+                                       mfa_secret_encrypted = $1,
+                                       mfa_enrolled_at = NOW(),
+                                       mfa_last_used_totp_step = NULL,
+                                       mfa_last_used_at = NULL
+                                 WHERE id = $2
+                                """,
+                                pending["secret_encrypted"],
+                                current["id"],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                current["id"],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        # Log AFTER commit so the audit row reflects what actually persisted.
+        await log_user_activity(
+            user_id=current["id"],
+            action="mfa.enrollment.failed",
+            resource_type="mfa",
+            resource_id=str(current["id"]),
+            details={
+                "reason": failure["reason"],
+                "attempts": failure.get("attempts"),
+            },
+            ip_address=ip,
+            user_agent=ua,
+        )
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.confirmed",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"method": "totp"},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaEnrollConfirmResponse(enabled=True, backup_codes=plain_codes, method="totp")
+
+
+# ---------------------------------------------------------------------------
+# Disable + regenerate
+# ---------------------------------------------------------------------------
+
+
+async def _verify_user_code(conn, user_row: dict, code: str) -> Optional[str]:
+    """Verify a TOTP-or-backup code against the user's stored secret.
+
+    Returns the method used ('totp' / 'backup') on success, None on failure.
+
+    On TOTP success the step counter is bumped *atomically* — the UPDATE
+    only succeeds if no other request consumed the same (or a newer) step
+    in between. On backup success the consumed row's used_at is set with
+    an atomic ``WHERE used_at IS NULL RETURNING id`` pattern.
+    """
+    secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+    if secret_plain:
+        ok, step = mfa_service.verify_totp_with_replay_guard(
+            secret_plain, code, user_row["mfa_last_used_totp_step"]
+        )
+        if ok:
+            bumped = await conn.fetchval(
+                """
+                UPDATE users
+                   SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                 WHERE id = $2
+                   AND (mfa_last_used_totp_step IS NULL
+                        OR mfa_last_used_totp_step < $1)
+                 RETURNING id
+                """,
+                step,
+                user_row["id"],
+            )
+            if bumped:
+                return "totp"
+
+    rows = await conn.fetch(
+        """
+        SELECT id, code_hash FROM mfa_backup_codes
+         WHERE user_id = $1 AND used_at IS NULL
+        """,
+        user_row["id"],
+    )
+    for row in rows:
+        if await mfa_service.check_backup_code(code, row["code_hash"]):
+            consumed = await conn.fetchval(
+                """
+                UPDATE mfa_backup_codes
+                   SET used_at = NOW()
+                 WHERE id = $1 AND used_at IS NULL
+                 RETURNING id
+                """,
+                row["id"],
+            )
+            if consumed:
+                return "backup"
+    return None
+
+
+@router.post("/disable", summary="Disable MFA (requires current TOTP or backup)")
+@limiter.limit(MFA_LIMITS.disable, key_func=mfa_rate_limit_key)
+async def mfa_disable(
+    payload: MfaDisableRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        user_row = await conn.fetchrow(
+            """
+            SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users WHERE id = $1
+            """,
+            current["id"],
+        )
+        if not user_row or not user_row["mfa_enabled"]:
+            raise HTTPException(status_code=400, detail="MFA is not enabled")
+
+        method_used = await _verify_user_code(conn, dict(user_row), payload.code)
+        if not method_used:
+            await log_user_activity(
+                user_id=current["id"],
+                action="mfa.disable.failed",
+                resource_type="mfa",
+                resource_id=str(current["id"]),
+                details={"reason": "invalid_code"},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get("user-agent"),
+            )
+            raise HTTPException(status_code=401, detail="Invalid code")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                current["id"],
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", current["id"]
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.disabled.self",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"verified_via": method_used},
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"enabled": False}
+
+
+@router.post(
+    "/backup-codes/regenerate",
+    summary="Issue 10 fresh backup codes (TOTP required)",
+    response_model=MfaRegenerateBackupResponse,
+)
+@limiter.limit(MFA_LIMITS.regenerate_backup_codes, key_func=mfa_rate_limit_key)
+async def mfa_regenerate_backup_codes(
+    payload: MfaRegenerateBackupRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # bcrypt-hash the new codes outside the DB transaction (~2-3s of CPU work).
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            user_row = await conn.fetchrow(
+                """
+                SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+                  FROM users WHERE id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+            if not user_row or not user_row["mfa_enabled"]:
+                failure = {"http_status": 400, "detail": "MFA is not enabled"}
+            else:
+                secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+                if not secret_plain:
+                    failure = {"http_status": 500, "detail": "MFA secret unreadable"}
+                else:
+                    ok, step = mfa_service.verify_totp_with_replay_guard(
+                        secret_plain, payload.code, user_row["mfa_last_used_totp_step"]
+                    )
+                    if not ok:
+                        failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                    else:
+                        bumped = await conn.fetchval(
+                            """
+                            UPDATE users
+                               SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                             WHERE id = $2
+                               AND (mfa_last_used_totp_step IS NULL
+                                    OR mfa_last_used_totp_step < $1)
+                             RETURNING id
+                            """,
+                            step,
+                            current["id"],
+                        )
+                        if not bumped:
+                            failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                        else:
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.backup_codes.regenerated",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"codes_count": len(plain_codes)},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaRegenerateBackupResponse(backup_codes=plain_codes)
+
+
+# ---------------------------------------------------------------------------
+# Admin operations
+# ---------------------------------------------------------------------------
+
+
+@router.post("/admin-reset/{user_id}", summary="Admin: reset a single user's MFA")
+@limiter.limit(MFA_LIMITS.admin_reset, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset(
+    user_id: int,
+    payload: MfaAdminResetRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    if user_id == admin["id"]:
+        raise HTTPException(status_code=400, detail="Use /api/mfa/disable for self-reset")
+
+    conn = await get_database_connection()
+    try:
+        target = await conn.fetchrow(
+            "SELECT id, username, mfa_enabled FROM users WHERE id = $1", user_id
+        )
+        if not target:
+            raise HTTPException(status_code=404, detail="User not found")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                user_id,
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", user_id
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_reset",
+        resource_type="mfa",
+        resource_id=str(user_id),
+        details={
+            "target_user_id": user_id,
+            "target_username": target["username"],
+            "admin_user_id": admin["id"],
+            "reason": payload.reason,
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset": True, "user_id": user_id}
+
+
+@router.post(
+    "/admin-reset-all",
+    summary="Admin: emergency reset of MFA for all users (double confirm)",
+)
+@limiter.limit(MFA_LIMITS.admin_reset_all, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset_all(
+    payload: MfaAdminResetAllRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    # Pydantic's Literal already enforces the magic string, but check defensively too.
+    if payload.confirm != "RESET ALL MFA":
+        raise HTTPException(status_code=400, detail="Invalid confirmation string")
+
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            reset_count = await conn.fetchval(
+                """
+                WITH affected AS (
+                    UPDATE users
+                       SET mfa_enabled = FALSE,
+                           mfa_method = NULL,
+                           mfa_secret_encrypted = NULL,
+                           mfa_enrolled_at = NULL,
+                           mfa_last_used_at = NULL,
+                           mfa_last_used_totp_step = NULL
+                     WHERE mfa_enabled = TRUE
+                     RETURNING id
+                )
+                SELECT COUNT(*) FROM affected
+                """
+            )
+            await conn.execute("DELETE FROM mfa_backup_codes")
+            await conn.execute("DELETE FROM mfa_pending_logins")
+            await conn.execute("DELETE FROM mfa_pending_enrollments")
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_bulk_reset",
+        resource_type="mfa",
+        resource_id=str(admin["id"]),
+        details={
+            "reset_count": int(reset_count or 0),
+            "reason": payload.reason,
+            "admin_user_id": admin["id"],
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset_count": int(reset_count or 0)}

backend/routers/settings.py

@@ -1,10 +1,12 @@
 from fastapi import APIRouter, HTTPException, Header
 from pydantic import BaseModel
 from typing import Dict, Any
+import json
 import logging
 from datetime import datetime
 
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import AcmeBackendUrlError, validate_acme_backend_url
 
 logger = logging.getLogger(__name__)
 
@@ -50,6 +52,39 @@ async def get_settings_by_category(category: str, authorization: str = Header(No
         await close_database_connection(conn)
 
 
+def _validate_acme_challenge_backend_url(value):
+    """Validate `acme.challenge_backend_url` exactly as the config renderer reads it.
+
+    Settings values are stored as jsonb, so the renderer json.loads them before use
+    (services/haproxy_config.py). Validating the raw column text instead of the
+    decoded string would check the quoting rather than the URL.
+    """
+    decoded = value
+    if isinstance(decoded, str):
+        try:
+            decoded = json.loads(decoded)
+        except (json.JSONDecodeError, TypeError):
+            pass
+    if decoded is None:
+        return
+    try:
+        validate_acme_backend_url(str(decoded))
+    except AcmeBackendUrlError as exc:
+        raise HTTPException(
+            status_code=422,
+            detail=f"acme.challenge_backend_url: {exc}",
+        ) from None
+
+
+# Per-key validators for settings that end up in generated configuration or in
+# outbound requests. Everything else is still written through unchecked; this is a
+# deliberate allow-list of the keys where a bad value causes silent breakage rather
+# than an obvious one.
+_SETTING_VALIDATORS = {
+    "acme.challenge_backend_url": _validate_acme_challenge_backend_url,
+}
+
+
 @router.put("/{category}")
 async def update_settings_by_category(
     category: str,
@@ -59,6 +94,13 @@ async def update_settings_by_category(
     current_user = await _get_admin_user(authorization)
     conn = await get_database_connection()
     try:
+        # Validate the whole batch before writing any of it, so a rejected key cannot
+        # leave the category half-applied.
+        for key_suffix, value in body.settings.items():
+            validator = _SETTING_VALIDATORS.get(f"{category}.{key_suffix}")
+            if validator is not None:
+                validator(value)
+
         updated = []
         for key_suffix, value in body.settings.items():
             full_key = f"{category}.{key_suffix}"
@@ -104,16 +146,40 @@ async def test_acme_connection(authorization: str = Header(None), directory_url:
         finally:
             await close_database_connection(conn)
 
+    # SECURITY (GHSA-3vh4-gvxx-wm2p): validate the URL before any outbound request
+    # (https-only; block loopback/RFC1918/link-local/cloud-metadata after DNS),
+    # pin the connector to IPv4, and never follow redirects. Also do NOT reflect
+    # arbitrary upstream JSON keys back to the caller — that was an information-
+    # disclosure oracle. Only report presence of the FIXED, known ACME directory
+    # field names (never attacker-controlled data).
+    from utils.ssrf_guard import assert_public_url, safe_connector, SSRFValidationError
+
+    directory_url = str(directory_url)
+    try:
+        await assert_public_url(directory_url)
+    except SSRFValidationError as e:
+        return {"success": False, "error": f"Refused to fetch directory URL: {e}"}
+
+    _KNOWN_ACME_FIELDS = ["newNonce", "newAccount", "newOrder", "newAuthz", "revokeCert", "keyChange"]
     try:
         import aiohttp
-        async with aiohttp.ClientSession() as session:
-            async with session.get(str(directory_url), timeout=aiohttp.ClientTimeout(total=10)) as resp:
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
+            async with session.get(
+                directory_url,
+                timeout=aiohttp.ClientTimeout(total=10),
+                allow_redirects=False,
+            ) as resp:
                 if resp.status == 200:
-                    data = await resp.json()
+                    data = await resp.json(content_type=None)
+                    if not isinstance(data, dict):
+                        return {"success": False, "error": "Directory URL did not return a JSON object"}
+                    present = [k for k in _KNOWN_ACME_FIELDS if k in data]
+                    if not present:
+                        return {"success": False, "error": "Response is not a valid ACME directory"}
                     return {
                         "success": True,
-                        "directory": str(directory_url),
-                        "endpoints": list(data.keys()) if isinstance(data, dict) else []
+                        "directory": directory_url,
+                        "endpoints": present,
                     }
                 else:
                     return {"success": False, "error": f"HTTP {resp.status} from directory URL"}

backend/routers/site_wizard.py

@@ -1046,9 +1046,45 @@ async def suggest_defaults(
 
         slug = "newhost"
         if domain:
-            base = domain.replace("*.", "").split(".")
-            slug = (base[0] or "newhost")[:32].lower()
-            slug = "".join(c if (c.isalnum() or c in ("-", "_")) else "-" for c in slug)
+            # Bulgu #93 (round-24 audit) — IDN / Unicode safety. PRE-FIX
+            # the suggest endpoint used `c.isalnum()`, which is Unicode-
+            # aware and returns True for non-ASCII letters (ü, é, ñ, …).
+            # An operator who typed `bücher.example.com` got back
+            # `backend_name="be-bücher"`, dropped that into the wizard
+            # form, and then hit a hard 422 at create time because the
+            # backend/frontend name validator regex
+            # `^[a-zA-Z][a-zA-Z0-9_-]{0,63}$` is ASCII-only. The wizard
+            # CREATE path also forces the domain itself through punycode
+            # (the validator rejects raw Unicode with a "use 'xn--…'"
+            # hint). Make `suggest` honour the same on-the-wire ASCII
+            # contract: convert each label to its IDN/punycode form
+            # FIRST, then sanitise to the alphanumeric / `-_` set the
+            # entity-name regex permits. The result is a name the
+            # operator can submit to /api/sites without re-typing.
+            first_label = (
+                domain.replace("*.", "").split(".")[0]
+                if domain.replace("*.", "")
+                else ""
+            )
+            ascii_label = first_label
+            if first_label and not first_label.isascii():
+                try:
+                    ascii_label = first_label.encode("idna").decode("ascii")
+                except (UnicodeError, UnicodeDecodeError):
+                    # IDN encoding failed (empty label, invalid chars,
+                    # etc.) — fall back to stripping non-ASCII to '-'
+                    # so we still produce a usable slug.
+                    ascii_label = "".join(
+                        c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                        else "-"
+                        for c in first_label
+                    )
+            slug = (ascii_label or "newhost")[:32].lower()
+            slug = "".join(
+                c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                else "-"
+                for c in slug
+            )
             if slug.startswith("_"):
                 slug = "h-" + slug.lstrip("_")
             if not slug or not slug[0].isalpha():
@@ -1146,6 +1182,53 @@ async def preview_create(
     try:
         await _validate_user_cluster_access(current_user["id"], body.cluster_id, conn)
 
+        # Bulgu #88 / #89 (round-24 audit) — cluster-RBAC parity with
+        # `create_site` for SSL certificate references. PRE-FIX the
+        # preview path (`POST /api/sites/preview`) skipped the
+        # `select_existing_cert()` gate that `create_site` runs at
+        # lines 2298-2308 (per-server CA bundle) and 2398-2405
+        # (HTTPS bind cert). The omission let an authenticated wizard
+        # user pass `ssl.mode='existing', ssl_certificate_id=<X>` (or
+        # `servers[i].ssl_certificate_id=<X>`) where cert `X` belongs
+        # to a DIFFERENT cluster and receive the full rendered
+        # `would_create` envelope back — leaking cert id metadata
+        # across tenant boundaries. The actual submit (`POST /api/
+        # sites`) does enforce the gate, so this is a preview-only
+        # information leak, not a write-path escalation. The fix is
+        # to mirror the same `select_existing_cert(conn, id, cluster_
+        # id)` predicate (which already encodes the "global OR
+        # junction-bound" rule defined in `ssl_service.py:331-370`)
+        # so the preview returns 400 with a clear hint rather than
+        # 200 with a leaked render. We deliberately keep the error
+        # phrasing identical to the create-time message so wizard
+        # UI handlers that already match on "not found / inactive"
+        # need no client changes.
+        if body.ssl.mode == "existing" and body.ssl.ssl_certificate_id:
+            _preview_resolved = await select_existing_cert(
+                conn, body.ssl.ssl_certificate_id, body.cluster_id,
+            )
+            if not _preview_resolved:
+                raise HTTPException(
+                    status_code=400,
+                    detail=(
+                        f"ssl_certificate_id {body.ssl.ssl_certificate_id} "
+                        "not found / inactive / not bound to this cluster"
+                    ),
+                )
+        for _idx, _srv in enumerate(body.servers or []):
+            _srv_cert_id = getattr(_srv, "ssl_certificate_id", None)
+            if _srv_cert_id:
+                if not await select_existing_cert(
+                    conn, _srv_cert_id, body.cluster_id,
+                ):
+                    raise HTTPException(
+                        status_code=400,
+                        detail=(
+                            f"servers[{_idx}].ssl_certificate_id={_srv_cert_id} "
+                            "not found / inactive / not bound to this cluster"
+                        ),
+                    )
+
         # Phase K Phase C: rate-limit ONLY the dry-run code path so
         # legacy callers (e.g. `SiteDrafts.handlePreview` which never
         # sets the flag) keep their unrestricted preview budget. The
@@ -1848,6 +1931,28 @@ async def create_site(
     try:
         await _validate_user_cluster_access(user_id, body.cluster_id, conn)
 
+        # Bulgu #87 (round-23 audit) — rate-limit the actual create
+        # endpoint. Pre-fix /api/sites/preview and
+        # /api/sites/preflight-acme were rate-limited (5/min, see
+        # `_enforce_rate_limit` callers above) but POST /api/sites
+        # itself had NO cap. Round-3 live testing fired 10 wizard
+        # creates in a single second against demo-cluster1; without
+        # the cap an authenticated user / script can spam-create
+        # entities until the cluster apply queue clogs.
+        #
+        # Action name must match what the router itself logs at the
+        # end of create_site (`wizard_create_site` — see
+        # `_log_user_activity(... action="wizard_create_site", ...)`
+        # at the success path below). The activity_logger middleware
+        # intentionally skips this endpoint to avoid double-logging
+        # (see middleware/activity_logger.py:74-110), so we must
+        # rate-limit on the SAME action the router emits.
+        #
+        # 5/min matches the preview / preflight budget — bulk
+        # operators have the dedicated /api/sites/bulk-import path
+        # for larger batches.
+        await _enforce_rate_limit(conn, user_id, "wizard_create_site")
+
         # ----- Pre-create checks (must succeed before transaction)
 
         # Phase 3 (R11-audit follow-up): reserved-name check parity with
@@ -2133,7 +2238,31 @@ async def create_site(
         # rebrand. The reject path on `cluster.py` recognises BOTH
         # prefixes so historical APPLIED versions (created before this
         # rename) keep behaving correctly during reject/undo.
-        version_name = f"bulk-site-create-{ts}"
+        #
+        # Bulgu #86 (round-23 audit) — append a short UUID suffix so
+        # two wizard POST /api/sites calls submitted within the SAME
+        # epoch second cannot collide on the
+        # `config_versions(cluster_id, version_name)` UNIQUE
+        # constraint. Pre-fix `version_name = f"bulk-site-create-{ts}"`
+        # gave seconds resolution, so an operator who clicked "Create"
+        # twice in rapid succession (or any back-to-back API
+        # automation) saw the SECOND call 409 with the generic
+        # `UniqueViolationError` fall-through message:
+        #
+        #     "A wizard entity with this name already exists on the
+        #      cluster (UNIQUE constraint). Pick a different name."
+        #
+        # — even though the operator-chosen backend / frontend / SSL
+        # names were unique. The actual collision was on the
+        # auto-generated `version_name` and re-naming the wizard
+        # inputs did NOT help. The reject_pending_changes path on
+        # cluster.py:4179 prefix-matches `bulk-site-create-*` so
+        # appending a unique suffix preserves the historical
+        # rollback / undo semantics. 6 hex chars give 16M-room before
+        # birthday collisions, vastly more than the per-second
+        # request volume an operator can sustain through the wizard.
+        import uuid
+        version_name = f"bulk-site-create-{ts}-{uuid.uuid4().hex[:6]}"
         bulk_snapshots: List[dict] = []
         created_ids: Dict[str, Any] = {}
 
@@ -2929,22 +3058,68 @@ async def create_site(
         # broke".
         msg = str(uve)
         logger.info(f"WIZARD: name conflict on create: {msg}")
-        if "backends_name_cluster_id_key" in msg or 'backends_name' in msg:
+        # Bulgu #85 (round-23 audit) — extract the offending constraint
+        # name from the asyncpg message so the operator-visible detail
+        # can pin-point WHICH entity collided. Pre-fix the handler only
+        # recognised `backends_*_key` and `frontends_*_key`; any other
+        # constraint (ssl_certificates, backend_servers, config_versions
+        # …) fell through to the generic "wizard entity with this name"
+        # message which is useless for debugging — the operator has to
+        # open the server log and the engineer has to ssh-bounce to
+        # extract `constraint=<name>` from the exception detail.
+        #
+        # ``asyncpg.UniqueViolationError.constraint_name`` is the
+        # canonical structured field; ``str(uve)`` only contains the
+        # human-formatted DETAIL line. Prefer the attribute, fall back
+        # to substring scanning so we stay robust if asyncpg ever stops
+        # exposing it.
+        constraint = getattr(uve, "constraint_name", None) or ""
+        msg_lower = msg.lower()
+        if "backends_name_cluster_id_key" in msg or 'backends_name' in msg \
+                or constraint == "backends_name_cluster_id_key":
             detail = (
-                "A backend with this name already exists on the cluster "
-                "(possibly soft-deleted). Pick a different backend name "
-                "or restore/permanently-delete the existing row."
+                f"A backend named '{body.backend.name}' already exists on "
+                "the cluster (possibly soft-deleted). Pick a different "
+                "backend name or restore/permanently-delete the existing "
+                "row."
             )
-        elif "frontends_name_cluster_id_key" in msg or "frontends_name" in msg:
+        elif "frontends_name_cluster_id_key" in msg or "frontends_name" in msg \
+                or constraint == "frontends_name_cluster_id_key":
             detail = (
-                "A frontend with this name already exists on the cluster "
+                f"A frontend named '{body.frontend.name}' (or its auto-"
+                f"derived HTTPS sibling) already exists on the cluster "
                 "(possibly soft-deleted). Pick a different frontend name "
                 "or restore/permanently-delete the existing row."
             )
-        else:
+        elif "ssl_certificates" in msg_lower or "ssl_certificates" in constraint \
+                or "ssl_cert" in constraint:
+            ssl_name = getattr(body.ssl, "name", None) or "(unnamed)"
             detail = (
-                "A wizard entity with this name already exists on the "
-                "cluster (UNIQUE constraint). Pick a different name."
+                f"An SSL certificate named '{ssl_name}' already exists on "
+                "the cluster (possibly soft-deleted). Pick a different "
+                "ssl.name or restore/permanently-delete the existing "
+                "certificate row."
+            )
+        elif "backend_servers" in msg_lower or "backend_servers" in constraint:
+            detail = (
+                "Two servers in the same backend share a server_name. "
+                "HAProxy requires `server <name>` tokens to be unique "
+                "within a backend block. Rename the duplicate(s) and "
+                "resubmit."
+            )
+        else:
+            # Echo the constraint name (a stable, non-secret schema
+            # identifier) in the detail so a human reading the toast
+            # can grep the codebase for the matching CREATE TABLE
+            # without needing server-log access. Names like
+            # `proxied_hosts_pkey` or `config_versions_unique` are
+            # safe to surface — they're public schema info.
+            con_hint = f" (constraint={constraint})" if constraint else ""
+            detail = (
+                f"A wizard entity with this name already exists on the "
+                f"cluster (UNIQUE constraint{con_hint}). Pick a different "
+                "name or retry; if the conflict persists contact the "
+                "platform team."
             )
         raise HTTPException(status_code=409, detail=detail)
     except Exception as e:

backend/routers/ssl.py

@@ -9,7 +9,7 @@ from datetime import datetime, timezone
 
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
-from auth_middleware import get_current_user_from_token
+from auth_middleware import get_current_user_from_token, require_authenticated_user
 from models.ssl import SSLCertificate, SSLCertificateCreate, SSLCertificateUpdate, SSLCertificateResponse
 from utils.ssl_parser import parse_ssl_certificate, validate_private_key, validate_certificate_chain, format_certificate_info
 from utils.activity_log import log_user_activity
@@ -825,7 +825,8 @@ async def get_ssl_certificate(cert_id: int, authorization: str = Header(None)):
         logger.error(f"Error getting SSL certificate: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/certificates/{cert_id}/config-versions")
+@router.get("/certificates/{cert_id}/config-versions",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): was unauthenticated
 async def get_ssl_certificate_config_versions(cert_id: int):
     """Get config version history for specific SSL certificate"""
     try:

backend/routers/user.py

@@ -38,18 +38,19 @@ async def get_users(authorization: str = Header(None)):
         # Get users with their roles (only active users)
         try:
             users = await conn.fetch("""
-                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active, 
-                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at
+                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active,
+                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at,
+                       COALESCE(u.mfa_enabled, FALSE) AS mfa_enabled
                 FROM users u
                 WHERE u.is_active = TRUE
                 ORDER BY u.username
             """)
         except Exception as schema_error:
             logger.warning(f"Schema error in users query, using fallback: {schema_error}")
-            # Fallback query with minimal columns
+            # Fallback query with minimal columns (pre-MFA-migration deploys)
             users = await conn.fetch("""
                 SELECT id, username, email, is_active, is_admin, created_at
-                FROM users 
+                FROM users
                 WHERE is_active = TRUE
                 ORDER BY username
             """)

backend/routers/waf.py

@@ -1,4 +1,5 @@
-from fastapi import APIRouter, HTTPException, Request, Header
+from fastapi import APIRouter, HTTPException, Request, Header, Depends
+from auth_middleware import require_authenticated_user
 from typing import Optional
 import logging
 import time
@@ -182,7 +183,8 @@ async def get_waf_stats(cluster_id: Optional[int] = None, authorization: str = H
         logger.error(f"Error fetching WAF stats: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
-@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules")
+@router.get("/rules", summary="Get WAF Rules", response_description="List of WAF rules",
+            dependencies=[Depends(require_authenticated_user)])  # SECURITY (GHSA-3p5c): WAF rule definitions aid bypass crafting
 async def get_waf_rules(cluster_id: Optional[int] = None):
     """
     # Get WAF Rules

backend/services/acme_diagnostics.py

@@ -8,9 +8,13 @@ suitable for an Antd Tabs/Steps display.
 Key constraints (Section 3.3 of the v1.5.0 plan):
 - DNS resolution uses stdlib socket.gethostbyname_ex via run_in_executor (we
   intentionally avoid pulling aiodns as a runtime dep for v1.5.0).
-- Port-80 probe is HEAD-only, target locked to the order's domains, success on
-  HTTP 200 OR 404, warns on egress timeout (don't fail-hard — corp egress
-  policies often blackhole outbound 80).
+- Port-80 probe is a GET (not HEAD) locked to the order's domains, because the
+  status code alone cannot tell a working challenge endpoint from a web UI: a
+  reverse proxy that has lost its /.well-known/acme-challenge/ location serves
+  its SPA with HTTP 200. The body's shape decides. Warns rather than fails on
+  egress timeout (corp egress policies often blackhole outbound 80) and on a
+  wrong responder (this probe sees the PUBLIC domain, not the challenge backend,
+  so it is evidence rather than a verdict).
 - All checks have hard wall-clock timeouts (asyncio.wait_for) to bound impact
   on the API event loop.
 - humanize_error_detail covers >= 11 RFC8555 problem types and is backwards
@@ -116,6 +120,10 @@ _PROBLEM_HUMANIZED: Dict[str, Dict[str, str]] = {
         "title": "HTTP-01 challenge response mismatch",
         "hint": "The CA fetched the challenge URL but received the wrong key authorization. Confirm the challenge was served from the right backend.",
     },
+    "urn:ietf:params:acme:error:externalAccountRequired": {
+        "title": "External Account Binding (EAB) required",
+        "hint": "This CA (e.g. ZeroSSL, Google) requires EAB. Enter the EAB Key ID and HMAC Key from your CA account when registering.",
+    },
     "urn:ietf:params:acme:error:invalidContact": {
         "title": "Invalid contact email",
         "hint": "The ACME account email is malformed. Update the LE account email.",
@@ -199,6 +207,23 @@ def humanize_error_detail(error_detail: Any) -> Dict[str, Any]:
     status = parsed.get("status")
     subproblems = parsed.get("subproblems") or []
 
+    # Issue #35: DNS-01 failures are recorded as {stage, reason, timestamp} (no RFC8555 "type"),
+    # so without this fallback the humanized alert would show a bare "ACME error" with no message.
+    # Surface the reason and a targeted hint so the operator knows exactly what to fix.
+    if not problem_type and parsed.get("reason"):
+        reason = str(parsed.get("reason"))
+        message = message or reason
+        title = "DNS-01 validation failed"
+        rlow = reason.lower()
+        if "decrypt" in rlow or "credential" in rlow:
+            hint = hint or "Re-enter the DNS provider credentials for this account in ACME Automation."
+        elif "zone" in rlow:
+            hint = hint or "Confirm the domain's DNS zone is managed by the configured provider and the token has access to it."
+        elif "deadline" in rlow or "expired" in rlow or "confirm" in rlow:
+            hint = hint or "The manual confirmation window passed. Create a new certificate request and publish the TXT record promptly."
+        else:
+            hint = hint or "Check the DNS TXT record and provider credentials, then retry."
+
     out = {
         "title": title,
         "message": message,
@@ -309,10 +334,41 @@ async def check_dns(domains: List[str]) -> Dict[str, Any]:
     )
 
 
+# Enough to classify a response without turning the diagnostic into a way to pull
+# arbitrary amounts of a third party's content into our JSON.
+_PROBE_BODY_LIMIT = 65536
+
+
+def _classify_probe_body(body: bytes, content_type: str) -> str:
+    """Coarse shape of a probe response: html | json | text | empty | binary.
+
+    The body itself is deliberately NOT retained anywhere — the shape is all that is
+    needed to tell "served me a web page" from "served me a token", and keeping the
+    bytes would open a new read surface onto whatever is behind the address.
+    """
+    if not body:
+        return "empty"
+    ct = (content_type or "").lower()
+    head = body[:512].lstrip().lower()
+    if ct.startswith("text/html") or head.startswith((b"<!doctype", b"<html")):
+        return "html"
+    if ct.startswith("application/json") or head[:1] in (b"{", b"["):
+        return "json"
+    try:
+        body.decode("utf-8")
+    except UnicodeDecodeError:
+        return "binary"
+    return "text"
+
+
 async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict[str, Any]:
-    """Probe HTTP-01 readiness on port 80 with a HEAD request to a synthetic
-    challenge URL. Success on 200 OR 404 (404 means the well-known path is
-    served but no challenge yet — fine).
+    """Probe HTTP-01 readiness on port 80 with a GET to a synthetic challenge URL.
+
+    404 means the path is served but no challenge is outstanding, which is fine. A
+    200 is only fine if the body is NOT a web page: a proxy that has lost its
+    /.well-known/acme-challenge/ route falls through to its catch-all and answers
+    200 with index.html, which a status-code-only check accepts as healthy while
+    every real validation fails.
 
     On egress timeout we WARN rather than FAIL because many corporate egress
     policies blackhole port 80 outbound; that does not impair LE's ingress
@@ -377,12 +433,53 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
                 continue
             url = f"http://{d}/.well-known/acme-challenge/diagnostic-probe"
             try:
-                async with session.head(url, allow_redirects=False) as resp:
-                    targets.append({
+                # GET, not HEAD: the status code alone cannot tell a working challenge
+                # endpoint from a SPA. A reverse proxy that has lost its
+                # /.well-known/acme-challenge/ location falls through to its catch-all
+                # and serves index.html with HTTP 200 — which the old
+                # `status in (200, 404)` rule accepted as healthy while every real
+                # validation failed. Only the body distinguishes them.
+                async with session.get(url, allow_redirects=False) as resp:
+                    # The body is EVIDENCE, not a precondition. If it cannot be read —
+                    # connection reset mid-response, a server that hangs after headers —
+                    # fall back to the status-only semantics this check has always had
+                    # rather than turning a healthy 404 into a hard failure. The stricter
+                    # rule below applies only when there is something to judge.
+                    try:
+                        body = await resp.content.read(_PROBE_BODY_LIMIT)
+                    except Exception:
+                        body = None
+                    content_type = (resp.headers.get("content-type") or "").split(";")[0].strip()
+                    body_class = (
+                        _classify_probe_body(body, content_type) if body is not None else "unread"
+                    )
+                    target = {
                         "domain": d,
                         "status": resp.status,
-                        "ok": resp.status in (200, 404),
-                    })
+                        "content_type": content_type or None,
+                        "body_len": len(body) if body is not None else None,
+                        "body_class": body_class,
+                    }
+                    if resp.status == 200 and body_class == "html":
+                        # Reachable, wrong responder. Reported as a warning rather than
+                        # a failure: this check probes the PUBLIC domain and cannot see
+                        # the challenge backend, so it is evidence, not a verdict — and
+                        # a new `fail` here would block the site wizard on upgrade day
+                        # for every install.
+                        target["warn"] = True
+                        target["diagnosis"] = (
+                            "responded 200 with an HTML page, not a challenge token — "
+                            "the request is reaching a web UI instead of the ACME endpoint"
+                        )
+                    elif resp.status in (301, 302, 303, 307, 308):
+                        target["warn"] = True
+                        target["redirect_location"] = resp.headers.get("location")
+                        target["diagnosis"] = (
+                            "redirected instead of serving the challenge path"
+                        )
+                    else:
+                        target["ok"] = resp.status in (200, 404)
+                    targets.append(target)
             except asyncio.TimeoutError:
                 targets.append({"domain": d, "error": "egress timeout", "warn": True})
                 skip_reason = "egress timeout"
@@ -404,7 +501,11 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
             details={"targets": targets},
             duration_ms=duration_ms,
         )
-    if warns and not [t for t in targets if t.get("ok")]:
+    # Surface warnings even when OTHER domains answered correctly. The old condition
+    # ("warn only if nothing succeeded") hid the single most diagnostic outcome there
+    # is: a multi-domain certificate where one name reaches a web UI instead of the
+    # challenge endpoint reported a clean pass.
+    if warns:
         # R18b audit fix (round 7): branch the rollup message on the
         # actual cause. Pre-fix the message was always "Egress to
         # port 80 appears blocked" — even when every target was
@@ -414,6 +515,36 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
         # corporate firewall logs while the real cause was an
         # internal-only DNS A record. Also harden against
         # `skip_reason=None` so the message never reads "(None)".
+        # Wrong-responder warnings take priority over every other cause: they are the
+        # only ones that mean "your server answered, and answered wrong", which is a
+        # different problem from "we could not test".
+        wrong_responder = [t for t in targets if t.get("diagnosis")]
+        if wrong_responder:
+            first = wrong_responder[0]
+            if first.get("body_class") == "html":
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} with an HTML "
+                    f"page ({first.get('content_type') or 'unknown type'}, "
+                    f"{first.get('body_len')} bytes) instead of a challenge token. The "
+                    "path is reaching a web interface, not the ACME endpoint — check "
+                    "that the reverse proxy in front of OpenManager routes "
+                    "/.well-known/acme-challenge/ to the API."
+                )
+            else:
+                human = (
+                    f"{first['domain']} answered HTTP {first.get('status')} "
+                    f"({first.get('diagnosis')})"
+                )
+            return _check_result(
+                "port80",
+                "Port 80 reachability",
+                "warn",
+                human,
+                severity="warn",
+                details={"targets": targets},
+                duration_ms=duration_ms,
+            )
+
         ssrf_skip = any(
             "non-public" in (t.get("skip") or "")
             or "SSRF" in (t.get("skip") or "")
@@ -470,11 +601,20 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             duration_ms=int((time.time() - started) * 1000),
         )
 
+    # The WHERE clause is deliberately identical to the pre-existing one, so `not rows`
+    # still means exactly what it meant before and the `fail` branch below cannot fire
+    # in any situation where it previously passed. Narrowing it here (e.g. by adding a
+    # mode filter) would turn a tcp-only port-80 cluster from "ok" into "fail", and the
+    # site wizard blocks submit on any failing check — locking those installs the day
+    # this ships. Mode is examined afterwards, in Python, and only ever downgrades to
+    # `warn`.
     rows = await conn.fetch(
         """
-        SELECT id, name, bind_address, bind_port, mode, default_backend
-        FROM frontends
-        WHERE cluster_id = ANY($1::int[]) AND is_active = TRUE AND bind_port = 80
+        SELECT f.id, f.name, f.bind_address, f.bind_port, f.mode, f.default_backend,
+               f.cluster_id, c.acme_enabled
+        FROM frontends f
+        JOIN haproxy_clusters c ON c.id = f.cluster_id
+        WHERE f.cluster_id = ANY($1::int[]) AND f.is_active = TRUE AND f.bind_port = 80
         """,
         cluster_ids,
     )
@@ -489,13 +629,144 @@ async def check_routing(conn, domains: List[str], cluster_ids: List[int]) -> Dic
             details={"cluster_ids": cluster_ids},
             duration_ms=duration_ms,
         )
+
+    # Character-for-character the renderer's normalisation (services/haproxy_config.py),
+    # so this can never disagree with what actually gets emitted.
+    http_rows = [r for r in rows if (r["mode"] or "http").strip().lower() == "http"]
+    if not http_rows:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                "The only port-80 frontend(s) in the target cluster(s) are in tcp mode. "
+                "A tcp-mode frontend cannot carry the /.well-known/acme-challenge/ ACL, "
+                "so HTTP-01 cannot be served — use DNS-01, or add an http-mode frontend "
+                "on port 80."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows]},
+            duration_ms=duration_ms,
+        )
+    rows = http_rows
+
+    # A frontend row proves only that the DATABASE describes port-80 routing. The
+    # renderer gates the challenge ACL on `acme_enabled`, and the nodes run whatever
+    # config was last APPLIED — so the row said "ok" during an incident where the
+    # live config had no usable challenge route at all. Check the two things the row
+    # cannot tell us. Both report `warn`, never `fail`: the site wizard blocks submit
+    # on any `fail`, so a new failing condition would lock every install on the day
+    # it ships.
+    # `.get()` rather than `[]`: the column gates a WARNING, so a row shape without
+    # it should not blow up the whole diagnostic. Absent means "assume enabled" —
+    # the applied-config check below is the authoritative one either way.
+    acme_off = sorted({r["cluster_id"] for r in rows if not r.get("acme_enabled", True)})
+    if acme_off:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {acme_off} have ACME Challenge Routing disabled, so the "
+                "generated config contains no /.well-known/acme-challenge/ route. "
+                "Enable it in Cluster Management, then apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "acme_disabled_clusters": acme_off},
+            duration_ms=duration_ms,
+        )
+
+    missing_in_applied = []
+    challenge_backends = {}
+    for cluster_id in sorted({r["cluster_id"] for r in rows}):
+        # Selector matched to the one the AGENT uses to fetch its config
+        # (routers/agent.py: status='APPLIED' AND is_active=TRUE), because the question
+        # here is "what are the nodes running right now?". Without `is_active` this can
+        # read a superseded row and report on a config that was never delivered.
+        # Extracting in SQL rather than pulling whole configs back per cluster: these
+        # files run to hundreds of KB on real installs.
+        applied = await conn.fetchrow(
+            """
+            SELECT position('use_backend _acme_challenge_backend' in config_content) > 0
+                       AS has_route,
+                   substring(config_content from 'server _acme_mgmt [^\\n]*') AS server_line
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+              AND config_content IS NOT NULL
+            ORDER BY created_at DESC LIMIT 1
+            """,
+            cluster_id,
+        )
+        if not applied or not applied["has_route"]:
+            missing_in_applied.append(cluster_id)
+            continue
+        server_line = (applied["server_line"] or "").strip()
+        challenge_backends[cluster_id] = (
+            server_line[len("server _acme_mgmt "):].strip() if server_line else None
+        )
+
+    if missing_in_applied:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {missing_in_applied} have no applied configuration carrying "
+                "the challenge route. The change exists in the database but the nodes are "
+                "still running an older config — apply the cluster."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_not_applied": missing_in_applied},
+            duration_ms=duration_ms,
+        )
+
+    # A backend section with no `server` line: the route exists, `haproxy -c` passes,
+    # and every challenge request gets a 503 from an empty backend. Without this branch
+    # the falsy target slips past the loopback filter below and the check reports "ok".
+    serverless = sorted(cid for cid, target in challenge_backends.items() if not target)
+    if serverless:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"Cluster(s) {serverless} route the challenge path to a backend that has "
+                "no server line, so every request returns 503. The configured ACME "
+                "Challenge Backend URL could not be resolved into an address — check it "
+                "in Cluster Management, or Settings > ACME for the global value."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "clusters_without_server": serverless},
+            duration_ms=duration_ms,
+        )
+
+    loopback = {
+        cid: target for cid, target in challenge_backends.items()
+        if target and target.split(":")[0].strip("[]").lower()
+        in ("localhost", "127.0.0.1", "::1", "0.0.0.0")
+    }
+    if loopback:
+        return _check_result(
+            "routing",
+            "HAProxy routing",
+            "warn",
+            (
+                f"The applied config points the challenge backend at {sorted(loopback.values())}. "
+                "HAProxy resolves that on the HAProxy node, so it means the node itself, not "
+                "this management server. Set ACME Challenge Backend URL to a routable address."
+            ),
+            severity="warn",
+            details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
+            duration_ms=duration_ms,
+        )
+
     return _check_result(
         "routing",
         "HAProxy routing",
         "ok",
-        f"Found {len(rows)} HTTP frontend(s) on port 80",
+        f"Found {len(rows)} HTTP frontend(s) on port 80; challenge route present in applied config",
         severity="info",
-        details={"frontends": [dict(r) for r in rows]},
+        details={"frontends": [dict(r) for r in rows], "challenge_backends": challenge_backends},
         duration_ms=duration_ms,
     )
 
@@ -573,9 +844,18 @@ async def check_agents(conn, cluster_ids: List[int]) -> Dict[str, Any]:
             severity="warn",
             duration_ms=int((time.time() - started) * 1000),
         )
+    # Bulgu #84 (round-23 audit) — the canonical timestamp column on the
+    # `agents` table is `last_seen`. Pre-fix this query referenced a
+    # non-existent `a.last_heartbeat`, so every ACME preflight call
+    # (`POST /api/sites/preflight-acme`) crashed at the `check_agents`
+    # stage with `UndefinedColumnError: column a.last_heartbeat does
+    # not exist`, blocking the entire wizard's ACME pre-validation
+    # gate. Every other agents.last_seen reader in the codebase
+    # (routers/cluster.py:695-702, routers/agent.py, routers/dashboard
+    # *.py) uses `last_seen`; aligning here.
     rows = await conn.fetch(
         """
-        SELECT a.id, a.hostname, a.status, a.last_heartbeat, hc.id AS cluster_id, hc.name AS cluster_name
+        SELECT a.id, a.hostname, a.status, a.last_seen, hc.id AS cluster_id, hc.name AS cluster_name
         FROM agents a
         JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
         WHERE hc.id = ANY($1::int[])
@@ -623,6 +903,61 @@ async def check_agents(conn, cluster_ids: List[int]) -> Dict[str, Any]:
 CHECK_IDS = ("dns", "port80", "routing", "account", "agents")
 
 
+def _coerce_cluster_ids(raw) -> List[int]:
+    """Coerce a cluster_ids list to ints, dropping non-integer-compatible
+    values. JSONB-stored lists occasionally land as ["1", "2"] (string form)
+    due to legacy paths; asyncpg's `$1::int[]` cast then fails the diagnostic
+    query with InvalidTextRepresentationError. We normalise here so the
+    diagnostic surface is the same regardless of how the order was written.
+    """
+    out: List[int] = []
+    if not raw:
+        return out
+    for v in raw:
+        try:
+            out.append(int(v))
+        except (TypeError, ValueError):
+            continue
+    return out
+
+
+# Bulgu #94 (Round-25 audit) — the entire point of the diagnostic panel
+# is to SHOW the operator what went wrong. Pre-fix, a single check raising
+# an uncaught exception (e.g. an asyncpg cast error from a malformed
+# cluster_ids JSONB, a DNS resolver outage, an SSRF-guard glitch) would
+# propagate up to the router's `try/finally` block, which had no `except`
+# clause, and return HTTP 500 with no body. The operator saw only
+# "Internal Server Error" in DevTools — the inverse of what a diagnostic
+# panel should ever produce. We now wrap every check inside `run_checks`
+# so that a check crash becomes a structured `fail` row instead of
+# bubbling up; the operator gets the exception type + message in the
+# UI and can carry it forward, and the rest of the panel still renders.
+async def _safe_check(check_id: str, label: str, coro):
+    """Run an awaitable that produces a check result; swallow exceptions
+    and convert them to a structured `fail` result so the diagnostic
+    response is never short-circuited by a single broken check."""
+    started = time.time()
+    try:
+        return await coro
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        duration_ms = int((time.time() - started) * 1000)
+        logger.exception(
+            "ACME diagnostic check %s raised", check_id
+        )
+        return _check_result(
+            check_id,
+            label,
+            "fail",
+            f"Diagnostic check crashed: {exc.__class__.__name__}: {exc}",
+            severity="error",
+            details={
+                "exception_type": exc.__class__.__name__,
+                "exception_message": str(exc),
+            },
+            duration_ms=duration_ms,
+        )
+
+
 async def run_checks(
     conn,
     *,
@@ -630,22 +965,53 @@ async def run_checks(
     cluster_ids: List[int],
     account_id: Optional[int],
     only: Optional[List[str]] = None,
+    challenge_type: str = "http-01",
 ) -> List[Dict[str, Any]]:
     """Execute the full pre-flight check suite. `only` lets callers re-run a
     subset (per-check rerun in the UI).
+
+    Every individual check is wrapped in `_safe_check` so the diagnostic
+    endpoint NEVER 500s because of one broken check — the operator gets
+    a structured `fail` row identifying which check crashed and why.
     """
     selected = set(only) if only else set(CHECK_IDS)
     results: List[Dict[str, Any]] = []
 
+    # Normalise inputs once so the per-check error stays in the right
+    # bucket (a malformed cluster_ids should not crash routing/agents).
+    safe_domains = [d for d in (domains or []) if isinstance(d, str) and d]
+    safe_cluster_ids = _coerce_cluster_ids(cluster_ids)
+    try:
+        safe_account_id = int(account_id) if account_id is not None else None
+    except (TypeError, ValueError):
+        safe_account_id = None
+
+    # Issue #35: DNS-01 validates via a TXT record, so the HTTP-01 reachability checks
+    # (public A record, inbound port 80, ACME Challenge Routing) do not apply — report them
+    # as `skipped` rather than failing an internal/isolated host that is actually fine.
+    is_dns01 = (challenge_type == "dns-01")
     if "dns" in selected:
-        results.append(await check_dns(domains))
+        if is_dns01:
+            results.append(_check_result("dns", "DNS resolution", "skipped",
+                                         "DNS-01: a public A record is not required (validation is via a TXT record).",
+                                         severity="info"))
+        else:
+            results.append(await _safe_check("dns", "DNS resolution", check_dns(safe_domains)))
     if "port80" in selected:
-        results.append(await check_port80(domains))
+        if is_dns01:
+            results.append(_check_result("port80", "Port 80 reachability", "skipped",
+                                         "DNS-01: inbound port 80 is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("port80", "Port 80 reachability", check_port80(safe_domains)))
     if "routing" in selected:
-        results.append(await check_routing(conn, domains, cluster_ids))
+        if is_dns01:
+            results.append(_check_result("routing", "HAProxy routing", "skipped",
+                                         "DNS-01: ACME Challenge Routing is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("routing", "HAProxy routing", check_routing(conn, safe_domains, safe_cluster_ids)))
     if "account" in selected:
-        results.append(await check_account(conn, account_id))
+        results.append(await _safe_check("account", "ACME account", check_account(conn, safe_account_id)))
     if "agents" in selected:
-        results.append(await check_agents(conn, cluster_ids))
+        results.append(await _safe_check("agents", "HAProxy agents", check_agents(conn, safe_cluster_ids)))
 
     return results

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -65,25 +69,43 @@ class ACMEService:
             if cached.get('_fetched_at', 0) > time.time() - 3600:
                 return cached
 
-        async with aiohttp.ClientSession() as session:
-            async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): directory_url can come from a stored
+        # account row; validate it (https + public IP, no redirects) before the
+        # server-side fetch so it cannot be pointed at internal/metadata targets.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(directory_url)
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
+            async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
                 if resp.status != 200:
                     raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
                 data = await resp.json()
                 if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
+                    self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
                 data['_fetched_at'] = time.time()
                 self._directory_cache[directory_url] = data
                 return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
-            nonce = self._nonce
-            self._nonce = None
-            return nonce
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         directory = await self.get_directory(directory_url)
-        async with aiohttp.ClientSession() as session:
-            async with session.head(directory['newNonce']) as resp:
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): newNonce is taken from the (attacker-
+        # influenceable) directory JSON and is fetched here BEFORE the guarded
+        # _signed_request POST, so it must be guarded too — otherwise a directory
+        # that returns an internal newNonce (and omits Replay-Nonce) is a live SSRF.
+        # https + public IP only, IPv4-pinned connector, no redirects, bounded timeout.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        nonce_url = directory['newNonce']
+        await assert_public_url(nonce_url)
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
+            async with session.head(nonce_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
                 return resp.headers['Replay-Nonce']
 
     def _generate_account_key(self) -> Tuple[str, dict]:
@@ -128,6 +150,20 @@ class ACMEService:
         digest = hashlib.sha256(ordered.encode('utf-8')).digest()
         return _b64url(digest)
 
+    @staticmethod
+    def _dns_txt_value(key_authorization: str) -> str:
+        """RFC 8555 §8.4: the DNS-01 TXT value is base64url(SHA256(key_authorization)) over the
+        RAW 32-byte digest (NOT the hexdigest)."""
+        return _b64url(hashlib.sha256(key_authorization.encode('utf-8')).digest())
+
+    @staticmethod
+    def _challenge_dns_name(identifier: str) -> str:
+        """The `_acme-challenge.<base>` record name for an ACME identifier. A leading wildcard
+        `*.` is stripped, so both `*.example.com` and bare `example.com` map to the SAME name
+        `_acme-challenge.example.com` (which is why apex+wildcard need two coexisting TXT values)."""
+        base = identifier[2:] if identifier.startswith('*.') else identifier
+        return f"_acme-challenge.{base}"
+
     def _sign_jws(self, private_key, protected: dict, payload: Any) -> dict:
         protected_b64 = _b64url(json.dumps(protected).encode('utf-8'))
         if payload == "":
@@ -165,20 +201,34 @@ class ACMEService:
 
         body = self._sign_jws(private_key, protected, payload)
 
-        async with aiohttp.ClientSession() as session:
+        # SECURITY (GHSA-3vh4-gvxx-wm2p): `url` is taken from the CA directory /
+        # order responses. The directory is already fetched from a validated
+        # public CA, but guard the follow-up POST target too (defence in depth)
+        # so a tampered/malicious directory cannot steer the request internally.
+        from utils.ssrf_guard import assert_public_url, safe_connector
+        await assert_public_url(url)
+
+        async with aiohttp.ClientSession(connector=safe_connector()) as session:
             for attempt in range(3):
                 async with session.post(
                     url,
                     json=body,
                     headers={"Content-Type": "application/jose+json"},
                     timeout=aiohttp.ClientTimeout(total=30),
+                    allow_redirects=False,
                 ) as resp:
                     if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
+                    if resp.status == 400 and attempt < 2:
                         err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
+                        etype = (err.get('type') or '')
+                        edetail = (err.get('detail') or '').lower()
+                        # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                        # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                        # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                        # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                        if etype.endswith('badNonce') or 'nonce' in edetail:
                             nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
                             protected['nonce'] = nonce
                             body = self._sign_jws(private_key, protected, payload)
@@ -208,6 +258,8 @@ class ACMEService:
         tos_agreed: bool = True,
         eab_kid: Optional[str] = None,
         eab_hmac_key: Optional[str] = None,
+        challenge_type: str = 'http-01',
+        dns_provider: Optional[str] = None,
     ) -> dict:
         directory = await self.get_directory(directory_url)
         pem, jwk = self._generate_account_key()
@@ -250,13 +302,14 @@ class ACMEService:
         conn = await get_database_connection()
         try:
             row = await conn.fetchrow("""
-                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid, challenge_type, dns_provider)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 ON CONFLICT (email, directory_url) DO UPDATE SET
-                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6, updated_at = NOW()
-                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at
+                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6,
+                    challenge_type = $8, dns_provider = $9, updated_at = NOW()
+                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at, challenge_type, dns_provider
             """, email, directory_url, account_url, pem,
-                data.get('status') or 'valid', tos_agreed, eab_kid)
+                data.get('status') or 'valid', tos_agreed, eab_kid, challenge_type, dns_provider)
             return dict(row)
         finally:
             await close_database_connection(conn)
@@ -302,8 +355,10 @@ class ACMEService:
         account_id: int,
         domains: List[str],
         cluster_ids: Optional[List[int]] = None,
+        challenge_type: str = 'http-01',
+        created_by: Optional[int] = None,
     ) -> dict:
-        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}")
+        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}, challenge_type={challenge_type}")
         conn = await get_database_connection()
         try:
             account = await conn.fetchrow(
@@ -341,12 +396,12 @@ class ACMEService:
 
             order_row = await conn.fetchrow("""
                 INSERT INTO letsencrypt_orders
-                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids, challenge_type, created_by)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 RETURNING id
             """, account_id, order_url, data.get('status') or 'pending',
                 json.dumps(domains), data.get('finalize') or '', expires_at,
-                json.dumps(cluster_ids or []))
+                json.dumps(cluster_ids or []), challenge_type, created_by)
 
             order_id = order_row['id']
 
@@ -383,18 +438,24 @@ class ACMEService:
                 domain = (auth_data.get('identifier') or {}).get('value', '')
                 http01_for_domain = False
                 for challenge in (auth_data.get('challenges') or []):
-                    if challenge.get('type') == 'http-01':
+                    # Store only the challenge of the CHOSEN method (default 'http-01' keeps the
+                    # existing behaviour byte-identical; 'dns-01' selects the TXT challenge instead).
+                    if challenge.get('type') == challenge_type:
                         token = challenge['token']
                         jwk = self._get_jwk(private_key)
                         thumbprint = self._jwk_thumbprint(jwk)
                         key_auth = f"{token}.{thumbprint}"
+                        dns_txt = self._dns_txt_value(key_auth) if challenge_type == 'dns-01' else None
 
                         await conn.execute("""
-                            INSERT INTO acme_challenges (order_id, domain, token, key_authorization, challenge_url, status)
-                            VALUES ($1, $2, $3, $4, $5, $6)
+                            INSERT INTO acme_challenges
+                                (order_id, domain, token, key_authorization, challenge_url, status,
+                                 challenge_type, dns_txt_value)
+                            VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
                         """, order_id, domain, token, key_auth,
-                            challenge.get('url') or '', challenge.get('status') or 'pending')
-                        logger.info(f"ACME: Challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
+                            challenge.get('url') or '', challenge.get('status') or 'pending',
+                            challenge_type, dns_txt)
+                        logger.info(f"ACME: {challenge_type} challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
                         http01_for_domain = True
                 if http01_for_domain and domain:
                     domains_with_http01.add(domain)
@@ -413,7 +474,7 @@ class ACMEService:
                     error_payload, order_id
                 )
                 raise Exception(
-                    f"ACME order {order_id} created but no http-01 challenges available "
+                    f"ACME order {order_id} created but no {challenge_type} challenges available "
                     f"(auth fetch failures: {len(auth_fetch_failures)}). See order.error_detail for diagnostics."
                 )
             elif auth_fetch_failures:
@@ -449,10 +510,15 @@ class ACMEService:
         try:
             # Issue #12 / Commit 5a: include 'failed' challenges so they can be retried,
             # but rate-limit per challenge: max 5 attempts in last 5 minutes.
+            # DNS-01 skip-gate (the single safe choke point): never POST a challenge response for a
+            # dns-01 row whose TXT record has not been published yet — that would make the CA validate
+            # against a missing record and burn the order. http-01 rows (challenge_type 'http-01'/NULL)
+            # are never excluded, so the existing flow is byte-identical.
             challenges = await conn.fetch(
                 """SELECT * FROM acme_challenges
                    WHERE order_id = $1
-                     AND (status IN ('pending', 'failed') OR status IS NULL)""",
+                     AND (status IN ('pending', 'failed') OR status IS NULL)
+                     AND NOT (COALESCE(challenge_type, 'http-01') = 'dns-01' AND COALESCE(dns_record_published, FALSE) = FALSE)""",
                 order_id
             )
             order = await conn.fetchrow(

backend/services/csr_service.py

@@ -0,0 +1,461 @@
+"""
+csr_service: CSR (Certificate Signing Request) generation + signed-certificate
+import (v1.9.0).
+
+Flow:
+  1. `generate_csr_bundle` builds a private key + CSR locally (pure crypto,
+     no DB/IO — callers MUST run it via `asyncio.to_thread`: RSA-4096
+     generation takes seconds and would stall the single-worker event loop).
+  2. The bundle is persisted to `ssl_csrs` (`insert_csr_row`); the operator
+     downloads the CSR PEM and has it signed by an external CA.
+  3. `import_signed_certificate` pairs the CA response with the stored key,
+     creates a normal `ssl_certificates` row (source='csr',
+     last_config_status='PENDING' — agents never see it before Apply) and
+     NULLs the key copy on the CSR row.
+
+The CSR builder generalises the in-repo ACME reference
+(services/acme_service.py finalize_order): PEM output instead of DER, full
+subject instead of CN-only, ECDSA support, same PKCS8/NoEncryption key
+serialisation (the agent concatenates cert+key+chain into one PEM and HAProxy
+cannot read passphrase-protected keys).
+
+Private keys are ENCRYPTED AT REST from v1.10.1 (Issue #53): the Fernet token
+replaces the PEM in the same `ssl_csrs.private_key_pem` column, so there is no
+schema change and no SCHEMA_VERSION bump. Rows written earlier hold a raw PEM
+and are still read transparently — see utils/csr_key_crypto.py for the format
+discriminator and the key-rotation caveat. The pending CSR key is the one key
+in the system worth encrypting: it sits idle for the whole signing window and
+is never transmitted, unlike ssl_certificates.private_key_content and the ACME
+order keys, which agents must receive in plaintext on every poll.
+The key is NEVER returned by any CSR API response — `csr_row_to_dict` strips
+it unconditionally.
+"""
+
+import json
+import logging
+from typing import Any, Dict, List, Optional
+from types import SimpleNamespace
+
+import asyncpg
+from fastapi import HTTPException
+
+from cryptography import x509
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import NameOID
+
+from services import ssl_service
+from utils.csr_key_crypto import decrypt_csr_private_key, encrypt_csr_private_key
+
+logger = logging.getLogger(__name__)
+
+
+_KEY_FACTORIES = {
+    'rsa-2048': lambda: rsa.generate_private_key(public_exponent=65537, key_size=2048),
+    'rsa-4096': lambda: rsa.generate_private_key(public_exponent=65537, key_size=4096),
+    'ecdsa-p256': lambda: ec.generate_private_key(ec.SECP256R1()),
+    'ecdsa-p384': lambda: ec.generate_private_key(ec.SECP384R1()),
+}
+
+# (payload attribute, x509 OID, subject-JSON key)
+_SUBJECT_OID_MAP = [
+    ('organization', NameOID.ORGANIZATION_NAME, 'O'),
+    ('organizational_unit', NameOID.ORGANIZATIONAL_UNIT_NAME, 'OU'),
+    ('locality', NameOID.LOCALITY_NAME, 'L'),
+    ('state', NameOID.STATE_OR_PROVINCE_NAME, 'ST'),
+    ('country', NameOID.COUNTRY_NAME, 'C'),
+    ('email', NameOID.EMAIL_ADDRESS, 'emailAddress'),
+]
+
+
+def generate_csr_bundle(payload: Any) -> Dict[str, Any]:
+    """Generate a private key + CSR for a validated SSLCSRCreate payload.
+
+    Pure CPU-bound crypto — no DB, no network. Callers must offload via
+    `asyncio.to_thread` (see module docstring).
+
+    Returns {'csr_pem', 'private_key_pem', 'sans', 'subject'}.
+    """
+    key = _KEY_FACTORIES[payload.key_algorithm]()
+
+    attrs = [x509.NameAttribute(NameOID.COMMON_NAME, payload.common_name)]
+    subject_json: Dict[str, str] = {}
+    for attr_name, oid, json_key in _SUBJECT_OID_MAP:
+        value = getattr(payload, attr_name, None)
+        if value and str(value).strip():
+            cleaned = str(value).strip()
+            attrs.append(x509.NameAttribute(oid, cleaned))
+            subject_json[json_key] = cleaned
+
+    # CN always first in the SAN list, then the extra names, deduped with
+    # order preserved (mirrors the ACME flow where domains[0] is the CN).
+    sans = list(dict.fromkeys([payload.common_name, *(payload.sans or [])]))
+
+    builder = (
+        x509.CertificateSigningRequestBuilder()
+        .subject_name(x509.Name(attrs))
+        .add_extension(
+            x509.SubjectAlternativeName([x509.DNSName(d) for d in sans]),
+            critical=False,
+        )
+    )
+    csr = builder.sign(key, hashes.SHA256())
+
+    return {
+        'csr_pem': csr.public_bytes(serialization.Encoding.PEM).decode('utf-8'),
+        'private_key_pem': key.private_bytes(
+            serialization.Encoding.PEM,
+            serialization.PrivateFormat.PKCS8,
+            serialization.NoEncryption(),
+        ).decode('utf-8'),
+        'sans': sans,
+        'subject': subject_json,
+    }
+
+
+def diff_domains(csr_sans: Optional[List[str]], cert_domains: Optional[List[str]]) -> List[str]:
+    """Human-readable warnings for SAN drift between the CSR and the signed
+    certificate (case-insensitive set diff). CAs legitimately add/normalise
+    SANs, so drift is WARN-only — the hard gate is the key match."""
+    csr_set = {d.lower() for d in (csr_sans or []) if d}
+    cert_set = {d.lower() for d in (cert_domains or []) if d}
+    warnings: List[str] = []
+    added = sorted(cert_set - csr_set)
+    dropped = sorted(csr_set - cert_set)
+    if added:
+        warnings.append(
+            f"The CA added domains that were not in the CSR: {', '.join(added)}"
+        )
+    if dropped:
+        warnings.append(
+            f"The CA dropped domains that were requested in the CSR: {', '.join(dropped)}"
+        )
+    return warnings
+
+
+def _maybe_json_list(value: Any) -> List[str]:
+    """asyncpg returns JSONB columns as str unless a codec is registered."""
+    if isinstance(value, str):
+        try:
+            parsed = json.loads(value)
+            return parsed if isinstance(parsed, list) else []
+        except Exception:
+            return []
+    return list(value) if value else []
+
+
+def csr_row_to_dict(row: Any, include_pem: bool = False) -> Dict[str, Any]:
+    """Row → API dict. ALWAYS strips private_key_pem — the key never leaves
+    the server via a CSR endpoint. csr_pem included only on demand
+    (detail/create responses, not lists)."""
+    d = dict(row)
+    d.pop('private_key_pem', None)
+    if not include_pem:
+        d.pop('csr_pem', None)
+    for key in ('subject', 'sans'):
+        if key in d and isinstance(d[key], str):
+            try:
+                d[key] = json.loads(d[key])
+            except Exception:
+                pass
+    return d
+
+
+async def assert_csr_name_available(conn, name: str) -> None:
+    """Reject a CSR name that is already taken by an ACTIVE certificate or
+    another PENDING CSR. Called BEFORE key generation (cheap fail-fast) and
+    re-run inside `insert_csr_row` (the unique index closes the race)."""
+    existing_cert = await conn.fetchval(
+        "SELECT id FROM ssl_certificates WHERE name = $1 AND is_active = TRUE",
+        name,
+    )
+    if existing_cert:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"An active SSL certificate named '{name}' already exists. "
+                "The CSR name becomes the certificate name at import — choose "
+                "a different name or remove the existing certificate first."
+            ),
+        )
+    existing_csr = await conn.fetchval(
+        "SELECT id FROM ssl_csrs WHERE name = $1 AND status = 'pending'",
+        name,
+    )
+    if existing_csr:
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{name}' already exists (id={existing_csr}). "
+                "Import or delete it first, or choose a different name."
+            ),
+        )
+
+
+async def insert_csr_row(conn, payload: Any, bundle: Dict[str, Any], user_id: Optional[int]) -> int:
+    """Persist a freshly generated CSR bundle. Returns the new csr id.
+
+    Issue #53 (v1.10.1): the private key is Fernet-encrypted before it is stored. The token goes
+    into the SAME private_key_pem TEXT column — no schema change — and is only ever decrypted
+    in-process by import_signed_certificate. No CSR endpoint returns the column either way.
+    """
+    await assert_csr_name_available(conn, payload.name)
+    stored_key = encrypt_csr_private_key(bundle['private_key_pem'])
+    try:
+        csr_id = await conn.fetchval(
+            """
+            INSERT INTO ssl_csrs
+            (name, common_name, subject, sans, key_algorithm, csr_pem,
+             private_key_pem, status, created_by)
+            VALUES ($1, $2, $3::jsonb, $4::jsonb, $5, $6, $7, 'pending', $8)
+            RETURNING id
+            """,
+            payload.name,
+            payload.common_name,
+            json.dumps(bundle['subject']),
+            json.dumps(bundle['sans']),
+            payload.key_algorithm,
+            bundle['csr_pem'],
+            stored_key,
+            user_id,
+        )
+    except asyncpg.exceptions.UniqueViolationError:
+        # uq_ssl_csrs_name_pending — a concurrent request won the name.
+        raise HTTPException(
+            status_code=400,
+            detail=(
+                f"A pending CSR named '{payload.name}' was just created by a "
+                "concurrent request — choose a different name."
+            ),
+        )
+    return csr_id
+
+
+async def import_signed_certificate(conn, csr_id: int, imp: Any, user_id: Optional[int]) -> Dict[str, Any]:
+    """Pair the CA-signed certificate with the stored CSR key and create the
+    ssl_certificates row. Atomic: cert row + CSR state change commit together.
+
+    Returns {'certificate_id', 'certificate_name', 'primary_domain',
+    'warnings', 'reactivated'}. Raises HTTPException on every failure
+    (404 missing, 409 already completed, 400 validation).
+    """
+    async with conn.transaction():
+        # Row lock serialises concurrent imports AND a concurrent DELETE of
+        # the same CSR; works across multiple uvicorn workers (DB-level lock).
+        row = await conn.fetchrow(
+            "SELECT * FROM ssl_csrs WHERE id = $1 FOR UPDATE", csr_id
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="CSR not found")
+        if row['status'] == 'completed':
+            raise HTTPException(
+                status_code=409,
+                detail=(
+                    f"CSR '{row['name']}' is already completed — certificate "
+                    f"id {row['ssl_certificate_id']} was imported from it. "
+                    "Create a new CSR to reissue."
+                ),
+            )
+        if not row['private_key_pem']:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Stored CSR private key is missing — the CSR row is "
+                    "corrupt. Delete it and create a new CSR."
+                ),
+            )
+        # Issue #53: the column holds a Fernet token from v1.10.1 on, and a raw PEM for rows
+        # written before it. decrypt_csr_private_key accepts both, so no data migration is
+        # needed. A None here means the token cannot be decrypted — SECRET_KEY was rotated
+        # without CSR_ENCRYPTION_KEY set. Fail loudly: the key is gone, so the CA's certificate
+        # can never be paired with it, and silently falling through would surface as the far
+        # more confusing "certificate does not match this CSR's private key".
+        stored_key = decrypt_csr_private_key(row['private_key_pem'])
+        if not stored_key:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    f"The stored private key for CSR '{row['name']}' cannot be decrypted. This "
+                    "happens when SECRET_KEY was rotated while CSR_ENCRYPTION_KEY was not set. "
+                    "The key is unrecoverable, so this CSR can no longer be completed — delete "
+                    "it and create a new one (then have the new CSR signed)."
+                ),
+            )
+
+        effective_name = getattr(imp, 'name', None) or row['name']
+
+        # Parse the pasted certificate FIRST so a malformed/truncated CA
+        # response gets the manual flow's 400, not a 500 from the key-match
+        # step below (verify_certificate_key_match reports an unparseable
+        # cert as match=None, which we treat as an integrity failure).
+        from utils.ssl_parser import parse_ssl_certificate, verify_certificate_key_match
+        precheck = parse_ssl_certificate(imp.certificate_content)
+        if precheck.get('error'):
+            raise HTTPException(
+                status_code=400,
+                detail=f"Invalid SSL certificate: {precheck['error']}",
+            )
+
+        # THE defining check of this feature: the CA response must match the
+        # key we generated. Deliberately stricter than create_cert_row's
+        # lenient fallback — we generated this key ourselves, so an
+        # unverifiable pair is an integrity failure, not operator input.
+        match_result = verify_certificate_key_match(imp.certificate_content, stored_key)
+        if match_result.get('match') is False:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    "The signed certificate does not match this CSR's private "
+                    "key — the CA response likely belongs to a different "
+                    "CSR/key. Verify you pasted the certificate that was "
+                    "issued for this exact CSR."
+                ),
+            )
+        if match_result.get('match') is not True:
+            raise HTTPException(
+                status_code=500,
+                detail=(
+                    "Could not verify the certificate/key pair: "
+                    f"{match_result.get('reason', 'unknown')}"
+                ),
+            )
+
+        # Full parse/validation pipeline shared with the manual + wizard
+        # flows: invalid PEM, bad chain and already-expired certs all 400.
+        payload = SimpleNamespace(
+            name=effective_name,
+            certificate_content=imp.certificate_content,
+            private_key_content=stored_key,
+            chain_content=getattr(imp, 'chain_content', None),
+            usage_type=getattr(imp, 'usage_type', 'frontend') or 'frontend',
+        )
+        fields = ssl_service._prepare_cert_fields(payload)
+
+        # Global name uniqueness (ssl_certificates.cluster_id is always NULL
+        # under the R38 schema, so name is effectively a global namespace).
+        existing = await conn.fetchrow(
+            "SELECT id, is_active FROM ssl_certificates WHERE name = $1 LIMIT 1",
+            effective_name,
+        )
+        if existing and existing['is_active']:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    f"An active SSL certificate named '{effective_name}' "
+                    "already exists (created after this CSR). Delete or "
+                    "rename it, or pass a different `name` in the import "
+                    "request — the CSR stays pending and can be re-imported."
+                ),
+            )
+
+        reactivated = False
+        if existing and not existing['is_active']:
+            # Reactivate the soft-deleted row (mirrors create_cert_row):
+            # preserves the row id so historical references keep working.
+            await conn.execute(
+                "DELETE FROM ssl_certificate_clusters WHERE ssl_certificate_id = $1",
+                existing['id'],
+            )
+            await conn.execute(
+                """
+                UPDATE ssl_certificates
+                SET is_active = TRUE,
+                    last_config_status = 'PENDING',
+                    certificate_content = $2,
+                    private_key_content = $3,
+                    chain_content = $4,
+                    primary_domain = $5,
+                    all_domains = $6::jsonb,
+                    expiry_date = $7,
+                    usage_type = $8,
+                    issuer = $9,
+                    fingerprint = $10,
+                    status = $11,
+                    days_until_expiry = $12,
+                    source = 'csr',
+                    updated_at = CURRENT_TIMESTAMP
+                WHERE id = $1
+                """,
+                existing['id'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['primary_domain'],
+                json.dumps(fields['all_domains']),
+                fields['expiry_date'],
+                fields['usage_type'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+            )
+            cert_id = existing['id']
+            reactivated = True
+            logger.info(
+                f"csr_service.import_signed_certificate: reactivated "
+                f"soft-deleted cert '{effective_name}' (id={cert_id}) for CSR {csr_id}"
+            )
+        else:
+            cert_id = await conn.fetchval(
+                """
+                INSERT INTO ssl_certificates (
+                    name, primary_domain, certificate_content, private_key_content,
+                    chain_content, expiry_date, issuer, fingerprint, status,
+                    days_until_expiry, all_domains, is_active, cluster_id,
+                    last_config_status, usage_type, source
+                ) VALUES (
+                    $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb,
+                    TRUE, NULL, 'PENDING', $12, 'csr'
+                )
+                RETURNING id
+                """,
+                effective_name,
+                fields['primary_domain'],
+                fields['cert_content'],
+                fields['private_key_content'],
+                fields['chain_content'],
+                fields['expiry_date'],
+                fields['issuer'],
+                fields['fingerprint'],
+                fields['status'],
+                fields['days_until_expiry'],
+                json.dumps(fields['all_domains']),
+                fields['usage_type'],
+            )
+
+        # Cluster bindings: global = zero junction rows (existing convention).
+        if not getattr(imp, 'is_global', False):
+            for cluster_id in (getattr(imp, 'cluster_ids', None) or []):
+                await ssl_service.ensure_cluster_junction(conn, cert_id, cluster_id)
+
+        # Complete the CSR and destroy the key copy — the key now lives on
+        # the certificate row only, like every other key in the system.
+        await conn.execute(
+            """
+            UPDATE ssl_csrs
+            SET status = 'completed',
+                ssl_certificate_id = $2,
+                private_key_pem = NULL,
+                completed_at = CURRENT_TIMESTAMP,
+                updated_at = CURRENT_TIMESTAMP
+            WHERE id = $1
+            """,
+            csr_id,
+            cert_id,
+        )
+
+    warnings = diff_domains(_maybe_json_list(row['sans']), fields['all_domains'])
+    if reactivated:
+        warnings.append(
+            f"A soft-deleted certificate named '{effective_name}' was "
+            f"reactivated (row id {cert_id}) — existing entities that still "
+            "reference that id now serve the newly imported certificate."
+        )
+
+    return {
+        'certificate_id': cert_id,
+        'certificate_name': effective_name,
+        'primary_domain': fields['primary_domain'],
+        'warnings': warnings,
+        'reactivated': reactivated,
+    }

backend/services/dns01_orchestrator.py

@@ -0,0 +1,321 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): non-blocking per-cycle orchestration.
+
+Driven by the existing `complete_pending_acme_orders` background task (which already claims
+in-progress orders with `FOR UPDATE SKIP LOCKED`). For a dns-01 order this module advances AT MOST
+ONE step per 60s cycle — publish TXT, then (after a short min-age) respond — so the serial claim
+loop is never blocked by a multi-minute wait, and NO DNS library is needed (the CA is the source of
+truth; a propagation-lag `invalid` is recovered by a bounded fresh-order chain).
+
+Design invariants (from the hardening review):
+- Additive at the RRset level: publish/cleanup operate on a single (name, value), so wildcard+apex
+  (two values at one name) coexist.
+- No new order-status value: a failed order stays `invalid`; a boolean `dns01_retry_claimed` does the
+  winner-only CAS + claim exclusion, so existing `status` consumers are untouched.
+- Secrets (provider API tokens) are NEVER logged or written to error_detail/events.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta, timezone
+from typing import Dict, List, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+from services.acme_service import acme_service as acme_svc, ACMEService
+from services.dns_providers import get_provider, is_supported, DnsProviderError
+from utils.dns_credentials import decrypt_dns_credentials
+from utils.activity_log import record_event
+
+logger = logging.getLogger(__name__)
+
+# Tunables (kept conservative vs Let's Encrypt rate limits: 5 failed-validations/host/hour,
+# 300 new-orders/account/3h).
+PROPAGATION_GRACE_SECONDS = 25          # min age before we tell the CA to validate
+MANUAL_CONFIRM_TTL = timedelta(hours=48)
+MAX_RETRIES = 3                          # bounded fresh-order chain (1 original + 3 retries = 4 orders)
+# Retry backoff floor (minutes) indexed by the order's current dns01_attempts: [15, 30, 60].
+# The AUTHORITATIVE implementation is the SQL CASE in main.py's claim query
+# (complete_pending_acme_orders), so the backoff is evaluated atomically with the
+# FOR UPDATE SKIP LOCKED claim. Documented here only — do not reintroduce a second copy.
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _aware(dt) -> Optional[datetime]:
+    if dt is None:
+        return None
+    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+
+
+async def _load_credentials(conn, account_id: int) -> Tuple[Optional[Dict[str, str]], bool]:
+    """Return (credentials_dict_or_None, row_exists). credentials None + row_exists True means the
+    stored token could not be decrypted (e.g. SECRET_KEY rotated)."""
+    row = await conn.fetchrow(
+        "SELECT credentials_encrypted FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+        account_id,
+    )
+    if not row:
+        return None, False
+    return decrypt_dns_credentials(row["credentials_encrypted"]), True
+
+
+async def _fail_order(conn, order_id: int, reason: str) -> None:
+    """Mark an order invalid with a sanitized reason (no secrets) + event."""
+    import json
+    payload = json.dumps({"stage": "dns01", "reason": reason, "timestamp": _now().isoformat()})
+    await conn.execute(
+        "UPDATE letsencrypt_orders SET status = 'invalid', error_detail = $1, updated_at = NOW() WHERE id = $2",
+        payload, order_id,
+    )
+    await record_event(order_id, "acme.dns01.validation", severity="ERROR", message=reason, conn=conn)
+
+
+async def advance_dns01_order(order_id: int) -> None:
+    """One non-blocking step for a claimed pending/processing dns-01 order. No-op for http-01 or
+    orders not in a publishable state. Safe to call every cycle (idempotent via CAS flags)."""
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, o.account_id, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if order["status"] not in ("pending", "processing"):
+            return
+
+        challenges = await conn.fetch(
+            "SELECT * FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'", order_id
+        )
+        if not challenges:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+
+        # --- Manual provider: the user publishes + confirms; we only enforce the deadline. ---
+        if provider_name == "manual" or not is_supported(provider_name):
+            deadline = _aware(challenges[0]["manual_confirm_deadline"])
+            if deadline is None:
+                new_deadline = _now() + MANUAL_CONFIRM_TTL
+                await conn.execute(
+                    "UPDATE acme_challenges SET manual_confirm_deadline = $1 WHERE order_id = $2 AND manual_confirm_deadline IS NULL",
+                    new_deadline, order_id,
+                )
+            elif _now() > deadline:
+                await _fail_order(conn, order_id,
+                                  "Manual DNS-01 confirmation deadline passed without confirmation.")
+            return  # respond happens via the dns-confirm endpoint
+
+        # --- Automated provider (e.g. Cloudflare). ---
+        creds, row_exists = await _load_credentials(conn, order["account_id"])
+        if not row_exists:
+            await _fail_order(conn, order_id,
+                              f"No DNS provider credentials configured for provider '{provider_name}'.")
+            return
+        if creds is None:
+            await _fail_order(conn, order_id,
+                              "DNS provider credentials could not be decrypted; re-enter them in Settings.")
+            return
+
+        provider = get_provider(provider_name, creds)
+
+        # Publish any not-yet-published challenge (CAS so two replicas can't double-publish).
+        for ch in challenges:
+            if ch["dns_record_published"]:
+                continue
+            flipped = await conn.fetchval(
+                """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = NOW()
+                   WHERE id = $1 AND dns_record_published = FALSE RETURNING id""",
+                ch["id"],
+            )
+            if not flipped:
+                continue
+            name = ACMEService._challenge_dns_name(ch["domain"])
+            try:
+                await provider.add_txt_record(name, ch["dns_txt_value"])
+                await record_event(order_id, "acme.dns01.publish",
+                                   message=f"Published TXT {name}; waiting for DNS propagation before asking the CA to validate.",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+            except DnsProviderError as exc:
+                # Revert so the next cycle retries the publish; keep the order pending. exc is sanitized.
+                await conn.execute(
+                    "UPDATE acme_challenges SET dns_record_published = FALSE, dns_published_at = NULL WHERE id = $1",
+                    ch["id"],
+                )
+                await record_event(order_id, "acme.dns01.publish", severity="WARNING",
+                                   message=f"Publish failed for {name}: {exc}",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+                return
+
+        # All published? Then respond once the min-age gate has elapsed (across cycles, no sleep).
+        rows = await conn.fetch(
+            "SELECT dns_record_published, dns_published_at FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+            order_id,
+        )
+        if any(not r["dns_record_published"] for r in rows):
+            return
+        published_ats = [_aware(r["dns_published_at"]) for r in rows if r["dns_published_at"]]
+        if not published_ats:
+            return
+        if (_now() - min(published_ats)).total_seconds() < PROPAGATION_GRACE_SECONDS:
+            return  # wait one more cycle
+
+        # Only POST the challenge response if something still needs validating — avoids re-POSTing
+        # every cycle (and bumping dns01_last_attempt_at) once the CA already has them processing.
+        still_pending = await conn.fetchval(
+            """SELECT 1 FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'
+               AND (status IN ('pending', 'failed') OR status IS NULL) LIMIT 1""",
+            order_id,
+        )
+        if not still_pending:
+            return
+
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Told the CA to validate the DNS-01 challenge(s).", conn=conn)
+    finally:
+        await close_database_connection(conn)
+
+
+async def confirm_manual_dns01(order_id: int) -> Dict:
+    """Called by POST /orders/{id}/dns-confirm for the manual provider: mark the TXT published and
+    tell the CA to validate. Returns a small status dict."""
+    conn = await get_database_connection()
+    try:
+        await conn.execute(
+            """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = COALESCE(dns_published_at, NOW())
+               WHERE order_id = $1 AND challenge_type = 'dns-01'""",
+            order_id,
+        )
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Manual DNS-01 confirmed; told the CA to validate.", conn=conn)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+async def retry_invalid_dns01(order_id: int) -> None:
+    """Bounded fresh-order recovery for a dns-01 order that went `invalid` (e.g. propagation lag).
+    Winner-only CAS on `dns01_retry_claimed`; cleans the old TXT, mints a child order. No-op for
+    http-01 or when the budget is exhausted."""
+    conn = await get_database_connection()
+    child_created = False
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.*, a.dns_provider FROM letsencrypt_orders o
+               JOIN letsencrypt_accounts a ON o.account_id = a.id WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if (order["dns01_attempts"] or 0) >= MAX_RETRIES:
+            return  # budget exhausted; stays terminal invalid
+
+        # Winner-only claim (closes the cross-replica double-mint race).
+        claimed = await conn.fetchval(
+            """UPDATE letsencrypt_orders SET dns01_retry_claimed = TRUE, updated_at = NOW()
+               WHERE id = $1 AND status = 'invalid' AND dns01_retry_claimed = FALSE RETURNING id""",
+            order_id,
+        )
+        if not claimed:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+        # Best-effort cleanup of this order's TXT before minting the replacement.
+        if provider_name != "manual" and is_supported(provider_name):
+            creds, _exists = await _load_credentials(conn, order["account_id"])
+            if creds:
+                provider = get_provider(provider_name, creds)
+                chs = await conn.fetch(
+                    "SELECT domain, dns_txt_value FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+                    order_id,
+                )
+                for ch in chs:
+                    try:
+                        await provider.remove_txt_record(ACMEService._challenge_dns_name(ch["domain"]), ch["dns_txt_value"])
+                    except DnsProviderError:
+                        pass  # tolerate; the reconcile sweep will retry
+        await conn.execute(
+            "UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE order_id = $1 AND dns_record_published = TRUE",
+            order_id,
+        )
+
+        import json
+        domains = json.loads(order["domains"]) if isinstance(order["domains"], str) else (order["domains"] or [])
+        cluster_ids = json.loads(order["cluster_ids"]) if isinstance(order["cluster_ids"], str) else (order["cluster_ids"] or [])
+        next_attempts = (order["dns01_attempts"] or 0) + 1
+        child = await acme_svc.create_order(
+            order["account_id"], domains, cluster_ids, challenge_type="dns-01",
+            created_by=order["created_by"],
+        )
+        child_created = True
+        await conn.execute(
+            """UPDATE letsencrypt_orders
+               SET dns01_attempts = $1, dns01_parent_order_id = $2, dns01_last_attempt_at = NOW()
+               WHERE id = $3""",
+            next_attempts, order_id, child["order_id"],
+        )
+        await record_event(order_id, "acme.dns01.validation", severity="WARNING",
+                           message=f"DNS-01 order invalid; minted retry #{next_attempts} (order {child['order_id']}).",
+                           details={"child_order_id": child["order_id"], "attempt": next_attempts}, conn=conn)
+    except Exception as exc:  # noqa: BLE001
+        logger.error(f"[DNS01-RETRY] order {order_id}: {exc}")
+        # A transient failure (e.g. CA rate limit) BEFORE the child was minted must NOT permanently
+        # burn the retry slot — reset the claim so the next cycle can retry. If the child was already
+        # created, leave the claim set (resetting would double-mint).
+        if not child_created:
+            try:
+                await conn.execute(
+                    "UPDATE letsencrypt_orders SET dns01_retry_claimed = FALSE WHERE id = $1 AND status = 'invalid'",
+                    order_id,
+                )
+            except Exception:
+                pass
+    finally:
+        await close_database_connection(conn)
+
+
+async def reconcile_dns01_cleanup() -> None:
+    """Best-effort sweep that removes any TXT records left published for terminal orders (covers a
+    cleanup that failed, or the kill-switch being flipped off mid-flight). NOT gated by the
+    kill-switch. Runs once per completion cycle."""
+    conn = await get_database_connection()
+    try:
+        rows = await conn.fetch(
+            """SELECT c.id AS chal_id, c.order_id, c.domain, c.dns_txt_value, o.account_id, a.dns_provider
+               FROM acme_challenges c
+               JOIN letsencrypt_orders o ON c.order_id = o.id
+               JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE c.challenge_type = 'dns-01'
+                 AND c.dns_record_published = TRUE
+                 AND COALESCE(c.dns_record_cleaned, FALSE) = FALSE
+                 AND o.status IN ('valid', 'invalid', 'cancelled')
+               LIMIT 50""",
+        )
+        for r in rows:
+            provider_name = (r["dns_provider"] or "manual").strip()
+            if provider_name == "manual" or not is_supported(provider_name):
+                # Manual: nothing to call; mark cleaned so we stop revisiting.
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                continue
+            creds, _exists = await _load_credentials(conn, r["account_id"])
+            if creds is None:
+                continue  # can't clean without creds; leave for a later pass
+            provider = get_provider(provider_name, creds)
+            try:
+                await provider.remove_txt_record(ACMEService._challenge_dns_name(r["domain"]), r["dns_txt_value"])
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                await record_event(r["order_id"], "acme.dns01.cleanup",
+                                   message=f"Cleaned up TXT for {r['domain']}", conn=conn)
+            except DnsProviderError:
+                pass  # retry next sweep
+    except Exception as exc:  # noqa: BLE001
+        logger.debug(f"[DNS01-RECONCILE] skipped: {exc}")
+    finally:
+        await close_database_connection(conn)

backend/services/dns_providers/__init__.py

@@ -0,0 +1,14 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): pluggable DNS provider package.
+
+A small adapter layer so DNS-01 challenges can publish/clean up the
+`_acme-challenge.<domain>` TXT record via different DNS providers. The interface is
+additive at the RRset level (add/remove a single value by name+content, never
+overwrite-by-name) so multiple coexisting values at one name (wildcard + apex) work.
+
+Providers: manual (user publishes the TXT themselves), Cloudflare, and GoDaddy (v1.10.0). New
+providers plug in via the registry without touching the orchestration.
+"""
+from .base import DnsProvider, DnsProviderError
+from .registry import get_provider, list_providers, is_supported
+
+__all__ = ["DnsProvider", "DnsProviderError", "get_provider", "list_providers", "is_supported"]

backend/services/dns_providers/base.py

@@ -0,0 +1,53 @@
+"""Abstract DNS provider interface for ACME DNS-01 (Issue #35)."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Dict, List
+
+
+class DnsProviderError(Exception):
+    """A DNS provider failure with a SANITIZED, user-safe message.
+
+    The message must NEVER contain API tokens, request headers, or other secrets — it is
+    persisted to acme_order_events / order error_detail and shown in the UI. Raise this (not a
+    raw aiohttp/json error) so credentials can't leak into logs or the order timeline.
+    """
+
+
+class DnsProvider(ABC):
+    """Base class for a pluggable DNS provider.
+
+    RRset semantics are ADDITIVE: ``add_txt_record`` ensures a (name, value) TXT exists WITHOUT
+    removing other values at the same name, and ``remove_txt_record`` deletes ONLY the record
+    matching (name, value). This is required because a cert for ``example.com`` + ``*.example.com``
+    publishes two distinct values at the SAME name ``_acme-challenge.example.com``.
+    """
+
+    # Stable machine name (used in DB + API); human label; whether the provider automates publishing.
+    name: str = "base"
+    label: str = "Base"
+    automated: bool = True
+
+    # Declarative schema the UI renders to collect credentials. Each field:
+    #   {"key", "label", "type" ("text"|"password"), "required" (bool), "max_length" (int), "help" (str)}
+    credential_fields: List[Dict] = []
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        self.credentials = credentials or {}
+
+    @abstractmethod
+    async def verify_credentials(self) -> Dict:
+        """Validate the stored credentials against the provider. Returns
+        ``{"ok": bool, "detail": str}`` (detail is user-safe). Must not raise on auth failure —
+        return ``ok=False`` with a sanitized detail; may raise DnsProviderError on transport errors.
+        """
+
+    @abstractmethod
+    async def add_txt_record(self, name: str, value: str) -> None:
+        """Ensure a TXT record (name, value) exists. Idempotent; must not remove other values
+        at the same name. Raise DnsProviderError (sanitized) on failure."""
+
+    @abstractmethod
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        """Remove ONLY the TXT record matching (name, value). Tolerate 'already gone'.
+        Raise DnsProviderError (sanitized) on a real failure."""

backend/services/dns_providers/cloudflare.py

@@ -0,0 +1,179 @@
+"""Cloudflare DNS provider for ACME DNS-01 (Issue #35).
+
+Uses the Cloudflare API v4 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name influences which zone is used). Errors are wrapped in DnsProviderError with a
+sanitized message so the API token never reaches logs / order events.
+
+Token scope required: Zone:DNS:Edit + Zone:Read.
+"""
+from __future__ import annotations
+
+import logging
+import re
+from typing import Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+CLOUDFLARE_API_BASE = "https://api.cloudflare.com/client/v4"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# Characters NOT valid in an HTTP bearer credential (RFC 6750 token68: A-Za-z0-9-._~+/=).
+# Cloudflare API tokens are a strict subset of this set, so removing anything outside it can
+# never corrupt a valid token, but it does strip the paste artifacts that make Cloudflare
+# reject the Authorization header with HTTP 400 "Invalid request headers" (CF code 6003):
+# surrounding/embedded quotes, interior spaces/tabs, zero-width/unicode chars, and CR/LF
+# (the latter would otherwise make aiohttp raise client-side before the request is even sent).
+_NON_TOKEN68 = re.compile(r"[^A-Za-z0-9._~+/=-]")
+
+
+def _strip_quotes(s: str) -> str:
+    s = (s or "").strip()
+    if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
+        return s[1:-1]
+    return s
+
+
+def _sanitize_token(s: str) -> str:
+    """Strip surrounding quotes/whitespace, then drop every character outside the token68 set."""
+    return _NON_TOKEN68.sub("", _strip_quotes(s))
+
+
+class CloudflareDNSProvider(DnsProvider):
+    name = "cloudflare"
+    label = "Cloudflare"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_token",
+            "label": "API Token",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": "Scoped API token with Zone:DNS:Edit and Zone:Read permissions.",
+        }
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        self._raw_token = (self.credentials.get("api_token") or "").strip()
+        # Sanitize to the token68 set so a pasted token with quotes/spaces/control/unicode chars
+        # cannot produce an invalid Authorization header (CF 6003 "Invalid request headers").
+        self._token = _sanitize_token(self._raw_token)
+
+    def _headers(self) -> Dict[str, str]:
+        return {"Authorization": f"Bearer {self._token}", "Content-Type": "application/json"}
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> dict:
+        """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
+        DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
+        url = f"{CLOUDFLARE_API_BASE}{path}"
+        try:
+            async with session.request(
+                method, url, headers=self._headers(), allow_redirects=False, **kwargs
+            ) as resp:
+                try:
+                    body = await resp.json()
+                except Exception:  # noqa: BLE001
+                    body = {}
+                if resp.status in (401, 403):
+                    raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                if resp.status >= 400 or not body.get("success", False):
+                    # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                    # human message text, never the request (which carries the token header).
+                    msgs = "; ".join(
+                        str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                    )
+                    raise DnsProviderError(
+                        f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                    )
+                return body
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Do NOT include exc verbatim everywhere; aiohttp client errors are URL/transport only
+            # (no token), but keep the message generic and stable.
+            raise DnsProviderError(f"Could not reach the Cloudflare API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected Cloudflare API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._token:
+            return {"ok": False, "detail": "No Cloudflare API token provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                body = await self._request(session, "GET", "/zones?per_page=1")
+            total = ((body.get("result_info") or {}).get("total_count"))
+            detail = "Cloudflare token valid."
+            if isinstance(total, int):
+                detail = f"Cloudflare token valid; {total} zone(s) visible."
+            return {"ok": True, "detail": detail}
+        except DnsProviderError as exc:
+            # Always surface the real Cloudflare reason (e.g. token scope). If sanitizing also changed
+            # the token, append a hint that stray characters were stripped (never echo the token).
+            detail = str(exc)
+            if self._raw_token != self._token:
+                detail += (" Note: the token contained characters that were stripped; if it still "
+                           "fails, re-copy it from Cloudflare without quotes or spaces.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the Cloudflare token."}
+
+    async def _resolve_zone(self, session: aiohttp.ClientSession, record_name: str) -> Tuple[str, str]:
+        """Find the most-specific (longest-suffix) managed zone for a record name.
+        Returns (zone_id, zone_name). Raises DnsProviderError if no zone matches."""
+        labels = record_name.split(".")
+        # Walk suffixes from longest to shortest; a zone needs at least 2 labels.
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break
+            body = await self._request(
+                session, "GET", f"/zones?name={quote(candidate)}&status=active&per_page=50"
+            )
+            results = body.get("result") or []
+            if results:
+                return results[0]["id"], candidate
+        raise DnsProviderError(f"No managed Cloudflare zone found for {record_name}.")
+
+    async def _find_record_id(
+        self, session: aiohttp.ClientSession, zone_id: str, name: str, value: str
+    ) -> Optional[str]:
+        body = await self._request(
+            session, "GET", f"/zones/{zone_id}/dns_records?type=TXT&name={quote(name)}&per_page=100"
+        )
+        for rec in body.get("result") or []:
+            if _strip_quotes(rec.get("content", "")) == value:
+                return rec.get("id")
+        return None
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            zone_id, _zone_name = await self._resolve_zone(session, name)
+            # Idempotent: only create if (name, value) is not already present (preserves coexisting values).
+            existing = await self._find_record_id(session, zone_id, name, value)
+            if existing:
+                return
+            await self._request(
+                session,
+                "POST",
+                f"/zones/{zone_id}/dns_records",
+                json={"type": "TXT", "name": name, "content": value, "ttl": 120},
+            )
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            try:
+                zone_id, _zone_name = await self._resolve_zone(session, name)
+            except DnsProviderError:
+                # Zone gone / not resolvable — nothing we can clean up.
+                return
+            record_id = await self._find_record_id(session, zone_id, name, value)
+            if not record_id:
+                return  # already gone — tolerate
+            await self._request(session, "DELETE", f"/zones/{zone_id}/dns_records/{record_id}")

backend/services/dns_providers/godaddy.py

@@ -0,0 +1,512 @@
+"""GoDaddy DNS provider for ACME DNS-01 (Issue #35 follow-up, v1.10.0).
+
+Uses the GoDaddy Domains API v1 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name selects which zone is touched), which is the same reason cloudflare.py is exempt from
+utils/ssrf_guard.py. Every failure is wrapped in DnsProviderError with a SANITIZED message: the
+API Key and Secret are scrubbed out of any text that could reach a log, an order event, or
+letsencrypt_orders.error_detail.
+
+Two GoDaddy-specific hazards drive the shape of this module — neither exists on Cloudflare:
+
+1. NO PER-VALUE WRITE. `PUT /v1/domains/{d}/records/TXT/{name}` REPLACES the entire RRset at that
+   type+name; it does not merge. A certificate for `example.com` + `*.example.com` publishes two
+   DIFFERENT TXT values at the SAME name `_acme-challenge.example.com` (base.py's additive
+   contract), so a naive single-value PUT would silently destroy the sibling and fail the wildcard
+   authorization. Every mutation here is therefore read-modify-write: GET the current RRset, merge,
+   PUT the whole list back. An EMPTY array is rejected (422 INVALID_BODY, "Records must be
+   specified"), so removing the LAST value must use DELETE — never `PUT []`.
+
+2. ZONE-DESTRUCTIVE SIBLING PATHS. `PUT /v1/domains/{d}/records/TXT` (three segments, no name)
+   wipes EVERY TXT in the zone — SPF, DKIM, DMARC, Microsoft/Google verification — and
+   `PUT /v1/domains/{d}/records` wipes the whole zone (this is dehydrated issue #430 verbatim).
+   The record path is built only by _rrset_path(), which refuses an empty zone or relative name so
+   a URL can never collapse onto one of those endpoints.
+
+Concurrency: v1 has no ETag, no If-Match and no per-record id, so read-modify-write can lose an
+update if two mutations at one name overlap. Today they cannot: orders are advanced sequentially
+(`for oid in claimed_ids: await advance_dns01_order(oid)` in main.py) and an order's challenges are
+published sequentially (`for ch in challenges: await provider.add_txt_record(...)` in
+dns01_orchestrator.py), so the apex+wildcard pair is strictly ordered and the second publish sees
+the first. _rrset_lock() makes that safety structural rather than incidental. Across REPLICAS the
+window is real but narrow (two orders publishing at the same record name in overlapping cycles) and
+self-healing: a lost publish ends `invalid` and the bounded retry chain mints a fresh order, a lost
+cleanup is retried by the reconcile sweep, and an orphaned `_acme-challenge` TXT is inert. The real
+fix is the v3 API (POST + DELETE by recordId, natively per-value), which is PAT-only and a
+follow-up; it is deliberately not used here because v1 + sso-key is what operators can use today.
+
+Credentials: an API Key + Secret pair from https://developer.godaddy.com/keys. It must be a
+PRODUCTION key — the first key the dashboard issues is an OTE (test) key and an OTE credential
+against api.godaddy.com returns 401. A Personal Access Token also works: paste it as the API Key
+and leave the Secret blank, and the Authorization header becomes `Bearer <token>`. That path is not
+cosmetic — GoDaddy marks sso-key "deprecated, supported through 2026" and the current v1 OpenAPI
+advertises only bearer auth, so the PAT is the migration target, not an alternative.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any, Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+GODADDY_API_BASE = "https://api.godaddy.com/v1"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# GoDaddy enforces a 600s (10 min) TTL floor at request time. The published v1 OpenAPI declares no
+# minimum, so a smaller value is not caught by the schema — it fails with
+# 422 {"code":"INVALID_BODY","fields":[{"message":"must have a minimum value of 600", ...}]}.
+# Pin the floor; DNS-01 has no reason to want anything longer.
+_TXT_TTL = 600
+
+# Read-modify-write serialization, keyed by the RRset (record name), not the zone — the RRset is the
+# actual unit of contention, and keying on it avoids serializing unrelated subdomains of one zone.
+# The orchestrator is sequential today (see the module docstring), so this is defence in depth: it
+# is what stops a future `asyncio.gather()` over the publish loop from silently breaking every
+# wildcard+apex certificate. Bounded in practice by the certificate inventory of one process, so
+# there is no eviction; the entries are empty Lock objects.
+_RRSET_LOCKS: Dict[str, asyncio.Lock] = {}
+
+
+def _rrset_lock(record_name: str) -> asyncio.Lock:
+    key = (record_name or "").rstrip(".").lower()
+    lock = _RRSET_LOCKS.get(key)
+    if lock is None:
+        # Safe without a guard: a single event loop never preempts between the get and the assign.
+        lock = _RRSET_LOCKS[key] = asyncio.Lock()
+    return lock
+
+
+def _scrub(text: str, *secrets: str) -> str:
+    """Remove credential substrings from a message before it can reach a log or an order event.
+
+    GoDaddy error bodies do not echo the Authorization header, so this is belt-and-braces — but it
+    makes base.py's "never leak a secret" invariant structural instead of a matter of care. Short
+    strings are skipped so a 1-2 char credential fragment cannot blank out ordinary prose.
+    """
+    out = text or ""
+    for secret in secrets:
+        if secret and len(secret) >= 4:
+            out = out.replace(secret, "***")
+    return out[:300]
+
+
+def _relative_name(fqdn: str, zone: str) -> str:
+    """Convert an absolute record name to the zone-relative form GoDaddy's API requires.
+
+    GoDaddy record names are RELATIVE to the zone with NO trailing dot, and the zone apex is the
+    literal "@" — never an empty string (which would collapse the URL onto the zone-wide TXT
+    endpoint) and never the domain name itself.
+
+      ("_acme-challenge.example.com",         "example.com") -> "_acme-challenge"
+      ("_acme-challenge.foo.bar.example.com", "example.com") -> "_acme-challenge.foo.bar"
+      ("example.com",                         "example.com") -> "@"
+    """
+    f = (fqdn or "").rstrip(".").lower()
+    z = (zone or "").rstrip(".").lower()
+    if z and f == z:
+        return "@"
+    if z and f.endswith("." + z):
+        return f[: -(len(z) + 1)]
+    # Defensive: callers always pass a zone that _resolve_domain derived from this very name.
+    return f or "@"
+
+
+def _rrset_path(zone: str, rel_name: str) -> str:
+    """Build the 4-segment record path `/domains/{zone}/records/TXT/{name}`.
+
+    SAFETY GATE: an empty rel_name would collapse the URL to `/domains/{zone}/records/TXT` — the
+    endpoint that replaces EVERY TXT record in the zone (SPF, DKIM, DMARC, domain verifications).
+    A "." or ".." segment does the same thing one step later: `quote()` leaves both untouched
+    (they are unreserved) and yarl normalizes dot segments away when it builds the URL, so
+    ".../records/TXT/.." would resolve to ".../records" — the whole-zone endpoint. Refuse both
+    rather than build them. `safe=''` percent-encodes the apex "@" as "%40" (accepted bare too,
+    but safer through proxies); "_", "-" and "." are unreserved and pass through unchanged, so a
+    multi-label relative name stays one readable path segment.
+    """
+    if not zone or not rel_name:
+        raise DnsProviderError("Internal error: refusing to build a zone-wide GoDaddy TXT record path.")
+    if rel_name.strip(".") == "" or any(part in (".", "..") for part in rel_name.split("/")):
+        raise DnsProviderError("Internal error: refusing to build a GoDaddy TXT path from a dot segment.")
+    return f"/domains/{quote(zone, safe='')}/records/TXT/{quote(rel_name, safe='')}"
+
+
+def _live_values(records: List[Dict]) -> List[str]:
+    """The non-empty `data` values in an RRset read.
+
+    GoDaddy leaves tombstone rows with `"data": ""` behind at a name after some removals. Echoing
+    one back in a PUT body is rejected with 422 INVALID_BODY, so every field implementation
+    (lego, acme.sh, Posh-ACME) filters them independently — so do we.
+    """
+    out: List[str] = []
+    for rec in records or []:
+        data = (rec or {}).get("data") or ""
+        if data:
+            out.append(data)
+    return out
+
+
+def _merge_add(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that adds `value` while preserving every coexisting sibling value.
+
+    Returns None when `value` is already present — an idempotent no-op, which is where an ACME
+    retry cycle lands.
+    """
+    live = _live_values(existing)
+    if value in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live] + [{"data": value, "ttl": _TXT_TTL}]
+
+
+def _merge_remove(existing: List[Dict], value: str) -> Optional[List[Dict]]:
+    """PUT body that removes ONLY `value`, keeping every sibling.
+
+    Three-state result, because GoDaddy needs three different calls:
+      None -> `value` is not there; already gone, tolerate (base.py's remove contract).
+      []   -> it was the last value; the caller must DELETE, since `PUT []` is rejected.
+      list -> PUT this body.
+    """
+    live = _live_values(existing)
+    if value not in live:
+        return None
+    return [{"data": d, "ttl": _TXT_TTL} for d in live if d != value]
+
+
+def _require_rrset(body: Any) -> List[Dict]:
+    """The RRset read, or a refusal.
+
+    FAIL CLOSED. A read that did not come back as a JSON array must never be treated as "the RRset
+    is empty" — the very next call is a full-RRset PUT, so coercing an unreadable read to [] would
+    replace every coexisting sibling value with just ours. Failing instead is free: the orchestrator
+    reverts the publish flag and retries next cycle, while a destructive PUT is unrecoverable.
+    """
+    if not isinstance(body, list):
+        raise DnsProviderError(
+            "GoDaddy returned an unreadable TXT record list; refusing to replace the record set."
+        )
+    return body
+
+
+def _error_fields(body: Any) -> Tuple[str, str]:
+    """The whitelisted (code, message) pair from a GoDaddy error body.
+
+    Only these two string fields are ever read; the raw body is never interpolated into a
+    user-facing message.
+    """
+    if not isinstance(body, dict):
+        return "", ""
+    code = body.get("code")
+    message = body.get("message")
+    return (code if isinstance(code, str) else ""), (message if isinstance(message, str) else "")
+
+
+def _retry_after_seconds(headers, body: Any) -> int:
+    """Seconds to wait after a 429.
+
+    The current platform sends `Retry-After` and `ratelimit-reset` headers with no body, while the
+    legacy v1 OpenAPI documents an `ErrorLimit` body carrying `retryAfterSec`. All three shapes are
+    live in the wild — and so is none of them, hence the 60s default.
+    """
+    for key in ("Retry-After", "ratelimit-reset"):
+        raw = (headers or {}).get(key)
+        if raw:
+            try:
+                return max(1, int(str(raw).strip()))
+            except (TypeError, ValueError):
+                pass
+    if isinstance(body, dict):
+        raw = body.get("retryAfterSec")
+        if isinstance(raw, int) and raw > 0:
+            return raw
+    return 60
+
+
+class _GoDaddyHTTPError(DnsProviderError):
+    """A DnsProviderError that also carries the HTTP status and GoDaddy `code`.
+
+    Callers INSIDE this module branch on the status (tolerate a 404 read-back, fall through a
+    zone probe), while everything outside — dns01_orchestrator, letsencrypt.py — still sees a
+    plain sanitized DnsProviderError and needs no change.
+    """
+
+    def __init__(self, message: str, status: int, code: str = ""):
+        super().__init__(message)
+        self.status = status
+        self.code = code
+
+
+class GoDaddyDNSProvider(DnsProvider):
+    name = "godaddy"
+    label = "GoDaddy"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_key",
+            "label": "API Key",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": ("Production API Key from developer.godaddy.com/keys — the first key the dashboard "
+                     "issues is an OTE (test) key and will be rejected. A Personal Access Token also "
+                     "works: paste it here and leave the Secret blank."),
+        },
+        {
+            "key": "api_secret",
+            "label": "API Secret",
+            "type": "password",
+            "required": False,
+            "max_length": 200,
+            "help": ("The Secret half of the same API Key pair. Leave blank ONLY if the field above "
+                     "holds a Personal Access Token. The account also needs at least one registered "
+                     "domain for GoDaddy to allow DNS API access at all."),
+        },
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        # Normalize, never validate: dns01_orchestrator.py calls get_provider() OUTSIDE any
+        # DnsProviderError guard, so a constructor that raised on malformed credentials would escape
+        # as an unhandled exception in the 60s background cycle. The UI drops blank fields before
+        # submitting, so a left-blank field arrives as a MISSING key rather than "" — `.get() or ""`
+        # covers both.
+        self._api_key = (self.credentials.get("api_key") or "").strip()
+        self._api_secret = (self.credentials.get("api_secret") or "").strip()
+        # Per-INSTANCE zone cache. A module-level cache would leak one ACME account's zone visibility
+        # into another's; an instance lives for exactly one orchestrator step, which is precisely the
+        # scope where caching pays off (apex + wildcard resolve the same zone from the same name).
+        self._zone_cache: Dict[str, str] = {}
+
+    def _auth_header(self) -> str:
+        """`sso-key <key>:<secret>` when a Secret is present, else `Bearer <token>` for a PAT.
+
+        Literal prefix, one space, a single colon — no base64, no URL-encoding, no quotes. Keeping
+        this as one swappable string is what makes GoDaddy's sso-key sunset a credential change
+        rather than a code change.
+        """
+        if self._api_secret:
+            return f"sso-key {self._api_key}:{self._api_secret}"
+        return f"Bearer {self._api_key}"
+
+    def _headers(self) -> Dict[str, str]:
+        # Accept is not optional: these endpoints content-negotiate application/xml and
+        # text/javascript. Content-Type is required on every write or GoDaddy answers 400/415.
+        return {
+            "Authorization": self._auth_header(),
+            "Accept": "application/json",
+            "Content-Type": "application/json",
+        }
+
+    def _http_error(self, status: int, code: str, message: str, retry_after: Optional[int]) -> _GoDaddyHTTPError:
+        """Map an HTTP status to a sanitized, operator-actionable DnsProviderError.
+
+        These strings land in acme_order_events and letsencrypt_orders.error_detail and are shown
+        in the order timeline, so each one names what to fix. GoDaddy's own `code`/`message` is
+        appended when present because the two 403 causes — account not eligible for the DNS API vs.
+        a PAT missing `domains.dns:update` — are indistinguishable by status alone. Scrubbing
+        happens HERE, at the single point where provider-supplied text enters a message, so a new
+        caller cannot forget it.
+        """
+        code = _scrub(code, self._api_key, self._api_secret)
+        message = _scrub(message, self._api_key, self._api_secret)
+        if status == 401:
+            detail = ("GoDaddy rejected the API credentials. Check they are a PRODUCTION Key/Secret pair "
+                      "from developer.godaddy.com/keys — the first key the dashboard issues is an OTE "
+                      "(test) key and is not valid here.")
+        elif status == 403:
+            detail = ("GoDaddy denied access to the DNS API. The account needs at least one registered "
+                      "domain, and a Personal Access Token needs the domains.domain:read and "
+                      "domains.dns:update scopes.")
+        elif status == 404:
+            detail = ("GoDaddy has no zone for this domain (check it is registered in this account and "
+                      "uses GoDaddy nameservers).")
+        elif status == 409:
+            detail = "GoDaddy reports this domain is not eligible to have its DNS records changed."
+        elif status == 422:
+            detail = "GoDaddy rejected the record change as invalid (HTTP 422)."
+        elif status == 429:
+            detail = f"GoDaddy rate limit reached; retry in ~{retry_after or 60}s."
+        else:
+            detail = f"GoDaddy API error (HTTP {status})."
+        if code or message:
+            detail += f" (GoDaddy: {code}{': ' + message if message else ''})"
+        return _GoDaddyHTTPError(detail, status=status, code=code)
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> Any:
+        """One GoDaddy API call. Returns the parsed JSON body, or None for the empty-bodied writes.
+
+        Raises a SANITIZED _GoDaddyHTTPError / DnsProviderError — never the credentials, never the
+        request, never a response body verbatim.
+        """
+        url = f"{GODADDY_API_BASE}{path}"
+        try:
+            async with session.request(
+                method, url, headers=self._headers(), allow_redirects=False, **kwargs
+            ) as resp:
+                try:
+                    # content_type=None: every GoDaddy write answers 200/204 with an EMPTY body, and
+                    # aiohttp would otherwise raise on the missing/other content type before parsing.
+                    body = await resp.json(content_type=None)
+                except ValueError:
+                    # ONLY a decode failure (JSONDecodeError subclasses ValueError) is swallowed —
+                    # an empty write body, or an HTML error page on a >=400. A transport failure
+                    # mid-read (ClientPayloadError, TimeoutError) must NOT land here: it would look
+                    # identical to "empty body", and a caller that reads an RRset would then see
+                    # None and could mistake it for an empty RRset. Those propagate to the handlers
+                    # below and become a real DnsProviderError.
+                    body = None
+                # 2xx only. Redirects are deliberately not followed (aiohttp would forward the
+                # Authorization header), so a 3xx is a failed call — treating `< 400` as success
+                # would report a redirected write as a silent no-op.
+                if 200 <= resp.status < 300:
+                    return body
+                code, message = _error_fields(body)
+                retry_after = _retry_after_seconds(resp.headers, body) if resp.status == 429 else None
+                raise self._http_error(resp.status, code, message, retry_after)
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Only the exception TYPE is interpolated: an aiohttp client error's str() can carry the
+            # request URL, and the message is persisted to the order timeline.
+            raise DnsProviderError(f"Could not reach the GoDaddy API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected GoDaddy API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._api_key:
+            return {"ok": False, "detail": "No GoDaddy API Key provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                # Cheapest read-only check: one request, no zone needed. Deliberately NOT
+                # GET /v1/domains/{domain} — GoDaddy has rejected that details call for small
+                # accounts since 2024-05 while record-level calls keep working, so verifying with it
+                # produces false negatives on accounts where DNS-01 would succeed.
+                body = await self._request(session, "GET", "/domains?limit=1")
+            if not isinstance(body, list):
+                return {"ok": False, "detail": "GoDaddy returned an unexpected response to the credential check."}
+            if not body:
+                # An empty list is NOT a failure: sub-zones delegated to GoDaddy nameservers are
+                # manageable via the records API but never appear in the domain listing.
+                return {"ok": True, "detail": ("GoDaddy credentials valid, but no domains are visible in this "
+                                               "account — the domain you validate must be registered here, or "
+                                               "be a zone delegated to GoDaddy nameservers.")}
+            return {"ok": True, "detail": "GoDaddy credentials valid."}
+        except DnsProviderError as exc:
+            detail = str(exc)
+            if not self._api_secret:
+                # The Bearer path is silent otherwise, and a half-filled form is the likeliest cause.
+                detail += (" Note: no API Secret was entered, so the API Key was sent as a Personal Access "
+                           "Token (Bearer). If you have a Key + Secret pair, enter both halves.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the GoDaddy credentials."}
+
+    async def _resolve_domain(self, session: aiohttp.ClientSession, record_name: str) -> str:
+        """Find the most-specific (longest-suffix) GoDaddy-managed zone for an absolute record name.
+
+        GoDaddy has no `/zones?name=` equivalent, so this walks suffixes longest-to-shortest and
+        probes `GET /v1/domains/{candidate}/records/NS`. That probe (rather than the domain listing
+        or the domain-details call) is deliberate: it finds sub-zones delegated to GoDaddy
+        nameservers, which never appear in `GET /v1/domains` at all, and it does not depend on the
+        details endpoint that small accounts are rejected from.
+        """
+        cached = self._zone_cache.get(record_name)
+        if cached:
+            return cached
+        labels = record_name.rstrip(".").lower().split(".")
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break  # a zone needs at least two labels
+            try:
+                body = await self._request(
+                    session, "GET", f"/domains/{quote(candidate, safe='')}/records/NS"
+                )
+            except _GoDaddyHTTPError as exc:
+                if exc.status in (404, 422):
+                    continue  # not a zone in this account — keep walking
+                # 401/403/409/429/5xx are credential, eligibility or platform failures, not
+                # "wrong zone". Continuing would burn the rate-limit budget re-failing on every
+                # remaining suffix and would bury the real cause under "no managed domain".
+                raise
+            if isinstance(body, list) and body:
+                self._zone_cache[record_name] = candidate
+                return candidate
+        raise DnsProviderError(f"No managed GoDaddy domain found for {record_name}.")
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                zone = await self._resolve_domain(session, name)
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status != 404:
+                        raise
+                    # Some accounts 404 reading back a record set in a zone whose WRITES succeed
+                    # (acme.sh #6517). Reachable only when the NS probe resolved the zone but the
+                    # TXT read 404s — if the NS probe itself 404s we never get here and the caller
+                    # sees "No managed GoDaddy domain found", which is the honest answer. We cannot
+                    # merge what we cannot read, and a single-value PUT would destroy any coexisting
+                    # sibling, so PATCH is the only correct recovery: it is the one genuinely
+                    # ADDITIVE primitive in v1 ("Appends DNS records ... Existing records with the
+                    # same type and name are preserved"). It cannot dedupe, but a duplicate
+                    # identical TXT is harmless for validation and cleanup removes the whole RRset.
+                    await self._request(
+                        session, "PATCH", f"/domains/{quote(zone, safe='')}/records",
+                        json=[{"type": "TXT", "name": _relative_name(name, zone),
+                               "data": value, "ttl": _TXT_TTL}],
+                    )
+                    return
+                body = _merge_add(_require_rrset(existing), value)
+                if body is None:
+                    return  # already published — idempotent, this is where ACME retries land
+                await self._request(session, "PUT", path, json=body)
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with _rrset_lock(name):
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                try:
+                    zone = await self._resolve_domain(session, name)
+                except _GoDaddyHTTPError as exc:
+                    # Raise only what a later sweep could plausibly succeed at. reconcile_dns01_cleanup
+                    # swallows the error and leaves dns_record_cleaned FALSE, so the row is re-selected
+                    # every cycle — and its query takes a bare LIMIT 50, so rows that can NEVER succeed
+                    # (revoked key, account lost DNS-API eligibility) would monopolise the whole
+                    # cleanup budget and starve every other account. For those terminal statuses we
+                    # give up quietly: the orphaned `_acme-challenge` TXT is inert, and the same
+                    # credential failure is already loud on the publish path, where it is actionable.
+                    if exc.status == 429 or exc.status >= 500:
+                        raise
+                    return
+                except DnsProviderError:
+                    return  # zone genuinely not resolvable — nothing we could clean up
+                path = _rrset_path(zone, _relative_name(name, zone))
+                try:
+                    existing = await self._request(session, "GET", path)
+                except _GoDaddyHTTPError as exc:
+                    if exc.status == 404:
+                        return  # RRset (or the read) is gone — tolerate
+                    raise
+                body = _merge_remove(_require_rrset(existing), value)
+                if body is None:
+                    return  # our value is not there — already gone, tolerate
+                if not body:
+                    # The LAST value at this name. `PUT []` is rejected (422 INVALID_BODY, "Records
+                    # must be specified"), so emptying an RRset REQUIRES DELETE. This removes only
+                    # TXT at this exact name; other names and other record types are preserved.
+                    # Do NOT fall back to the "write an empty string to delete" folklore — that hack
+                    # is what creates the tombstone rows _live_values has to filter.
+                    try:
+                        await self._request(session, "DELETE", path)
+                    except _GoDaddyHTTPError as exc:
+                        if exc.status == 404:
+                            return  # raced with another cleanup — tolerate
+                        raise
+                    return
+                await self._request(session, "PUT", path, json=body)

backend/services/dns_providers/manual.py

@@ -0,0 +1,31 @@
+"""Manual DNS provider for ACME DNS-01 (Issue #35).
+
+The user publishes the `_acme-challenge` TXT record in their own DNS (any provider, including
+fully internal/isolated DNS that no API can reach) and then confirms via the UI. There is no API
+to call, so add/remove are no-ops and the orchestration waits for an explicit `dns-confirm`.
+CNAME delegation works implicitly here: the CA follows a CNAME, so a user who delegates
+`_acme-challenge` elsewhere just publishes the value there and confirms.
+"""
+from __future__ import annotations
+
+from typing import Dict, List
+
+from .base import DnsProvider
+
+
+class ManualDNSProvider(DnsProvider):
+    name = "manual"
+    label = "Manual (publish the TXT record yourself)"
+    automated = False
+    credential_fields: List[Dict] = []  # no credentials needed
+
+    async def verify_credentials(self) -> Dict:
+        return {"ok": True, "detail": "Manual mode needs no credentials. You will publish the TXT record yourself."}
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user publishes the record and confirms via the UI.
+        return None
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user may remove the record manually after issuance.
+        return None

backend/services/dns_providers/registry.py

@@ -0,0 +1,46 @@
+"""DNS provider registry for ACME DNS-01 (Issue #35).
+
+Single source of truth mapping a provider name -> class. The API serves the credential-field
+schema from here (so the UI has no hardcoded provider fields) and validates inbound provider
+names against this allow-list. Adding a provider = add it here; nothing else changes.
+"""
+from __future__ import annotations
+
+from typing import Dict, List, Type
+
+from .base import DnsProvider
+from .cloudflare import CloudflareDNSProvider
+from .godaddy import GoDaddyDNSProvider
+from .manual import ManualDNSProvider
+
+_PROVIDERS: Dict[str, Type[DnsProvider]] = {
+    ManualDNSProvider.name: ManualDNSProvider,
+    CloudflareDNSProvider.name: CloudflareDNSProvider,
+    GoDaddyDNSProvider.name: GoDaddyDNSProvider,
+}
+
+
+def is_supported(name: str) -> bool:
+    return name in _PROVIDERS
+
+
+def get_provider(name: str, credentials: Dict[str, str] | None = None) -> DnsProvider:
+    cls = _PROVIDERS.get(name)
+    if cls is None:
+        raise ValueError(f"Unsupported DNS provider: {name}")
+    return cls(credentials or {})
+
+
+def list_providers() -> List[Dict]:
+    """Return the UI-facing provider catalog: name, label, automated flag, and credential schema."""
+    out: List[Dict] = []
+    for name, cls in _PROVIDERS.items():
+        out.append(
+            {
+                "name": cls.name,
+                "label": cls.label,
+                "automated": cls.automated,
+                "credential_fields": cls.credential_fields,
+            }
+        )
+    return out

backend/services/frontend_service.py

@@ -56,14 +56,14 @@ async def create_frontend_row(
             acl_rules, redirect_rules, use_backend_rules,
             request_headers, response_headers, options, tcp_request_rules, timeout_client, timeout_http_request,
             rate_limit, compression, log_separate, monitor_uri,
-            cluster_id, maxconn, updated_at
+            cluster_id, maxconn, log_format, filters, updated_at
         ) VALUES (
             $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12,
             $13, $14, $15, $16, $17, $18, $19,
             $20, $21, $22,
             $23, $24, $25, $26, $27, $28,
             $29, $30, $31, $32,
-            $33, $34, CURRENT_TIMESTAMP
+            $33, $34, $35, $36, CURRENT_TIMESTAMP
         )
         RETURNING id
         """,
@@ -101,6 +101,8 @@ async def create_frontend_row(
         getattr(payload, "monitor_uri", None),
         cluster_id,
         getattr(payload, "maxconn", None),
+        getattr(payload, "log_format", None),  # Issue #38
+        getattr(payload, "filters", None),     # Issue #38
     )
 
     if mark_pending:

backend/services/haproxy_config.py

@@ -6,9 +6,98 @@ import json
 import urllib.parse
 from typing import Optional, List, Dict, Any
 from database.connection import get_database_connection, close_database_connection
+from utils.acme_backend_url import resolve_acme_backend_target
 
 logger = logging.getLogger(__name__)
 
+# Sentinel prefixes returned by `generate_haproxy_config_for_cluster` INSTEAD of a
+# configuration when generation fails. They are plain strings (not exceptions) for
+# historical reasons: the function's outer `except` swallows everything and returns
+# a one-line comment as the "config".
+#
+# That is a data-loss primitive on its own: an exception anywhere in the generator —
+# e.g. `urllib.parse.urlparse('http://host:99999').port` raising ValueError for an
+# out-of-range port in `acme_backend_url` — collapses a whole cluster's haproxy.cfg
+# into a single comment line, which the apply path then stores as APPLIED and pushes
+# to every agent. Callers that PERSIST the returned text MUST reject it first; use
+# `is_config_generation_error()` rather than matching the string by hand.
+CONFIG_GENERATION_ERROR_PREFIXES = (
+    "# Error generating configuration:",
+    "# Error: Cluster not found",
+)
+
+
+def is_config_generation_error(config_content: Optional[str]) -> bool:
+    """True when `config_content` is a generator failure sentinel, not a configuration.
+
+    Persisting or shipping a sentinel silently destroys a cluster's configuration, so
+    every call site that writes the generator's output to `config_versions` (or hands
+    it to an agent) must guard with this.
+    """
+    if not config_content:
+        return True
+    return config_content.lstrip().startswith(CONFIG_GENERATION_ERROR_PREFIXES)
+
+
+def select_acme_backend_source(candidates: List[tuple]):
+    """Pick the first candidate that resolves into a usable address.
+
+    `candidates` is an ordered list of ``(source_name, url)`` from most to least
+    specific. Returns ``(source_name, url, target, skipped)`` where ``skipped`` lists
+    the ``(source_name, url, target)`` of candidates that were rejected.
+
+    Falling through on UNUSABLE values, not just empty ones, is the point. Values
+    predating validation are common — the settings field was free text — and a
+    scheme-less ``10.90.1.4:8080`` cannot be resolved. Stopping at the first non-empty
+    candidate would emit a backend section with no ``server`` line: ``haproxy -c``
+    still passes because the section exists, Apply succeeds, and every challenge
+    request then 503s from an empty backend with nothing to show for it.
+    """
+    skipped = []
+    for source, url in candidates:
+        if not url:
+            continue
+        target = resolve_acme_backend_target(url)
+        if target.error_code:
+            skipped.append((source, url, target))
+            continue
+        return source, url, target, skipped
+
+    # Nothing usable. Report against the last non-empty candidate so the rendered
+    # comment and the log name a concrete value rather than an empty one.
+    if skipped:
+        source, url, target = skipped[-1]
+        return source, url, target, skipped[:-1]
+    source, url = candidates[0] if candidates else ('none', '')
+    return source, url, resolve_acme_backend_target(url), skipped
+
+
+def extract_acme_backend_target(config_content: Optional[str]) -> Optional[str]:
+    """Return the `server _acme_mgmt` argument string from a rendered config.
+
+    e.g. ``"10.90.1.4:80"`` or ``"mgmt.internal:443 ssl verify none"``; ``None`` when
+    the cluster renders no ACME challenge backend at all.
+
+    Used to decide whether an ACME-related edit actually CHANGES the shipped
+    configuration. Comparing whole config texts would report a difference on every
+    unrelated pending edit; comparing this one line answers the only question that
+    matters here — "would the HAProxy nodes start talking to a different address?"
+    """
+    if not config_content:
+        return None
+    in_section = False
+    for line in config_content.splitlines():
+        stripped = line.strip()
+        if stripped.startswith("backend "):
+            in_section = stripped == "backend _acme_challenge_backend"
+            continue
+        if stripped.startswith(("frontend ", "listen ", "defaults", "global")):
+            in_section = False
+            continue
+        if in_section and stripped.startswith("server _acme_mgmt "):
+            return stripped[len("server _acme_mgmt "):].strip()
+    return None
+
 
 def _format_redirect_rule(rule: Any) -> Optional[str]:
     """Render a single redirect rule into a HAProxy `redirect ...` line.
@@ -393,6 +482,12 @@ def _categorize_haproxy_directive(line: str) -> str:
         return "prelude"
     if s.startswith("acl "):
         return "acl"
+    # Issue #38: SPOE (and other) `filter` directives must be declared BEFORE
+    # the `http-request send-spoe-group` rules that use them, otherwise HAProxy
+    # fails with "unable to find SPOE engine". Own bucket, flushed right after
+    # `prelude` and before tcp_req/acl/http_req (see flush order below).
+    if s.startswith("filter "):
+        return "filter"
     if s.startswith("stick-table") or s.startswith("stick "):
         return "stick"
     if s.startswith("tcp-request"):
@@ -414,6 +509,7 @@ def _categorize_haproxy_directive(line: str) -> str:
         or s.startswith("compression ")
         or s.startswith("monitor-uri")
         or s.startswith("log ")
+        or s.startswith("log-format")  # Issue #38: log-format / log-format-sd
         or s.startswith("description ")
         or s.startswith("disabled")
         or s.startswith("enabled")
@@ -871,7 +967,15 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                         f"    bind {frontend['bind_address']}:{frontend['bind_port']}"
                     )
             
-            config_lines.append(f"    mode {frontend['mode']}")
+            # `frontends.mode` is `VARCHAR(10) DEFAULT 'http'` but NULLABLE, and rows
+            # can arrive with it unset via agent sync or config import. Interpolating
+            # the raw value then emits a literal `mode None`, which HAProxy rejects —
+            # taking down the whole cluster config, not just this frontend. Normalise
+            # once here and use the result everywhere below, so the ACME gate, the
+            # backend-mode check and the rendered line can never disagree with each
+            # other about what mode this frontend is in.
+            frontend_mode = (frontend.get('mode') or 'http').strip().lower()
+            config_lines.append(f"    mode {frontend_mode}")
 
             # ─────────────────────────────────────────────────────────────────
             # R2.3 / R3.3 (PR-1 hotfix): emit ordering buckets.
@@ -903,11 +1007,18 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # "stick-table already declared").
             # ─────────────────────────────────────────────────────────────────
             _fe_buckets: Dict[str, List[str]] = {
-                "prelude": [], "stick": [], "tcp_req": [],
+                "prelude": [], "filter": [], "stick": [], "tcp_req": [],
                 "acl": [], "http_req": [], "http_resp": [],
                 "redirect": [], "use_be": [], "default_be": [],
             }
             _stick_table_emitted = False
+            # Whether this frontend emits ANY stick-counter usage (`track-sc<N>` or an
+            # `sc_*_rate(...)` fetch). If it does but no `stick-table` is declared, HAProxy
+            # fatally rejects the WHOLE cluster config with "table '<frontend>' used but not
+            # configured". This happens with rate-limit directives baked into a frontend's
+            # stored fields (request_headers/options) by an older version or a config import.
+            # We track it here and inject a default stick-table before flushing if needed.
+            _sc_counter_used = False
             # Phase K Phase D follow-up (Bulgu #13) — same dedup
             # contract for `http-request track-sc<N> <fetch>` lines.
             # HAProxy only NEEDS one tracking call per
@@ -931,9 +1042,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 correct frontend-block bucket. Idempotent for stick-table
                 lines (R3.3 dedup) AND http-request track-sc<N> lines
                 (Bulgu #13 dedup)."""
-                nonlocal _stick_table_emitted
+                nonlocal _stick_table_emitted, _sc_counter_used
                 cat = _categorize_haproxy_directive(line)
                 stripped = line.strip()
+                # Any stick-counter usage (track-sc<N> write, or an sc_*_rate(...) fetch like
+                # sc_http_req_rate(0)) requires a stick-table in this frontend.
+                if "track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped):
+                    _sc_counter_used = True
                 if cat == "stick":
                     if _stick_table_emitted and stripped.startswith("stick-table"):
                         logger.debug(
@@ -972,7 +1087,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 _fe_buckets[cat].append(line)
 
             # ACME HTTP-01 Challenge routing (auto-managed)
-            if frontend['mode'] == 'http' and cluster_info.get('acme_enabled', False):
+            if frontend_mode == 'http' and cluster_info.get('acme_enabled', False):
                 _emit_fe("    acl is_acme_challenge path_beg /.well-known/acme-challenge/")
                 _emit_fe("    http-request allow if is_acme_challenge")
                 _emit_fe("    use_backend _acme_challenge_backend if is_acme_challenge")
@@ -985,20 +1100,31 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
                         _emit_fe(f"    {line_stripped}")
 
+            # Issue #38: emit frontend log-format and SPOE (etc.) filter directives.
+            # `log_format` routes to the `prelude` bucket, `filters` to the `filter`
+            # bucket (both via _emit_fe → _categorize_haproxy_directive), guaranteeing
+            # `filter ...` is rendered before the `http-request send-spoe-group` rules.
+            for _fld in ('log_format', 'filters'):
+                if frontend.get(_fld):
+                    for line in frontend[_fld].split('\n'):
+                        line_stripped = line.strip()
+                        if line_stripped and line_stripped not in ('[]', '{}', 'null', 'None'):
+                            _emit_fe(f"    {line_stripped}")
+
             # CRITICAL: Validate frontend-backend mode compatibility
             if frontend.get('default_backend'):
                 default_backend_name = frontend['default_backend'].strip() if frontend['default_backend'] else ''
                 if default_backend_name and default_backend_name not in ('[]', '{}', 'null', 'None'):
                     backend_mode = backend_modes.get(default_backend_name)
 
-                    if backend_mode and backend_mode != frontend['mode']:
-                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend['mode']}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
+                    if backend_mode and backend_mode != frontend_mode:
+                        logger.error(f"CONFIG ERROR: Frontend '{frontend['name']}' mode '{frontend_mode}' does not match backend '{default_backend_name}' mode '{backend_mode}'")
                         # FIX-10 marker: 'BACKEND-MODE-WARNING' keyword in
                         # the comment body routes it to the 'default_be'
                         # bucket via _categorize_haproxy_directive, so the
                         # warning emits next to the actual default_backend
                         # directive instead of at the top of the block.
-                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend['mode']}'")
+                        _emit_fe(f"    # BACKEND-MODE-WARNING: Backend '{default_backend_name}' has mode '{backend_mode}' but frontend has mode '{frontend_mode}'")
                         _emit_fe(f"    # BACKEND-MODE-WARNING: HAProxy will reject this configuration! Please fix the mode mismatch in UI.")
 
                     _emit_fe(f"    default_backend {default_backend_name}")
@@ -1188,6 +1314,22 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 else:
                     logger.warning(f"Config Generation: No config lines generated for WAF rule '{waf_rule['name']}' (ID: {waf_rule['id']}, Type: {waf_rule['rule_type']})")
 
+            # Robustness fix: if this frontend uses a stick counter (track-sc<N> or an
+            # sc_*_rate(...) fetch) but declared NO stick-table, inject a default one so HAProxy
+            # doesn't fatally reject the whole cluster config with "table '<frontend>' used but
+            # not configured". This rescues rate-limit directives baked into a frontend's stored
+            # request_headers/options by an older version or import. Purely additive — it only
+            # fires when a counter is used AND no table exists (a config that is invalid today),
+            # so it never changes a frontend that already has a stick-table or doesn't rate-limit.
+            if _sc_counter_used and not _stick_table_emitted:
+                _fe_buckets["stick"].insert(
+                    0, "    stick-table type ip size 100k expire 30s store http_req_rate(10s)")
+                _stick_table_emitted = True
+                logger.info(
+                    f"STICK-TABLE AUTO-INJECT: frontend '{frontend['name']}' uses a stick "
+                    f"counter (track-sc/sc_*_rate) but declared no stick-table; injected a "
+                    f"default so the config stays valid.")
+
             # ─────────────────────────────────────────────────────────────
             # Flush the per-frontend buckets in canonical HAProxy order.
             # The order below is the single source of truth for emit
@@ -1196,6 +1338,7 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
             # ─────────────────────────────────────────────────────────────
             for _bucket_key in (
                 "prelude",
+                "filter",
                 "stick",
                 "tcp_req",
                 "acl",
@@ -1543,36 +1686,82 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                     f"frontend found (would create orphan backend section)."
                 )
             else:
-                acme_url = cluster_info.get('acme_backend_url') or ''
-                if not acme_url:
-                    try:
-                        acme_settings = await db_conn.fetchrow(
-                            "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
-                        )
-                        if acme_settings and acme_settings['value']:
-                            val = acme_settings['value']
-                            if isinstance(val, str):
-                                try:
-                                    val = json.loads(val)
-                                except (json.JSONDecodeError, TypeError):
-                                    pass
-                            if val:
-                                acme_url = str(val)
-                    except Exception:
-                        pass
-                if not acme_url:
-                    from config import MANAGEMENT_BASE_URL
-                    acme_url = MANAGEMENT_BASE_URL
+                # Track WHERE the effective URL came from. Support has no way today to
+                # tell an operator-set value from the shipped `localhost` default, and
+                # this whole block emits no log line at all (contrast the skip branches
+                # above), so a wrong challenge backend is invisible until Let's Encrypt
+                # fails. `acme_source` is logged with the rendered host:port below.
+                acme_candidates = [
+                    ('cluster.acme_backend_url', cluster_info.get('acme_backend_url') or '')
+                ]
+                _settings_url = ''
+                try:
+                    acme_settings = await db_conn.fetchrow(
+                        "SELECT value FROM system_settings WHERE key = 'acme.challenge_backend_url'"
+                    )
+                    if acme_settings and acme_settings['value']:
+                        val = acme_settings['value']
+                        if isinstance(val, str):
+                            try:
+                                val = json.loads(val)
+                            except (json.JSONDecodeError, TypeError):
+                                pass
+                        if val:
+                            _settings_url = str(val)
+                except Exception:
+                    pass
+                acme_candidates.append(
+                    ('system_settings.acme.challenge_backend_url', _settings_url)
+                )
+                from config import MANAGEMENT_BASE_URL
+                acme_candidates.append(('config.MANAGEMENT_BASE_URL', MANAGEMENT_BASE_URL))
 
-                parsed = urllib.parse.urlparse(acme_url)
-                host = parsed.hostname or 'localhost'
-                port = parsed.port or (443 if parsed.scheme == 'https' else 8080)
-                ssl_flag = ' ssl verify none' if parsed.scheme == 'https' else ''
+                acme_source, acme_url, target, _skipped = select_acme_backend_source(
+                    acme_candidates
+                )
+                for _s_source, _s_url, _s_target in _skipped:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} skipping unusable value from "
+                        f"{_s_source} (reason={_s_target.error_code}): "
+                        f"{_s_target.error_message} value={_s_url!r}"
+                    )
 
                 config_lines.append("# ACME Challenge Backend (auto-managed by HAProxy OpenManager)")
                 config_lines.append("backend _acme_challenge_backend")
                 config_lines.append("    mode http")
-                config_lines.append(f"    server _acme_mgmt {host}:{port}{ssl_flag}")
+                if target.error_code:
+                    # The value cannot produce an address. Emit the section without a
+                    # `server` line rather than guessing: every HTTP frontend already
+                    # carries `use_backend _acme_challenge_backend`, and a use_backend
+                    # with no matching backend is fatal to `haproxy -c`. The operator's
+                    # raw value is deliberately NOT echoed into the file — a value
+                    # containing a newline would inject directives into a config pushed
+                    # to every node. It goes to the log instead.
+                    config_lines.append(
+                        f"    # ACME challenge backend unavailable ({target.error_code}) — "
+                        f"see Cluster Management > ACME Challenge Backend URL"
+                    )
+                    logger.error(
+                        f"ACME-BACKEND: cluster {cluster_id} has an unusable challenge backend "
+                        f"URL (source={acme_source}, reason={target.error_code}): "
+                        f"{target.error_message} value={acme_url!r}"
+                    )
+                else:
+                    config_lines.append(
+                        f"    server _acme_mgmt {target.host}:{target.port}{target.ssl_flag}"
+                    )
+                    # One greppable line per render. `ACME-BACKEND` is the support
+                    # keyword: it answers "what address did we actually ship, and who
+                    # chose it?" without shell access to the node.
+                    logger.info(
+                        f"ACME-BACKEND: cluster {cluster_id} challenge backend rendered as "
+                        f"{target.host}:{target.port}{target.ssl_flag} "
+                        f"(source={acme_source}, url={acme_url!r})"
+                    )
+                for _warning in target.warnings:
+                    logger.warning(
+                        f"ACME-BACKEND: cluster {cluster_id} (source={acme_source}): {_warning}"
+                    )
                 config_lines.append("")
 
         # Only close the connection if it was created within this function

backend/services/keepalived_config.py

@@ -0,0 +1,192 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Standalone, DB-free renderer for a node's /etc/keepalived/keepalived.conf and the
+HAProxy health-check script, plus Fernet at-rest encryption for the VRRP secret.
+
+The router fetches DB rows and calls these pure functions; nothing here touches the
+database or logs secrets. Trivially unit-testable (see tests/test_keepalived_config.py).
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import re
+from typing import List, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# Written into every file we manage so the agent can tell "ours" from a
+# hand-maintained keepalived setup (ownership guard, B-3/T-2). Must match the
+# string the agent greps for in linux_install.sh.
+OWNERSHIP_MARKER = "# Managed by HAProxy OpenManager"
+CHECK_SCRIPT_PATH = "/etc/keepalived/check_haproxy.sh"
+
+
+# ---------------------------------------------------------------------------
+# VRRP secret at rest (mirrors backend/services/mfa_service.py)
+# ---------------------------------------------------------------------------
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit VIP_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so the secret survives restarts, like MFA)."""
+    explicit = os.getenv("VIP_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("VIP_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"vip-vrrp-secret-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_vrrp_secret(secret_plain: str) -> str:
+    return _get_fernet().encrypt(secret_plain.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_vrrp_secret(secret_encrypted: str) -> Optional[str]:
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt VRRP secret (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting VRRP secret: %s", exc)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Renderers
+# ---------------------------------------------------------------------------
+def build_haproxy_check_script(*, bin_path: Optional[str] = None,
+                               config_path: Optional[str] = None) -> str:
+    """Render the health-check the agent writes to CHECK_SCRIPT_PATH.
+
+    Derives the process name from the HAProxy binary basename (B-4) rather than a
+    blind hardcoded 'haproxy'. Returns non-zero when HAProxy isn't running so the
+    VRRP track_script lowers this node's priority and the VIP fails over.
+    """
+    proc = "haproxy"
+    if bin_path:
+        base = os.path.basename(bin_path.strip())
+        if re.match(r'^[A-Za-z0-9._-]{1,64}$', base):
+            proc = base
+    return (
+        "#!/bin/sh\n"
+        f"{OWNERSHIP_MARKER} — DO NOT EDIT\n"
+        "# Exits 0 while HAProxy is up; non-zero triggers VRRP failover.\n"
+        f"pidof {proc} >/dev/null 2>&1 || exit 1\n"
+        "exit 0\n"
+    )
+
+
+def _vrrp_instance_name(vip_id: int) -> str:
+    return f"VI_{int(vip_id)}"
+
+
+def _failover_weight(members: List[dict]) -> int:
+    """Negative weight so a failed MASTER drops strictly below every healthy BACKUP
+    (B-6). master_priority + weight < min(backup_priority)."""
+    master = next((m for m in members if str(m.get("role", "")).upper() == "MASTER"), None)
+    backups = [int(m["priority"]) for m in members if str(m.get("role", "")).upper() != "MASTER"]
+    if not master or not backups:
+        return -20
+    return -((int(master["priority"]) - min(backups)) + 1)
+
+
+def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
+                           peer_ips: List[str], auth_pass_plain: Optional[str]) -> str:
+    """Render one node's keepalived.conf from the VIP + member rows.
+
+    `vip` keys: id, name, virtual_ip, prefix_length, virtual_router_id, advert_int,
+                use_unicast, track_haproxy.
+    `this_agent` keys: role, priority, network_interface, ip_address (str).
+    `peer_ips`: the OTHER members' ip_address strings (already str()'d by the caller).
+    Caller must never log the returned string (it may contain auth_pass).
+    """
+    role = str(this_agent["role"]).upper()
+    iface = this_agent["network_interface"]
+    prio = int(this_agent["priority"])
+    track = bool(vip.get("track_haproxy", True))
+    use_unicast = bool(vip.get("use_unicast", True))
+    vrid = int(vip["virtual_router_id"])
+    advert = int(vip.get("advert_int", 1))
+    name = str(vip.get("name", ""))
+    inst = _vrrp_instance_name(vip["id"])
+
+    lines: List[str] = []
+    lines.append(f"{OWNERSHIP_MARKER} — DO NOT EDIT")
+    lines.append(f'# VIP "{name}" (id={vip["id"]}) — role {role}')
+    lines.append("global_defs {")
+    lines.append("    enable_script_security")
+    lines.append("    script_user root")
+    lines.append("}")
+    lines.append("")
+
+    if track:
+        weight = _failover_weight(members)
+        lines.append("vrrp_script chk_haproxy {")
+        lines.append(f'    script "{CHECK_SCRIPT_PATH}"')
+        lines.append("    interval 2")
+        lines.append("    fall 2")
+        lines.append("    rise 2")
+        lines.append(f"    weight {weight}")
+        lines.append("}")
+        lines.append("")
+
+    lines.append(f"vrrp_instance {inst} {{")
+    lines.append(f"    state {role}")
+    lines.append(f"    interface {iface}")
+    lines.append(f"    virtual_router_id {vrid}")
+    lines.append(f"    priority {prio}")
+    lines.append(f"    advert_int {advert}")
+    if auth_pass_plain:
+        lines.append("    authentication {")
+        lines.append("        auth_type PASS")
+        lines.append(f"        auth_pass {auth_pass_plain}")
+        lines.append("    }")
+    # Unicast only makes sense with at least one peer. For a single-node VIP (no peers)
+    # we deliberately omit the unicast block: keepalived treats a bare `unicast_src_ip`
+    # with no `unicast_peer` as deprecated, warns, and silently falls back to multicast —
+    # and `keepalived -t` flags it. Omitting it yields a clean multicast config that holds
+    # the VIP with no peer to talk to. Multi-node behaviour (peers present) is unchanged.
+    if use_unicast and peer_ips:
+        src = this_agent.get("ip_address")
+        if src:
+            lines.append(f"    unicast_src_ip {src}")
+        lines.append("    unicast_peer {")
+        for p in peer_ips:
+            lines.append(f"        {p}")
+        lines.append("    }")
+    lines.append("    virtual_ipaddress {")
+    lines.append(f'        {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
+    lines.append("    }")
+    if track:
+        lines.append("    track_script {")
+        lines.append("        chk_haproxy")
+        lines.append("    }")
+    lines.append("}")
+    return "\n".join(lines) + "\n"

backend/services/keepalived_parser.py

@@ -0,0 +1,533 @@
+"""Issue #27 follow-up — parse an EXISTING keepalived.conf so a hand-maintained VIP can be
+adopted into OpenManager's model (v1.10.4).
+
+Standalone and DB-free, like keepalived_config.py: the agent reports the file it found on a
+node, this module turns it into the fields `vip_instances` / `vip_members` need, and the
+adoption endpoint decides whether taking ownership is safe.
+
+WHY A PARSER AND NOT THE HEARTBEAT. The heartbeat carries two keepalived facts —
+`keepalive_state` (MASTER/BACKUP, best-effort from logs) and `keepalive_ip` (the first
+address grepped out of `virtual_ipaddress`). Rendering a node's config needs eleven:
+virtual_router_id, auth_pass, interface, priority, prefix_length, advert_int, unicast
+peers, track_haproxy, role and the address itself. Guessing the missing ones is not a
+cosmetic risk — a wrong VRID puts the nodes in two separate VRRP domains and a wrong
+auth_pass makes them reject each other, and either way both nodes claim the VIP.
+
+THE SAFETY CONTRACT. Adoption REPLACES the operator's file with our render, so anything in
+their file that `render_keepalived_conf` cannot reproduce would be silently destroyed on
+takeover — a `notify_master` failover hook, an LVS `virtual_server` section, a second
+address in one instance, a sync group. Extracting the fields is the easy half; the half
+that matters is `unsupported`, the list of directives we would drop. The caller must treat
+a non-empty `unsupported` as a refusal to adopt, not a warning to log.
+
+Secrets: a parsed instance carries `auth_pass` in cleartext because that is the only way to
+re-render an identical config. NEVER log a parse result. Callers persist it through
+`encrypt_vrrp_secret` and mask it in anything UI-facing, exactly as the VIP router already
+does for `auth_pass` in version diffs.
+"""
+from __future__ import annotations
+
+import ipaddress
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+# Directives `render_keepalived_conf` emits, and therefore the only ones a takeover can
+# reproduce. Anything else found in a vrrp_instance is reported in `unsupported`.
+_SUPPORTED_INSTANCE_KEYS = {
+    "state", "interface", "virtual_router_id", "priority", "advert_int",
+    "authentication", "unicast_src_ip", "unicast_peer", "virtual_ipaddress", "track_script",
+}
+# Top-level blocks we can account for. `vrrp_script` is reproduced only when it is the
+# check script we generate ourselves (see _classify_script).
+_SUPPORTED_TOP_KEYS = {"global_defs", "vrrp_script", "vrrp_instance"}
+
+# global_defs entries our render emits. An operator's file usually carries more (notification
+# email, router_id, ...) and losing those is a real change, so they are reported too.
+_SUPPORTED_GLOBAL_KEYS = {"enable_script_security", "script_user"}
+
+_IDENT_RE = re.compile(r"^[A-Za-z0-9._:-]+$")
+
+
+class KeepalivedParseError(ValueError):
+    """The text is not a keepalived.conf we can reason about (unbalanced braces etc.)."""
+
+
+# ---------------------------------------------------------------------------
+# Tokenizer / block reader
+# ---------------------------------------------------------------------------
+def _strip_comment(line: str) -> str:
+    """Drop a trailing comment. keepalived treats BOTH `#` and `!` as comment starters, and
+    neither is meaningful inside the quoted script paths we care about, so a quote-aware
+    scan is enough (a `#` inside quotes stays)."""
+    out: List[str] = []
+    quote: Optional[str] = None
+    for ch in line:
+        if quote:
+            out.append(ch)
+            if ch == quote:
+                quote = None
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            out.append(ch)
+            continue
+        if ch in ("#", "!"):
+            break
+        out.append(ch)
+    return "".join(out)
+
+
+def _split_tokens(line: str) -> List[str]:
+    """Whitespace split that keeps quoted strings whole and isolates braces, so
+    `virtual_ipaddress { 10.0.0.1/24 dev eth0 }` tokenizes the same as its multi-line form."""
+    tokens: List[str] = []
+    buf: List[str] = []
+    quote: Optional[str] = None
+
+    def flush() -> None:
+        if buf:
+            tokens.append("".join(buf))
+            buf.clear()
+
+    for ch in line:
+        if quote:
+            if ch == quote:
+                quote = None
+            else:
+                buf.append(ch)
+            continue
+        if ch in ('"', "'"):
+            quote = ch
+            continue
+        if ch.isspace():
+            flush()
+        elif ch in ("{", "}"):
+            flush()
+            tokens.append(ch)
+        else:
+            buf.append(ch)
+    flush()
+    return tokens
+
+
+def _read_blocks(text: str) -> List[Dict[str, Any]]:
+    """Parse the file into nested entries.
+
+    Each entry is either
+      {"kind": "block", "name": str, "args": [str], "body": [entries], "line": int}
+      {"kind": "line",  "tokens": [str], "line": int}
+
+    Line boundaries matter: inside `virtual_ipaddress` and `unicast_peer` each line is one
+    bare value, so a flat token stream could not tell two addresses apart.
+    """
+    root: List[Dict[str, Any]] = []
+    stack: List[List[Dict[str, Any]]] = [root]
+    # Blocks whose opening `{` we have seen, so a stray `}` can be reported with context.
+    open_blocks: List[str] = []
+
+    for lineno, raw in enumerate(text.splitlines(), start=1):
+        pending: List[str] = []
+        for tok in _split_tokens(_strip_comment(raw)):
+            if tok == "{":
+                name = pending[0] if pending else ""
+                args = pending[1:]
+                block = {"kind": "block", "name": name, "args": args, "body": [], "line": lineno}
+                stack[-1].append(block)
+                stack.append(block["body"])
+                open_blocks.append(name)
+                pending = []
+            elif tok == "}":
+                if pending:
+                    stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+                    pending = []
+                if len(stack) == 1:
+                    raise KeepalivedParseError(f"unbalanced '}}' on line {lineno}")
+                stack.pop()
+                open_blocks.pop()
+            else:
+                pending.append(tok)
+        if pending:
+            stack[-1].append({"kind": "line", "tokens": pending, "line": lineno})
+
+    if len(stack) != 1:
+        raise KeepalivedParseError(f"unclosed block '{open_blocks[-1] or '?'}' at end of file")
+    return root
+
+
+# ---------------------------------------------------------------------------
+# Interpretation
+# ---------------------------------------------------------------------------
+def _as_int(tokens: List[str]) -> Optional[int]:
+    if len(tokens) < 2:
+        return None
+    try:
+        return int(tokens[1])
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_vip_entry(tokens: List[str]) -> Optional[Dict[str, Any]]:
+    """One `virtual_ipaddress` line: `<addr>[/<prefix>] [dev <iface>] [label ...]`.
+
+    Returns None when the first token is not an address — a shape we do not understand must
+    surface as unsupported rather than be silently dropped.
+    """
+    spec = tokens[0]
+    addr, _, prefix = spec.partition("/")
+    try:
+        ip = ipaddress.ip_address(addr)
+    except ValueError:
+        return None
+    entry: Dict[str, Any] = {
+        "address": str(ip),
+        "prefix_length": None,
+        "dev": None,
+        "extra": [],
+    }
+    if prefix:
+        try:
+            entry["prefix_length"] = int(prefix)
+        except ValueError:
+            return None
+    rest = tokens[1:]
+    i = 0
+    while i < len(rest):
+        if rest[i] == "dev" and i + 1 < len(rest):
+            entry["dev"] = rest[i + 1]
+            i += 2
+            continue
+        # `label`, `scope`, `brd`, ... — all real directives we do not render.
+        entry["extra"].append(rest[i])
+        i += 1
+    return entry
+
+
+def _classify_script(block: Dict[str, Any]) -> Tuple[str, Optional[str]]:
+    """Return (name, script_path) for a vrrp_script block."""
+    name = block["args"][0] if block["args"] else (block["name"] or "")
+    path = None
+    for entry in block["body"]:
+        if entry["kind"] == "line" and entry["tokens"] and entry["tokens"][0] == "script":
+            path = " ".join(entry["tokens"][1:]) or None
+    return name, path
+
+
+def _parse_instance(block: Dict[str, Any]) -> Dict[str, Any]:
+    """Interpret one `vrrp_instance` block into VIP-model fields plus its own unsupported list."""
+    inst: Dict[str, Any] = {
+        "instance_name": block["args"][0] if block["args"] else "",
+        "state": None,
+        "interface": None,
+        "virtual_router_id": None,
+        "priority": None,
+        "advert_int": None,
+        "auth_type": None,
+        "auth_pass": None,
+        "unicast_src_ip": None,
+        "unicast_peers": [],
+        "virtual_ips": [],
+        "track_scripts": [],
+        "unsupported": [],
+        "line": block["line"],
+    }
+
+    def unsupported(what: str, lineno: int) -> None:
+        inst["unsupported"].append({"directive": what, "line": lineno})
+
+    for entry in block["body"]:
+        if entry["kind"] == "line":
+            tokens = entry["tokens"]
+            key = tokens[0]
+            if key == "state":
+                inst["state"] = (tokens[1].upper() if len(tokens) > 1 else None)
+            elif key == "interface":
+                inst["interface"] = tokens[1] if len(tokens) > 1 else None
+            elif key == "virtual_router_id":
+                inst["virtual_router_id"] = _as_int(tokens)
+            elif key == "priority":
+                inst["priority"] = _as_int(tokens)
+            elif key == "advert_int":
+                # keepalived accepts sub-second floats; our model column is an integer.
+                raw = tokens[1] if len(tokens) > 1 else ""
+                try:
+                    val = float(raw)
+                except (TypeError, ValueError):
+                    val = None
+                if val is None:
+                    unsupported(f"advert_int {raw}", entry["line"])
+                elif val != int(val):
+                    # Rounding would change VRRP timing, so refuse rather than adopt-and-alter.
+                    unsupported(f"advert_int {raw} (fractional; model stores whole seconds)",
+                                entry["line"])
+                else:
+                    inst["advert_int"] = int(val)
+            elif key == "unicast_src_ip":
+                inst["unicast_src_ip"] = tokens[1] if len(tokens) > 1 else None
+            else:
+                unsupported(" ".join(tokens), entry["line"])
+            continue
+
+        name = entry["name"]
+        if name == "authentication":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    continue
+                k = sub["tokens"][0]
+                if k == "auth_type":
+                    inst["auth_type"] = (sub["tokens"][1].upper() if len(sub["tokens"]) > 1 else None)
+                elif k == "auth_pass":
+                    # Everything after the keyword: a VRRP password may contain spaces.
+                    inst["auth_pass"] = " ".join(sub["tokens"][1:]) or None
+                else:
+                    unsupported(f"authentication/{' '.join(sub['tokens'])}", sub["line"])
+        elif name == "unicast_peer":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["unicast_peers"].append(sub["tokens"][0])
+                else:
+                    unsupported("unicast_peer/<block>", entry["line"])
+        elif name == "virtual_ipaddress":
+            for sub in entry["body"]:
+                if sub["kind"] != "line" or not sub["tokens"]:
+                    unsupported("virtual_ipaddress/<block>", entry["line"])
+                    continue
+                parsed = _parse_vip_entry(sub["tokens"])
+                if parsed is None:
+                    unsupported(f"virtual_ipaddress/{' '.join(sub['tokens'])}", sub["line"])
+                else:
+                    if parsed["extra"]:
+                        unsupported(
+                            f"virtual_ipaddress/{parsed['address']} "
+                            f"({' '.join(parsed['extra'])})", sub["line"])
+                    inst["virtual_ips"].append(parsed)
+        elif name == "track_script":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    inst["track_scripts"].append(sub["tokens"][0])
+        else:
+            unsupported(f"{name} {{...}}", entry["line"])
+
+    return inst
+
+
+def parse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse a keepalived.conf into VIP-model fields plus everything we could not model.
+
+    Raises KeepalivedParseError on structurally broken input. Never log the result: parsed
+    instances carry `auth_pass` in cleartext.
+    """
+    root = _read_blocks(text or "")
+    result: Dict[str, Any] = {
+        "instances": [],
+        "scripts": {},
+        "global_defs": {},
+        "unsupported": [],   # top-level directives our render would drop
+        "sync_groups": [],
+    }
+
+    for entry in root:
+        if entry["kind"] == "line":
+            # A bare top-level directive (e.g. `include /etc/keepalived/conf.d/*.conf`).
+            result["unsupported"].append(
+                {"directive": " ".join(entry["tokens"]), "line": entry["line"]})
+            continue
+        name = entry["name"]
+        if name == "global_defs":
+            for sub in entry["body"]:
+                if sub["kind"] == "line" and sub["tokens"]:
+                    key = sub["tokens"][0]
+                    result["global_defs"][key] = " ".join(sub["tokens"][1:])
+                    if key not in _SUPPORTED_GLOBAL_KEYS:
+                        result["unsupported"].append(
+                            {"directive": f"global_defs/{' '.join(sub['tokens'])}",
+                             "line": sub["line"]})
+                else:
+                    result["unsupported"].append(
+                        {"directive": f"global_defs/{sub.get('name', '?')} {{...}}",
+                         "line": sub["line"]})
+        elif name == "vrrp_script":
+            script_name, path = _classify_script(entry)
+            result["scripts"][script_name] = {"script": path, "line": entry["line"]}
+        elif name == "vrrp_instance":
+            result["instances"].append(_parse_instance(entry))
+        elif name == "vrrp_sync_group":
+            # A sync group ties instances together so they fail over as a unit. Our render has
+            # no equivalent, and dropping it changes failover semantics — never adopt silently.
+            group = entry["args"][0] if entry["args"] else ""
+            result["sync_groups"].append({"name": group, "line": entry["line"]})
+            result["unsupported"].append(
+                {"directive": f"vrrp_sync_group {group}", "line": entry["line"]})
+        else:
+            # virtual_server (LVS), static_routes, bfd_instance, ...
+            args = " ".join(entry["args"])
+            result["unsupported"].append(
+                {"directive": f"{name} {args} {{...}}".replace("  ", " "), "line": entry["line"]})
+
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Mapping to the VIP model + the adoption gate
+# ---------------------------------------------------------------------------
+# keepalived defaults we are willing to apply when a directive is absent, because the value
+# is unambiguous and re-rendering it changes nothing on the wire.
+_DEFAULT_ADVERT_INT = 1
+_DEFAULT_PRIORITY = 100
+_DEFAULT_STATE = "BACKUP"
+
+# The only track_script our renderer emits (keepalived_config.build_haproxy_check_script).
+OUR_CHECK_SCRIPT_NAME = "chk_haproxy"
+
+
+def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -> Dict[str, Any]:
+    """Map one parsed `vrrp_instance` onto vip_instances / vip_members fields.
+
+    Returns `adoptable` plus `blockers`. A blocker means taking ownership would change what
+    is running — either because our render cannot reproduce something in the file, or because
+    a value we must write is not knowable from the file. Adoption REPLACES the operator's
+    config, so "we could not read it" and "we would change it" are the same hazard, and both
+    have to stop the flow rather than be logged.
+
+    Never log the return value: `vip.auth_pass` is cleartext.
+    """
+    blockers: List[str] = []
+
+    # Directives we would drop. Report the file's own line numbers so the operator can look.
+    dropped = list(parsed.get("unsupported") or []) + list(instance.get("unsupported") or [])
+    for d in dropped:
+        blockers.append(
+            f"line {d['line']}: `{d['directive']}` — OpenManager's renderer cannot reproduce "
+            f"this, so adopting would delete it")
+
+    vips = instance.get("virtual_ips") or []
+    if len(vips) == 0:
+        blockers.append("the instance declares no virtual_ipaddress — nothing to adopt")
+    elif len(vips) > 1:
+        addrs = ", ".join(v["address"] for v in vips)
+        blockers.append(
+            f"the instance carries {len(vips)} addresses ({addrs}); a managed VIP holds exactly "
+            f"one, so adopting would drop all but the first")
+
+    vip_entry = vips[0] if vips else None
+
+    if instance.get("virtual_router_id") is None:
+        blockers.append("no virtual_router_id — it cannot be guessed: a wrong VRID puts the "
+                        "nodes in separate VRRP domains and both would claim the VIP")
+    if not instance.get("interface"):
+        blockers.append("no interface — required to render the instance and the address")
+
+    # An explicit prefix is required. Our renderer ALWAYS writes `<addr>/<prefix>`, the model
+    # column defaults to 24, and keepalived's own default for a bare address is a host route.
+    # Picking either one for the operator would silently change the VIP's netmask, so ask.
+    if vip_entry is not None and vip_entry.get("prefix_length") is None:
+        blockers.append(
+            f"`{vip_entry['address']}` has no explicit prefix length; state it during adoption "
+            f"so the netmask cannot change on takeover")
+
+    # The address must live on the instance's interface — that is the only `dev` we can render.
+    if vip_entry is not None and vip_entry.get("dev") and instance.get("interface") \
+            and vip_entry["dev"] != instance["interface"]:
+        blockers.append(
+            f"the address is bound to `dev {vip_entry['dev']}` but the instance uses "
+            f"`interface {instance['interface']}`; the render always uses the instance interface")
+
+    auth_type = instance.get("auth_type")
+    if auth_type not in (None, "PASS"):
+        blockers.append(f"auth_type {auth_type} is not supported (only PASS is rendered)")
+
+    # A tracked script that is not ours would be replaced by our HAProxy check.
+    tracked = [t for t in (instance.get("track_scripts") or [])]
+    foreign = [t for t in tracked if t != OUR_CHECK_SCRIPT_NAME]
+    if foreign:
+        blockers.append(
+            f"track_script {', '.join(foreign)} would be replaced by OpenManager's HAProxy "
+            f"health check")
+
+    state = instance.get("state") or _DEFAULT_STATE
+    if state not in ("MASTER", "BACKUP"):
+        blockers.append(f"state {state} is not MASTER or BACKUP")
+
+    peers = list(instance.get("unicast_peers") or [])
+    src = instance.get("unicast_src_ip")
+    # Our renderer emits unicast_src_ip and unicast_peer together, or neither.
+    if bool(src) != bool(peers):
+        which = "unicast_src_ip without unicast_peer" if src else "unicast_peer without unicast_src_ip"
+        blockers.append(f"{which} — the render emits both or neither")
+
+    candidate: Dict[str, Any] = {
+        "instance_name": instance.get("instance_name") or "",
+        "adoptable": not blockers,
+        "blockers": blockers,
+        "dropped_directives": dropped,
+        "vip": {
+            "virtual_ip": vip_entry["address"] if vip_entry else None,
+            "prefix_length": vip_entry.get("prefix_length") if vip_entry else None,
+            "virtual_router_id": instance.get("virtual_router_id"),
+            "advert_int": instance.get("advert_int") if instance.get("advert_int") is not None
+            else _DEFAULT_ADVERT_INT,
+            "use_unicast": bool(peers),
+            "track_haproxy": OUR_CHECK_SCRIPT_NAME in tracked,
+            "auth_pass": instance.get("auth_pass"),
+        },
+        "member": {
+            "network_interface": instance.get("interface"),
+            "role": state,
+            "priority": instance.get("priority") if instance.get("priority") is not None
+            else _DEFAULT_PRIORITY,
+        },
+        "peers": peers,
+        "unicast_src_ip": src,
+        # Which values came from a keepalived default rather than the file, so the UI can say so.
+        "defaulted": [
+            k for k, present in (
+                ("advert_int", instance.get("advert_int") is not None),
+                ("priority", instance.get("priority") is not None),
+                ("state", instance.get("state") is not None),
+            ) if not present
+        ],
+    }
+    return candidate
+
+
+# Substrings that identify the two blocker classes an operator is allowed to resolve. They are
+# matched rather than typed because the blocker text is what the UI shows; keeping the marker in
+# the sentence means the message and the rule cannot drift apart.
+_LOSS_MARKER = "would delete it"
+_PREFIX_MARKER = "no explicit prefix length"
+
+
+def remaining_blockers(blockers: List[str], *, prefix_supplied: bool = False,
+                       accept_data_loss: bool = False) -> List[str]:
+    """Blockers that survive what the operator is permitted to resolve.
+
+    Exactly two classes are resolvable, and the distinction is the whole safety argument:
+
+      * a missing prefix length is *unknown*, and the operator can supply it — we refuse to pick
+        a netmask for a live VIP ourselves;
+      * "our renderer cannot reproduce this, so adopting would delete it" is a *loss*, and losing
+        it can be an informed choice.
+
+    Everything else — an unknown virtual_router_id, a fractional advert_int, an unsupported
+    auth_type, an address on a different interface — is neither unknown nor a loss but an
+    impossibility, and no flag may wave it through. This is the single source of truth for that
+    rule; the endpoint and the UI both derive from it.
+    """
+    out: List[str] = []
+    for b in blockers or []:
+        if prefix_supplied and _PREFIX_MARKER in b:
+            continue
+        if accept_data_loss and _LOSS_MARKER in b:
+            continue
+        out.append(b)
+    return out
+
+
+def analyse_keepalived_conf(text: str) -> Dict[str, Any]:
+    """Parse + map in one call: the shape the discovery endpoint stores and the UI renders."""
+    parsed = parse_keepalived_conf(text)
+    return {
+        "instance_count": len(parsed["instances"]),
+        "sync_groups": parsed["sync_groups"],
+        "global_defs": parsed["global_defs"],
+        "candidates": [build_adoption_candidate(parsed, inst) for inst in parsed["instances"]],
+    }

backend/services/mfa_service.py

@@ -0,0 +1,238 @@
+"""MFA (TOTP + backup codes) service layer — Issue #18, v1.6.0.
+
+Owns the cryptographic and persistence-shape concerns of multi-factor auth:
+  - TOTP secret generation / verification with replay protection (RFC 6238)
+  - Backup code generation, hashing (bcrypt) and atomic single-use consumption
+  - Fernet-based encryption of TOTP secrets at rest
+
+Strictly no logging of secrets — only metadata (lengths, counts) is logged.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import secrets as _secrets
+import time
+from typing import List, Optional, Tuple
+from urllib.parse import quote
+
+import bcrypt
+import pyotp
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# RFC 6238 parameters — kept conservative for widest authenticator app compatibility.
+TOTP_DIGITS = 6
+TOTP_PERIOD = 30
+TOTP_DIGEST = "sha1"
+TOTP_VALID_WINDOW_STEPS = 1  # ±1 step (±30s) tolerance
+
+# Backup code spec (Plan section 5).
+# Alphabet drops the confusing pairs: 0/O, 1/I, L. Resulting size is 31, which
+# still yields 31**8 ≈ 8.5×10^11 combinations per half — far beyond brute-force.
+BACKUP_CODE_COUNT = 10
+BACKUP_CODE_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"
+BACKUP_CODE_HALF_LEN = 4  # XXXX-YYYY
+
+# OTP URI defaults.
+DEFAULT_ISSUER = "HAProxy OpenManager"
+ACCOUNT_LABEL_DOMAIN_FALLBACK = "haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Fernet key resolution
+# ---------------------------------------------------------------------------
+
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Resolve the Fernet key, preferring the explicit env var.
+
+    Falls back to HKDF over SECRET_KEY with a versioned info string so a future
+    rotation can be expressed by bumping the version suffix.
+    """
+    explicit = os.getenv("MFA_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:
+            logger.error("MFA_ENCRYPTION_KEY env var present but invalid: %s", exc)
+            # fall through to HKDF derivation rather than crashing the app
+
+    logger.warning(
+        "MFA_ENCRYPTION_KEY env var not set or invalid; deriving from SECRET_KEY (v1). "
+        "Set an explicit MFA_ENCRYPTION_KEY in production to enable key rotation."
+    )
+    hkdf = HKDF(
+        algorithm=hashes.SHA256(),
+        length=32,
+        salt=None,
+        info=b"mfa-totp-secret-v1",
+    )
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution of the Fernet key after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+# ---------------------------------------------------------------------------
+# TOTP secrets
+# ---------------------------------------------------------------------------
+
+
+def generate_totp_secret() -> str:
+    """Return a fresh base32 TOTP secret (32 chars)."""
+    return pyotp.random_base32()
+
+
+def encrypt_secret(secret_plain: str) -> str:
+    """Fernet-encrypt the base32 secret. Returns str for direct DB storage."""
+    token = _get_fernet().encrypt(secret_plain.encode("utf-8"))
+    return token.decode("utf-8")
+
+
+def decrypt_secret(secret_encrypted: str) -> Optional[str]:
+    """Decrypt a previously stored secret. Returns None when the token can't be
+    decrypted (e.g. key rotated without re-enroll). Never raises to the caller.
+    """
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt MFA secret (invalid Fernet token)")
+        return None
+    except Exception as exc:
+        logger.error("Unexpected error decrypting MFA secret: %s", exc)
+        return None
+
+
+def build_otpauth_uri(account_label: str, secret_plain: str, issuer: str = DEFAULT_ISSUER) -> str:
+    """Build an otpauth:// URI that all major authenticator apps accept.
+
+    Format: otpauth://totp/<issuer>:<account>?secret=<b32>&issuer=<issuer>&algorithm=SHA1&digits=6&period=30
+    """
+    issuer_q = quote(issuer, safe="")
+    label = f"{issuer}:{account_label}"
+    label_q = quote(label, safe=":@")
+    return (
+        f"otpauth://totp/{label_q}?secret={secret_plain}"
+        f"&issuer={issuer_q}&algorithm=SHA1&digits={TOTP_DIGITS}&period={TOTP_PERIOD}"
+    )
+
+
+def build_account_label(username: str, hostname_hint: Optional[str] = None) -> str:
+    """Compose the per-user otpauth label, respecting env > hostname > fallback."""
+    domain = (
+        os.getenv("MFA_ACCOUNT_LABEL_DOMAIN", "").strip()
+        or (hostname_hint or "").strip()
+        or ACCOUNT_LABEL_DOMAIN_FALLBACK
+    )
+    return f"{username}@{domain}"
+
+
+def verify_totp_with_replay_guard(
+    secret_plain: str,
+    code: str,
+    last_used_step: Optional[int],
+) -> Tuple[bool, Optional[int]]:
+    """Verify a 6-digit TOTP code with explicit per-step replay protection.
+
+    Returns (success, step_consumed). Caller persists the consumed step on success.
+
+    Implementation notes:
+      - pyotp.TOTP.at(seconds_since_epoch) — to target step N we pass step*PERIOD.
+      - secrets.compare_digest is used for constant-time comparison.
+      - Replay guard rejects codes whose step is <= the previously consumed step.
+    """
+    if not secret_plain or not code:
+        return (False, None)
+    code = code.strip()
+    if len(code) != TOTP_DIGITS or not code.isdigit():
+        return (False, None)
+
+    totp = pyotp.TOTP(secret_plain, digits=TOTP_DIGITS, interval=TOTP_PERIOD, digest=TOTP_DIGEST)
+    now = int(time.time())
+    current_step = now // TOTP_PERIOD
+
+    for offset in (0, -1, 1):
+        step = current_step + offset
+        expected = totp.at(step * TOTP_PERIOD)
+        if len(expected) == len(code) and _secrets.compare_digest(expected, code):
+            if last_used_step is not None and step <= last_used_step:
+                return (False, None)
+            return (True, step)
+    return (False, None)
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+def generate_backup_codes(count: int = BACKUP_CODE_COUNT) -> List[str]:
+    """Return ``count`` plain-text backup codes formatted as ``XXXX-YYYY``."""
+    codes: List[str] = []
+    for _ in range(count):
+        left = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        right = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        codes.append(f"{left}-{right}")
+    return codes
+
+
+def normalize_backup_code(user_input: str) -> str:
+    """Canonical form for comparison: uppercase, strip dashes/spaces."""
+    if not user_input:
+        return ""
+    return user_input.strip().upper().replace("-", "").replace(" ", "")
+
+
+async def _hash_one_backup_code(code_plain: str) -> str:
+    """Bcrypt-hash a single backup code on a worker thread."""
+    normalized = normalize_backup_code(code_plain)
+    hashed = await asyncio.to_thread(bcrypt.hashpw, normalized.encode("utf-8"), bcrypt.gensalt())
+    return hashed.decode("utf-8")
+
+
+async def hash_backup_codes(codes_plain: List[str]) -> List[str]:
+    """Hash backup codes in parallel (each bcrypt op runs in its own thread)."""
+    return await asyncio.gather(*(_hash_one_backup_code(c) for c in codes_plain))
+
+
+async def check_backup_code(user_input: str, code_hash: str) -> bool:
+    """Run a single bcrypt verify on the worker pool."""
+    normalized = normalize_backup_code(user_input)
+    if not normalized:
+        return False
+    return await asyncio.to_thread(
+        bcrypt.checkpw, normalized.encode("utf-8"), code_hash.encode("utf-8")
+    )
+
+
+# ---------------------------------------------------------------------------
+# Misc helpers
+# ---------------------------------------------------------------------------
+
+
+def generate_challenge_token() -> str:
+    """64-char hex challenge token for /api/auth/login → /mfa-verify hand-off."""
+    return _secrets.token_hex(32)

backend/services/ssl_service.py

@@ -40,10 +40,12 @@ flow.
   (callers translate to wizard step-jumpback toasts).
 """
 
+import hashlib
 import json
 import logging
+import time
 from datetime import datetime, timezone
-from typing import Any, Optional
+from typing import Any, List, Optional
 
 from fastapi import HTTPException
 
@@ -106,23 +108,22 @@ def _recompute_status_from_expiry(
         return cert_info_status or "valid", cert_info_days or 0
 
 
-async def create_cert_row(
-    conn,
-    payload: Any,
-    cluster_id: int,
-) -> int:
-    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
-    binding to the given cluster_id. Returns new ssl_certificate_id.
+def _prepare_cert_fields(payload: Any) -> dict:
+    """Parse + validate the PEM material on `payload` and derive every
+    ssl_certificates column value from it (v1.9.0 extraction — shared by
+    `create_cert_row` and the CSR import flow in services/csr_service.py,
+    byte-identical to the former inline body of `create_cert_row`).
 
     payload is expected to expose:
       name, certificate_content, private_key_content, chain_content,
       usage_type (optional, default 'frontend').
 
-    All cert metadata (primary_domain, all_domains, expiry_date,
-    issuer, fingerprint, status, days_until_expiry) is now parsed
-    FROM the PEM content via `parse_ssl_certificate` — operator-
-    supplied values on the payload are accepted as a graceful
-    fallback only when parsing fails (which itself raises 400).
+    Raises HTTPException(400) on any parse/validation failure (invalid PEM,
+    bad private key, cert/key mismatch, bad chain, already-expired cert).
+
+    Returns a dict with keys: cert_content, private_key_content,
+    chain_content, cert_info, primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry, usage_type.
     """
     cert_content = getattr(payload, "certificate_content", None) or ""
     if not cert_content.strip():
@@ -213,6 +214,53 @@ async def create_cert_row(
     )
     usage_type = getattr(payload, "usage_type", "frontend") or "frontend"
 
+    return {
+        "cert_content": cert_content,
+        "private_key_content": private_key_content,
+        "chain_content": chain_content,
+        "cert_info": cert_info,
+        "primary_domain": primary_domain,
+        "all_domains": all_domains,
+        "expiry_date": expiry_date,
+        "issuer": issuer,
+        "fingerprint": fingerprint,
+        "status": status,
+        "days_until_expiry": days_until_expiry,
+        "usage_type": usage_type,
+    }
+
+
+async def create_cert_row(
+    conn,
+    payload: Any,
+    cluster_id: int,
+) -> int:
+    """Insert a row into ssl_certificates (always cluster_id=NULL) + junction
+    binding to the given cluster_id. Returns new ssl_certificate_id.
+
+    payload is expected to expose:
+      name, certificate_content, private_key_content, chain_content,
+      usage_type (optional, default 'frontend').
+
+    All cert metadata (primary_domain, all_domains, expiry_date,
+    issuer, fingerprint, status, days_until_expiry) is now parsed
+    FROM the PEM content via `parse_ssl_certificate` — operator-
+    supplied values on the payload are accepted as a graceful
+    fallback only when parsing fails (which itself raises 400).
+    """
+    fields = _prepare_cert_fields(payload)
+    cert_content = fields["cert_content"]
+    private_key_content = fields["private_key_content"]
+    chain_content = fields["chain_content"]
+    expiry_date = fields["expiry_date"]
+    primary_domain = fields["primary_domain"]
+    all_domains = fields["all_domains"]
+    issuer = fields["issuer"]
+    fingerprint = fields["fingerprint"]
+    status = fields["status"]
+    days_until_expiry = fields["days_until_expiry"]
+    usage_type = fields["usage_type"]
+
     existing = await conn.fetchrow(
         """
         SELECT s.id, s.is_active
@@ -408,3 +456,81 @@ async def validate_server_ca_bundle_eligibility(
         cluster_id,
     )
     return row is not None
+
+
+async def stage_ssl_config_versions(
+    conn,
+    cert_id: int,
+    cluster_ids: List[int],
+    action: str = "create",
+    created_by: Optional[int] = None,
+) -> List[dict]:
+    """Stage one PENDING config version per affected cluster after an SSL
+    certificate mutation (v1.9.0 — distilled from the routers/ssl.py POST
+    /certificates staging loop; used by the CSR import flow).
+
+    Uses the EXACT `ssl-{cert_id}-{action}-{timestamp}` version-name scheme of
+    the manual SSL flow so Apply Management, the `has_pending_config`
+    LIKE-filter ('ssl-' || id || '-%'), and the agent delivery predicates
+    treat CSR-imported certificates identically to manually uploaded ones.
+    Agents are NOT notified here — the operator applies manually.
+
+    Per-cluster failures are caught and reported in the returned
+    sync_results list (the DB save has already succeeded — same semantics as
+    the manual flow, where a config-generation failure never rolls back the
+    certificate row).
+    """
+    # Local import: keeps services/haproxy_config free to import ssl helpers
+    # without a module-level cycle.
+    from services.haproxy_config import generate_haproxy_config_for_cluster
+
+    sync_results: List[dict] = []
+    for cluster_id in cluster_ids:
+        try:
+            config_content = await generate_haproxy_config_for_cluster(cluster_id)
+            config_hash = hashlib.sha256(config_content.encode()).hexdigest()
+            version_name = f"ssl-{cert_id}-{action}-{int(time.time())}"
+
+            version_created_by = created_by
+            if version_created_by is None:
+                version_created_by = await conn.fetchval(
+                    "SELECT id FROM users WHERE username = 'admin' LIMIT 1"
+                ) or 1
+
+            await conn.fetchval(
+                """
+                INSERT INTO config_versions
+                (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
+                VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                RETURNING id
+                """,
+                cluster_id,
+                version_name,
+                config_content,
+                config_hash,
+                version_created_by,
+            )
+            logger.info(
+                f"APPLY WORKFLOW: Created PENDING config version {version_name} "
+                f"for cluster {cluster_id} (ssl_service.stage_ssl_config_versions)"
+            )
+            sync_results.append({
+                'node': 'pending',
+                'success': True,
+                'cluster_id': cluster_id,
+                'version': version_name,
+                'status': 'PENDING',
+                'message': 'SSL certificate staged. Click Apply to activate.',
+            })
+        except Exception as e:
+            logger.error(
+                f"Cluster config staging failed for SSL certificate {cert_id} "
+                f"on cluster {cluster_id}: {e}"
+            )
+            sync_results.append({
+                'node': 'cluster',
+                'success': False,
+                'cluster_id': cluster_id,
+                'error': str(e),
+            })
+    return sync_results

backend/tests/test_acl_pattern_file_allow.py

@@ -0,0 +1,190 @@
+"""Issue #38 follow-up — ACL `-f <file>` pattern-file support (v1.8.9).
+
+The Bulgu #12 hard rejects were removed: pattern files are
+operator-managed host files (same policy as the SPOE
+`filter ... config <path>` reference preserved since v1.8.8), bulk
+import always accepted `-f`, and the agent runs `haproxy -c` before
+every reload so a missing file fails safely. These tests pin:
+
+1. ACCEPT — the manual FrontendConfig model and the wizard models
+   accept `-f` in every rule field (string + dict shapes).
+2. GUARDS KEPT — `$(`/backtick shell-substitution rejects and the
+   `X !X` contradiction machinery are unchanged.
+3. WARNINGS — `_pattern_file_warnings` emits exactly one advisory
+   listing the referenced files, and NOTHING for `-f`-free rules
+   (zero-noise: existing users see no new output).
+4. ADVISORY — the bulk-import preview advisory block scans
+   acl/use_backend rules (and only those fields).
+"""
+
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from models.frontend import FrontendConfig  # noqa: E402
+from routers.frontend import _pattern_file_warnings  # noqa: E402
+
+
+ACL_F = "blacklisted src -f /etc/haproxy/blacklist.lst"
+UB_F = "be-secure if { src -f /etc/haproxy/allowlist.lst }"
+REDIR_F = "location /blocked if { src -f /etc/haproxy/blacklist.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 1. ACCEPT — manual FrontendConfig model
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_frontend_config_accepts_acl_file_flag():
+    fe = FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[ACL_F])
+    assert fe.acl_rules == [ACL_F]
+
+
+def test_frontend_config_accepts_use_backend_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", use_backend_rules=[UB_F])
+    assert fe.use_backend_rules == [UB_F]
+
+
+def test_frontend_config_accepts_redirect_string_file_flag():
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[REDIR_F])
+    assert fe.redirect_rules == [REDIR_F]
+
+
+def test_frontend_config_accepts_redirect_dict_file_flag():
+    rule = {"type": "scheme", "scheme": "https",
+            "condition": "if { src -f /etc/haproxy/blacklist.lst }"}
+    fe = FrontendConfig(
+        name="fe1", bind_port=80, mode="http", redirect_rules=[rule])
+    assert fe.redirect_rules == [rule]
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 2. GUARDS KEPT — dangerous-content rejects unchanged
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "acl1 path $(rm -rf /)",
+    "acl1 path `id`",
+])
+def test_acl_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(name="fe1", bind_port=80, mode="http", acl_rules=[bad_rule])
+
+
+@pytest.mark.parametrize("bad_rule", [
+    "be1 if $(whoami)",
+    "be1 if `id`",
+])
+def test_use_backend_shell_substitution_still_rejected(bad_rule):
+    from pydantic import ValidationError
+    with pytest.raises(ValidationError):
+        FrontendConfig(
+            name="fe1", bind_port=80, mode="http", use_backend_rules=[bad_rule])
+
+
+def test_contradiction_detection_still_works_on_file_flag_rules():
+    """Interaction guard: a `-f` rule with an `X !X` contradiction is
+    still caught by the handler-level contradiction machinery — the
+    `-f` relaxation must not weaken that gate."""
+    from models.frontend import _frontend_has_acl_contradiction
+    assert _frontend_has_acl_contradiction(
+        "be1 if blacklisted !blacklisted") is True
+    # And a normal -f rule is NOT a contradiction.
+    assert _frontend_has_acl_contradiction(UB_F) is False
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 3. WARNINGS — _pattern_file_warnings (zero-noise contract)
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_pattern_file_warnings_lists_unique_paths():
+    warnings = _pattern_file_warnings(
+        acl_rules=[ACL_F, "other src -f /etc/haproxy/blacklist.lst"],
+        use_backend_rules=[UB_F],
+        redirect_rules=[{"condition": "if { src -f /etc/haproxy/geo.lst }"}],
+    )
+    assert len(warnings) == 1
+    w = warnings[0]
+    assert "/etc/haproxy/blacklist.lst" in w
+    assert "/etc/haproxy/allowlist.lst" in w
+    assert "/etc/haproxy/geo.lst" in w
+    # Duplicate path listed once.
+    assert w.count("/etc/haproxy/blacklist.lst") == 1
+    # Non-blocking framing: mentions fail-safe haproxy -c.
+    assert "haproxy -c" in w
+
+
+def test_pattern_file_warnings_empty_without_file_flag():
+    """Zero-noise: operators who don't use `-f` must see NO warning."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_api path_beg /api", "is_admin src 10.0.0.0/24"],
+        use_backend_rules=["be-api if is_api"],
+        redirect_rules=[{"type": "scheme", "scheme": "https",
+                         "condition": "if !{ ssl_fc }"}],
+    ) == []
+    assert _pattern_file_warnings() == []
+
+
+def test_pattern_file_warnings_ignores_dash_f_substrings():
+    """`-file`/`-foo` substrings must not trigger the advisory."""
+    assert _pattern_file_warnings(
+        acl_rules=["is_self path_beg /self-config-file",
+                   "is_foo path_beg /foo -m beg"],
+    ) == []
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 4. Wizard models accept `-f` (string + dict) — parity
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_wizard_models_accept_file_flag():
+    from models.site_wizard import FrontendStep
+
+    fe = FrontendStep(
+        name="fe1", mode="http", bind_address="*", bind_port=80,
+        acl_rules=[ACL_F],
+        use_backend_rules=["be-x if blacklisted"],
+        redirect_rules=[{"type": "scheme", "target": "https",
+                         "condition": "if { src -f /etc/haproxy/x.lst }"}],
+    )
+    assert fe.acl_rules == [ACL_F]
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/x.lst }"
+
+
+# ──────────────────────────────────────────────────────────────────────
+# 5. Bulk-import preview advisory — source-level pin
+# ──────────────────────────────────────────────────────────────────────
+
+
+def test_parse_bulk_advisory_scans_only_structured_rule_fields():
+    """The preview advisory scans acl_rules/use_backend_rules but NOT
+    request_headers/tcp_request_rules (always-free-form fields —
+    warning there would add new noise for existing users)."""
+    src = Path(__file__).resolve().parents[1] / "routers" / "config.py"
+    text = src.read_text()
+    block_start = text.index("pattern-file advisory")
+    block = text[block_start:block_start + 1200]
+    assert 'acl_rules' in block
+    assert 'use_backend_rules' in block
+    assert 'request_headers' not in block.split("_pattern_paths")[1], (
+        "advisory must not scan request_headers")
+
+
+def test_no_dash_f_reject_left_in_models():
+    """No model file may still hard-reject the `-f` flag."""
+    for rel in ("models/frontend.py", "models/site_wizard.py"):
+        text = (Path(__file__).resolve().parents[1] / rel).read_text()
+        for m in re.finditer(r"-f\(\\s\|\$\)", text):
+            ctx = text[max(0, m.start() - 400):m.start() + 400]
+            assert "raise ValueError" not in ctx, (
+                f"{rel}: a `-f` reject regex still sits next to a raise")

backend/tests/test_acme_backend_url.py

@@ -0,0 +1,291 @@
+"""ACME challenge backend URL validation, resolution and change detection.
+
+These pin the behaviour behind a real incident: a split deployment rendered
+`server _acme_mgmt <mgmt>:8080` against a port with no listener, HTTP-01 failed for
+weeks while DNS-01 kept working, and every existing check reported success. The
+three mechanisms below are what make that impossible to repeat silently.
+"""
+import pytest
+
+from services.haproxy_config import (
+    extract_acme_backend_target,
+    is_config_generation_error,
+    select_acme_backend_source,
+)
+from utils.acme_backend_url import (
+    AcmeBackendUrlError,
+    resolve_acme_backend_target,
+    validate_acme_backend_url,
+)
+
+
+# ---------------------------------------------------------------------------
+# 1. Boundary validation — what an operator may type.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "value,expected",
+    [
+        ("http://10.90.1.4:80", "http://10.90.1.4:80"),
+        ("http://10.90.1.4", "http://10.90.1.4"),
+        ("https://mgmt.internal:8443", "https://mgmt.internal:8443"),
+        # RFC1918 is the NORMAL answer here, unlike utils/ssrf_guard's policy: the
+        # operator is naming their own management host, which on a split deployment
+        # is private by definition.
+        ("http://192.168.1.5:8080", "http://192.168.1.5:8080"),
+        # Empty means "inherit from the next level of the resolution chain".
+        (None, None),
+        ("", None),
+        ("   ", None),
+        # Surrounding whitespace is normalised, not rejected — and the NORMALISED
+        # value is what callers persist, so it can never reach haproxy.cfg.
+        ("  http://10.0.0.5:80  ", "http://10.0.0.5:80"),
+    ],
+)
+def test_accepts_and_normalises_usable_values(value, expected):
+    assert validate_acme_backend_url(value) == expected
+
+
+@pytest.mark.parametrize(
+    "value,code",
+    [
+        # Scheme-less values used to be accepted and then silently became `localhost`
+        # in the renderer — the trap that makes a correct diagnosis un-actionable.
+        ("10.90.1.4:8080", "no_scheme"),
+        ("localhost:8080", "no_scheme"),
+        ("ftp://10.0.0.5", "bad_scheme"),
+        # A newline would inject directives into a file pushed to every node.
+        ("http://10.90.1.4\nbind :9", "whitespace"),
+        ("http://10.90.1.4 x", "whitespace"),
+        # Loopback by number AND by name: `localhost` is what both shipped defaults
+        # contain, so catching only the numeric form would miss the common case.
+        ("http://localhost:8080", "loopback"),
+        ("http://LOCALHOST", "loopback"),
+        ("http://127.0.0.1", "loopback"),
+        ("http://[::1]:80", "loopback"),
+        ("http://0.0.0.0:80", "unspecified"),
+        ("http://169.254.169.254", "link_local"),
+        ("http://u:p@10.0.0.5", "userinfo"),
+        ("http://10.0.0.5/api", "has_path"),
+        ("http://10.0.0.5?x=1", "has_path"),
+        # urlparse defers port parsing to attribute access; unguarded this raises
+        # inside the config generator and destroys the cluster's whole config.
+        ("http://10.0.0.5:99999", "bad_port"),
+        ("http://10.0.0.5:abc", "bad_port"),
+        ("http://-bad-.com", "invalid_host"),
+    ],
+)
+def test_rejects_unusable_values_with_stable_codes(value, code):
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url(value)
+    assert exc.value.code == code
+    assert str(exc.value), "every rejection must carry operator-facing prose"
+
+
+def test_rejects_values_longer_than_the_column():
+    # VARCHAR(500); without this the write fails as an opaque asyncpg 22001 -> 500.
+    with pytest.raises(AcmeBackendUrlError) as exc:
+        validate_acme_backend_url("http://" + "a" * 600 + ".com")
+    assert exc.value.code == "too_long"
+
+
+# ---------------------------------------------------------------------------
+# 2. Render-time resolution — never rejects, never raises.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "url,host,port,ssl_flag",
+    [
+        ("http://10.90.1.4:80", "10.90.1.4", 80, ""),
+        # Port-less http stays 8080, NOT the scheme default 80: the bundled compose
+        # publishes nginx on 8080, so installs relying on this have a working path
+        # today and changing it would break them silently in the renewal loop.
+        ("http://10.90.1.4", "10.90.1.4", 8080, ""),
+        ("https://m.io", "m.io", 443, " ssl verify none"),
+        ("http://localhost:8080", "localhost", 8080, ""),
+    ],
+)
+def test_resolution_preserves_existing_rendering(url, host, port, ssl_flag):
+    target = resolve_acme_backend_target(url)
+    assert (target.host, target.port, target.ssl_flag) == (host, port, ssl_flag)
+    assert target.error_code is None
+
+
+@pytest.mark.parametrize(
+    "url",
+    ["", None, "   ", "10.0.0.5:80", "http://h:99999", "http://10.0.0.5\nx", "http://ba d"],
+)
+def test_resolution_reports_instead_of_raising(url):
+    target = resolve_acme_backend_target(url)
+    assert target.error_code, "unusable values must be reported, not raised"
+    assert target.error_message
+
+
+def test_resolution_warns_on_loopback_rather_than_refusing():
+    # Refusing here would make every existing install unappliable: the shipped
+    # defaults ARE loopback, and the failure would block changes unrelated to ACME.
+    target = resolve_acme_backend_target("http://localhost:8080")
+    assert target.error_code is None
+    assert target.warnings and "Loopback" in target.warnings[0]
+
+
+def test_resolution_warns_when_the_port_is_omitted():
+    target = resolve_acme_backend_target("http://10.0.0.5")
+    assert target.port == 8080
+    assert any("port" in w.lower() for w in target.warnings)
+
+
+# ---------------------------------------------------------------------------
+# 3. Change detection — what makes a panel edit actually reach the nodes.
+# ---------------------------------------------------------------------------
+
+
+_CONFIG = """global
+    daemon
+
+frontend fe_http
+    bind 10.90.1.100:80
+    mode http
+    acl is_acme_challenge path_beg /.well-known/acme-challenge/
+    use_backend _acme_challenge_backend if is_acme_challenge
+    default_backend app
+
+backend app
+    server s1 10.0.0.9:8080
+
+# ACME Challenge Backend (auto-managed by HAProxy OpenManager)
+backend _acme_challenge_backend
+    mode http
+    server _acme_mgmt 10.90.1.4:80
+"""
+
+
+def test_extracts_the_challenge_backend_target():
+    assert extract_acme_backend_target(_CONFIG) == "10.90.1.4:80"
+
+
+def test_extracts_target_with_ssl_flag():
+    cfg = _CONFIG.replace("10.90.1.4:80", "m.io:443 ssl verify none")
+    assert extract_acme_backend_target(cfg) == "m.io:443 ssl verify none"
+
+
+def test_ignores_server_lines_in_other_backends():
+    # Comparing the whole config would flag every unrelated pending edit as a change;
+    # this must key on the ACME section alone.
+    cfg = _CONFIG.replace("backend _acme_challenge_backend", "backend something_else")
+    assert extract_acme_backend_target(cfg) is None
+
+
+def test_returns_none_when_the_section_has_no_server_line():
+    cfg = (
+        "backend _acme_challenge_backend\n"
+        "    mode http\n"
+        "    # ACME challenge backend unavailable (loopback)\n"
+    )
+    assert extract_acme_backend_target(cfg) is None
+
+
+@pytest.mark.parametrize("value", ["", None])
+def test_extraction_tolerates_empty_input(value):
+    assert extract_acme_backend_target(value) is None
+
+
+def test_url_change_that_renders_the_same_target_is_not_a_change():
+    # `http://10.90.1.4` and `http://10.90.1.4:8080` are different strings but the
+    # same shipped address; minting a config version for that would put a no-op
+    # pending change in front of the operator.
+    a = resolve_acme_backend_target("http://10.90.1.4")
+    b = resolve_acme_backend_target("http://10.90.1.4:8080")
+    assert (a.host, a.port, a.ssl_flag) == (b.host, b.port, b.ssl_flag)
+
+
+# ---------------------------------------------------------------------------
+# 4. Source selection — an unusable value must not shadow a usable one.
+# ---------------------------------------------------------------------------
+
+
+def test_prefers_the_most_specific_usable_source():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "http://10.0.0.1:80"),
+        ("settings", "http://10.0.0.2:80"),
+        ("env", "http://10.0.0.3:80"),
+    ])
+    assert (source, url, target.host) == ("cluster", "http://10.0.0.1:80", "10.0.0.1")
+    assert skipped == []
+
+
+def test_skips_an_unusable_value_and_uses_the_next_source():
+    # The regression this guards: a scheme-less value left over from the era when the
+    # settings field was free text resolves to nothing. Stopping there would emit a
+    # backend section with no `server` line — `haproxy -c` passes, Apply succeeds, and
+    # every challenge request 503s with no visible cause.
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.90.1.4:8080"),
+        ("settings", ""),
+        ("env", "http://10.90.1.4:8080"),
+    ])
+    assert source == "env"
+    assert target.error_code is None and target.host == "10.90.1.4"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_empty_sources_are_skipped_without_being_reported():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""),
+        ("settings", None),
+        ("env", "http://10.0.0.9:80"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert skipped == []
+
+
+def test_reports_the_last_attempted_value_when_nothing_resolves():
+    source, url, target, skipped = select_acme_backend_source([
+        ("cluster", "10.0.0.1:80"),
+        ("env", "not a url"),
+    ])
+    assert (source, url) == ("env", "not a url")
+    assert target.error_code, "the caller needs an error to render and log"
+    assert [s[0] for s in skipped] == ["cluster"]
+
+
+def test_all_sources_empty_yields_an_error_not_a_crash():
+    source, _url, target, skipped = select_acme_backend_source([
+        ("cluster", ""), ("settings", ""), ("env", ""),
+    ])
+    assert source == "cluster" and target.error_code == "empty" and skipped == []
+
+
+def test_loopback_is_usable_enough_to_render():
+    # Warned about, never skipped: the shipped defaults are loopback, so treating it as
+    # unusable would make the fallback chain fall off its own end on a stock install.
+    source, _url, target, _skipped = select_acme_backend_source([
+        ("env", "http://localhost:8080"),
+    ])
+    assert source == "env" and target.error_code is None
+    assert target.warnings
+
+
+# ---------------------------------------------------------------------------
+# 5. The generator's failure sentinel must never be mistaken for a config.
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "content",
+    [
+        "# Error generating configuration: boom",
+        "# Error: Cluster not found",
+        "",
+        None,
+    ],
+)
+def test_detects_generation_failure_sentinels(content):
+    assert is_config_generation_error(content) is True
+
+
+@pytest.mark.parametrize("content", [_CONFIG, "global\n    daemon\n"])
+def test_real_configs_are_not_flagged(content):
+    assert is_config_generation_error(content) is False

backend/tests/test_acme_diagnostics.py

@@ -104,13 +104,30 @@ async def test_check_dns_empty_ips_marks_failure(monkeypatch):
 
 
 # ----------------------------------------------------------------------------
-# Port-80 check (HEAD probe)
+# Port-80 check (GET probe)
+#
+# The probe is a GET, not a HEAD: a reverse proxy that has lost its
+# /.well-known/acme-challenge/ location falls through to its catch-all and serves
+# an SPA with HTTP 200, which a status-code-only check accepts as healthy while
+# every real validation fails. The fakes below therefore carry a body.
 # ----------------------------------------------------------------------------
 
 
-class _FakeHEADResp:
-    def __init__(self, status):
+class _FakeContent:
+    def __init__(self, body):
+        self._body = body
+
+    async def read(self, n=-1):
+        if self._body is None:
+            raise ConnectionResetError("reset mid-body")
+        return self._body if n is None or n < 0 else self._body[:n]
+
+
+class _FakeGETResp:
+    def __init__(self, status, body=b"", content_type="text/plain"):
         self.status = status
+        self.headers = {"content-type": content_type}
+        self.content = _FakeContent(body)
 
     async def __aenter__(self):
         return self
@@ -120,10 +137,13 @@ class _FakeHEADResp:
 
 
 class _FakeSession:
-    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False):
+    def __init__(self, *, statuses=None, raise_timeout=False, raise_client_error=False,
+                 bodies=None, content_types=None):
         self._statuses = list(statuses or [])
         self._raise_timeout = raise_timeout
         self._raise_client_error = raise_client_error
+        self._bodies = list(bodies or [])
+        self._content_types = list(content_types or [])
 
     async def __aenter__(self):
         return self
@@ -131,14 +151,16 @@ class _FakeSession:
     async def __aexit__(self, *args):
         return False
 
-    def head(self, url, allow_redirects=False):
+    def get(self, url, allow_redirects=False):
         if self._raise_timeout:
             raise asyncio.TimeoutError()
         if self._raise_client_error:
             import aiohttp
             raise aiohttp.ClientError("connection refused")
         status = self._statuses.pop(0) if self._statuses else 200
-        return _FakeHEADResp(status)
+        body = self._bodies.pop(0) if self._bodies else b""
+        ctype = self._content_types.pop(0) if self._content_types else "text/plain"
+        return _FakeGETResp(status, body, ctype)
 
 
 def _mock_public_dns(monkeypatch, ip="93.184.216.34"):
@@ -178,6 +200,65 @@ async def test_check_port80_ok_on_404(monkeypatch):
     assert out["status"] == "ok"
 
 
+@pytest.mark.asyncio
+async def test_check_port80_warns_when_200_carries_a_web_page(monkeypatch):
+    """The failure that motivated the GET probe.
+
+    A reverse proxy whose /.well-known/acme-challenge/ location has drifted away
+    falls through to its catch-all and serves the SPA. The status is 200, so the old
+    `status in (200, 404)` rule called the install healthy while every validation
+    failed. Only the body distinguishes them.
+    """
+    _mock_public_dns(monkeypatch)
+    spa = b'<!doctype html><html><head><title>HAProxy OpenManager</title></head>'
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200], bodies=[spa], content_types=["text/html"])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    target = out["details"]["targets"][0]
+    assert target["body_class"] == "html"
+    assert not target.get("ok")
+    assert "HTML" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_port80_falls_back_to_status_when_the_body_cannot_be_read(monkeypatch):
+    """A body that cannot be read is missing evidence, not a verdict.
+
+    Turning a connection reset mid-response into a hard failure would make a healthy
+    404 fail intermittently, so the check keeps its original status-only semantics
+    whenever there is nothing to judge.
+    """
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[404], bodies=[None])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "ok"
+    assert out["details"]["targets"][0]["body_class"] == "unread"
+
+
+@pytest.mark.asyncio
+async def test_check_port80_warns_on_a_redirect(monkeypatch):
+    _mock_public_dns(monkeypatch)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[301])
+
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    out = await check_port80(["a.example.com"])
+    assert out["status"] == "warn"
+    assert out["details"]["targets"][0]["diagnosis"]
+
+
 @pytest.mark.asyncio
 async def test_check_port80_warn_on_egress_timeout(monkeypatch):
     """Corporate egress blocks port 80 outbound — warn, don't fail."""
@@ -337,16 +418,92 @@ async def test_check_routing_fail_when_no_port80_frontend():
     assert "No HTTP frontend" in out["message"]
 
 
+def _routing_row(**over):
+    row = {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
+           "mode": "http", "default_backend": "be", "cluster_id": 1, "acme_enabled": True}
+    row.update(over)
+    return row
+
+
+def _applied(has_route=True, server_line="server _acme_mgmt 10.90.1.4:80"):
+    return {"has_route": has_route, "server_line": server_line}
+
+
 @pytest.mark.asyncio
-async def test_check_routing_ok_when_port80_frontend_present():
+async def test_check_routing_ok_when_challenge_route_is_in_the_applied_config():
+    # A port-80 frontend row alone is NOT enough. It describes what the database
+    # wants; the nodes run whatever was last applied. During the incident this
+    # function reported "ok" from the row count while the live config had no usable
+    # challenge route at all.
     conn = AsyncMock()
-    conn.fetch.return_value = [
-        {"id": 1, "name": "fe-http", "bind_address": "0.0.0.0", "bind_port": 80,
-         "mode": "http", "default_backend": "be"},
-    ]
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied()
     out = await check_routing(conn, ["a.example.com"], [1])
     assert out["status"] == "ok"
     assert len(out["details"]["frontends"]) == 1
+    assert out["details"]["challenge_backends"] == {1: "10.90.1.4:80"}
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_acme_is_disabled_on_the_cluster():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(acme_enabled=False)]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "disabled" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_route_is_not_in_the_applied_config():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(has_route=False)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "apply the cluster" in out["message"].lower()
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_has_no_server_line():
+    # `haproxy -c` passes because the section exists, so nothing else catches this;
+    # every challenge request 503s from an empty backend.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line=None)
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "503" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_when_the_challenge_backend_is_loopback():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row()]
+    conn.fetchrow.return_value = _applied(server_line="server _acme_mgmt 127.0.0.1:8080")
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "HAProxy node" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_warns_rather_than_fails_on_a_tcp_only_port80_cluster():
+    # The `fail` branch must stay reachable only when NO port-80 frontend exists at
+    # all: SiteWizard blocks submit on any failing check, so turning this into a
+    # failure would lock tcp-only installs the day it ships.
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode="tcp")]
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "warn"
+    assert "tcp mode" in out["message"]
+
+
+@pytest.mark.asyncio
+async def test_check_routing_treats_null_mode_as_http_like_the_renderer():
+    conn = AsyncMock()
+    conn.fetch.return_value = [_routing_row(mode=None)]
+    conn.fetchrow.return_value = _applied()
+    out = await check_routing(conn, ["a.example.com"], [1])
+    assert out["status"] == "ok"
 
 
 # ----------------------------------------------------------------------------
@@ -433,7 +590,7 @@ async def test_check_agents_fail_when_none_registered():
 async def test_check_agents_warn_when_none_active():
     conn = AsyncMock()
     conn.fetch.return_value = [
-        {"id": 1, "hostname": "h1", "status": "offline", "last_heartbeat": None,
+        {"id": 1, "hostname": "h1", "status": "offline", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
     ]
     out = await check_agents(conn, [1])
@@ -444,9 +601,9 @@ async def test_check_agents_warn_when_none_active():
 async def test_check_agents_ok_with_active():
     conn = AsyncMock()
     conn.fetch.return_value = [
-        {"id": 1, "hostname": "h1", "status": "active", "last_heartbeat": None,
+        {"id": 1, "hostname": "h1", "status": "active", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
-        {"id": 2, "hostname": "h2", "status": "offline", "last_heartbeat": None,
+        {"id": 2, "hostname": "h2", "status": "offline", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
     ]
     out = await check_agents(conn, [1])
@@ -454,6 +611,49 @@ async def test_check_agents_ok_with_active():
     assert "1 of 2" in out["message"]
 
 
+@pytest.mark.asyncio
+async def test_bulgu84_check_agents_uses_last_seen_column():
+    """Bulgu #84 (round-23 audit) — pin the SQL column name.
+
+    Pre-fix the query referenced a non-existent ``a.last_heartbeat``
+    column. AsyncMock returns whatever dict the test sets without
+    re-validating the SQL string, so the pre-fix test suite was
+    GREEN while every live ACME preflight call (the
+    ``POST /api/sites/preflight-acme`` endpoint that the wizard
+    runs before showing the ACME step) crashed with
+    ``UndefinedColumnError: column a.last_heartbeat does not
+    exist``. The crash blocked the entire wizard ACME preview
+    page on a production cluster, but unit tests never noticed
+    because they only assert on the helper's return shape, not
+    on the SQL string the helper sends to Postgres.
+
+    This pin inspects ``conn.fetch.call_args`` to assert the SQL
+    body actually queries ``a.last_seen`` (the canonical column
+    name used everywhere else in the codebase — see
+    routers/cluster.py:695-702 and routers/agent.py:281+ for
+    sibling readers). A future ``last_heartbeat`` typo would
+    re-fail this test without anyone noticing the live impact.
+    """
+    conn = AsyncMock()
+    conn.fetch.return_value = [
+        {"id": 1, "hostname": "h1", "status": "active", "last_seen": None,
+         "cluster_id": 1, "cluster_name": "c1"},
+    ]
+    await check_agents(conn, [1])
+    conn.fetch.assert_awaited_once()
+    sql_query = conn.fetch.call_args[0][0]
+    assert "a.last_seen" in sql_query, (
+        f"check_agents SQL must select a.last_seen (the canonical "
+        f"agents-table timestamp column); got: {sql_query!r}"
+    )
+    assert "a.last_heartbeat" not in sql_query, (
+        f"check_agents SQL still references a.last_heartbeat — this "
+        f"column does NOT exist on the agents table and the query "
+        f"will 500 with UndefinedColumnError at runtime. SQL: "
+        f"{sql_query!r}"
+    )
+
+
 # ----------------------------------------------------------------------------
 # run_checks orchestration
 # ----------------------------------------------------------------------------
@@ -506,3 +706,180 @@ async def test_run_checks_unknown_only_returns_empty():
         account_id=None, only=["bogus"],
     )
     assert out == []
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #94 / #95 (Round-25 audit) — diagnostic-runner robustness
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_swallows_single_check_crash(monkeypatch):
+    """Bulgu #94 — a single check exception must NOT collapse the suite.
+
+    Pre-fix, an asyncpg UndefinedColumnError from check_agents (e.g. the
+    Bulgu #84 ``a.last_heartbeat`` typo on an older deploy) propagated
+    up to the FastAPI router which had no `except`, so the operator saw
+    HTTP 500 with no body. The diagnostic panel is precisely the place
+    that should SURFACE this — never opaque-500 it. We now wrap each
+    check; the failing one becomes a structured `fail` row and the
+    other four still render.
+    """
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+
+    # check_routing + check_agents both call conn.fetch; explode on the
+    # FIRST call (which is check_routing) and return rows on the second.
+    call_count = {"n": 0}
+
+    async def _fetch(*args, **kwargs):
+        call_count["n"] += 1
+        if call_count["n"] == 1:
+            raise RuntimeError("simulated: column a.last_heartbeat does not exist")
+        return []
+
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # All five checks must still appear in the response.
+    ids = [c["id"] for c in out]
+    assert ids == ["dns", "port80", "routing", "account", "agents"]
+    routing = next(c for c in out if c["id"] == "routing")
+    assert routing["status"] == "fail"
+    assert "Diagnostic check crashed" in routing["message"]
+    assert routing["details"]["exception_type"] == "RuntimeError"
+    assert "last_heartbeat" in routing["details"]["exception_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_coerces_string_cluster_ids(monkeypatch):
+    """Bulgu #94 — cluster_ids stored as JSONB strings (legacy paths)
+    must not crash check_routing / check_agents with
+    ``invalid input syntax for type integer: "1"``."""
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    captured_args = []
+
+    async def _fetch(*args, **kwargs):
+        captured_args.append(args)
+        return []
+
+    conn = AsyncMock()
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=["1", "2", "garbage", None, 3],
+        account_id=None,
+    )
+    # The list passed to asyncpg should already be a pure-int list.
+    # check_routing is the first call that uses cluster_ids.
+    routing_call_args = [a for a in captured_args if "frontends" in a[0]]
+    assert routing_call_args, "check_routing should have queried frontends"
+    cluster_ids_arg = routing_call_args[0][1]
+    assert cluster_ids_arg == [1, 2, 3], (
+        f"cluster_ids must be coerced to ints before being passed to "
+        f"asyncpg's ::int[] cast; got: {cluster_ids_arg!r}"
+    )
+    assert all(c["status"] != "fail" or c["id"] != "routing"
+               for c in out
+               if c["id"] == "routing" and "Diagnostic check crashed" in (c.get("message") or "")
+              ), "routing should not have crashed on coerced cluster_ids"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_does_not_swallow_cancellation(monkeypatch):
+    """`_safe_check` must catch `Exception` but NOT `BaseException`.
+
+    asyncio.CancelledError is a BaseException (Python 3.8+) so it must
+    propagate out of `_safe_check` — otherwise a request that the
+    client cancelled mid-flight would silently keep running diagnostic
+    checks instead of unwinding cleanly. We hand `_safe_check` a coro
+    that raises CancelledError and assert it bubbles up.
+    """
+    from services.acme_diagnostics import _safe_check
+
+    async def _cancelled_coro():
+        raise asyncio.CancelledError()
+
+    with pytest.raises(asyncio.CancelledError):
+        await _safe_check("dns", "DNS resolution", _cancelled_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_handles_keyboardinterrupt(monkeypatch):
+    """`_safe_check` must also not swallow `KeyboardInterrupt`."""
+    from services.acme_diagnostics import _safe_check
+
+    async def _interrupt_coro():
+        raise KeyboardInterrupt()
+
+    with pytest.raises(KeyboardInterrupt):
+        await _safe_check("dns", "DNS resolution", _interrupt_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_coerce_cluster_ids_handles_none():
+    """Coercion must handle None input without raising."""
+    from services.acme_diagnostics import _coerce_cluster_ids
+    assert _coerce_cluster_ids(None) == []
+    assert _coerce_cluster_ids([]) == []
+    assert _coerce_cluster_ids([1, 2, 3]) == [1, 2, 3]
+    assert _coerce_cluster_ids(["1", "2"]) == [1, 2]
+    assert _coerce_cluster_ids([1.5]) == [1]  # int() truncates floats
+    assert _coerce_cluster_ids(["abc", None, "5"]) == [5]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_filters_invalid_domains(monkeypatch):
+    """Non-string entries in `domains` must not reach the DNS resolver."""
+    seen_domains = []
+
+    def fake_gethostbyname_ex(domain):
+        seen_domains.append(domain)
+        return (domain, [], ["10.0.0.1"])
+
+    monkeypatch.setattr(socket, "gethostbyname_ex", fake_gethostbyname_ex)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+    conn.fetch.return_value = []
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com", None, "", 42, "b.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # Both check_dns and check_port80 resolve DNS, so each valid domain
+    # may appear multiple times — but invalid entries (None, "", 42)
+    # must never reach the resolver.
+    assert set(seen_domains) == {"a.example.com", "b.example.com"}
+    assert None not in seen_domains
+    assert "" not in seen_domains
+    assert 42 not in seen_domains
+    dns_check = next(c for c in out if c["id"] == "dns")
+    assert dns_check["status"] == "ok"

backend/tests/test_acme_diagnostics_router_round25.py

@@ -0,0 +1,554 @@
+"""Router-level tests for the ACME diagnostic endpoints (Round-25 audit).
+
+These tests pin the contract introduced by Bulgu #94 / #95:
+
+* ``POST /api/letsencrypt/orders/{order_id}/diagnostics`` must NEVER return
+  HTTP 500 for an in-suite failure. Authentication / authorisation /
+  rate-limit / not-found errors still raise the appropriate 4xx, but any
+  unexpected exception during check execution is converted to HTTP 200
+  with a structured failure envelope so the UI can render the cause.
+
+* ``GET /api/letsencrypt/orders/{order_id}/events`` must NEVER return
+  HTTP 500 because of schema drift in ``user_activity_logs`` (the
+  original 500 cause: SELECTing a non-existent ``status`` column). A
+  partial failure is reported via ``meta.errors[]``.
+
+The tests use AsyncMock-based fake connections rather than spinning up a
+real Postgres so they run hermetically inside CI.
+"""
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from routers import acme_diagnostics as router_mod
+
+
+# ----------------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------------
+
+
+class _FakeUser(dict):
+    pass
+
+
+def _patch_auth_and_db(monkeypatch, conn, user_id=1):
+    """Patch the auth / db helpers used by both endpoints so the tests
+    don't have to construct a real FastAPI request stack."""
+    async def _fake_user(_auth):
+        return _FakeUser(id=user_id, username="t", email="t@x")
+
+    async def _fake_perm(_uid, *_a, **_k):
+        return True
+
+    async def _fake_get_conn():
+        return conn
+
+    async def _fake_close_conn(_c):
+        return None
+
+    async def _fake_rate_limit(*_a, **_kw):
+        return None
+
+    monkeypatch.setattr(router_mod, "get_current_user_from_token", _fake_user)
+    monkeypatch.setattr(router_mod, "check_user_permission", _fake_perm)
+    monkeypatch.setattr(router_mod, "get_database_connection", _fake_get_conn)
+    monkeypatch.setattr(router_mod, "close_database_connection", _fake_close_conn)
+    monkeypatch.setattr(router_mod, "_enforce_rate_limit", _fake_rate_limit)
+
+
+# ----------------------------------------------------------------------------
+# /diagnostics endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_load_order_crash_returns_envelope_not_500(monkeypatch):
+    """A DB crash during load_order must surface a 200 envelope, not 500.
+
+    Pre-fix, any RuntimeError between auth and run_checks bubbled out of
+    the bare ``try/finally`` block and FastAPI returned an opaque HTTP 500.
+    The Round-25 fix wraps load_order so the operator sees the cause
+    inside the diagnostic panel.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB connectivity loss")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.run_diagnostics(order_id=42, authorization="Bearer x")
+
+    assert out["order_id"] == 42
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["checks"][0]["status"] == "fail"
+    assert out["checks"][0]["id"] == "diagnostics_runner"
+    assert "simulated DB connectivity loss" in out["checks"][0]["message"]
+    assert out["meta"]["correlation_id"]
+    assert out["meta"]["error_stage"] == "load_order"
+    assert "humanized_error" in out
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_run_checks_crash_returns_envelope(monkeypatch):
+    """A crash inside run_checks (after order is loaded) must also stay 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated check orchestrator crash")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["meta"]["error_stage"] == "run_checks"
+    assert out["meta"]["error_type"] == "RuntimeError"
+    assert "simulated check orchestrator crash" in out["meta"]["error_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_returns_meta_summary_on_success(monkeypatch):
+    """Successful diagnostics responses carry a meta summary the UI uses
+    to surface 'N checks failed, M warnings' without recomputing."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _fake_checks(*_a, **_kw):
+        return [
+            {"id": "dns", "label": "DNS", "status": "ok", "severity": "info", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "routing", "label": "Routing", "status": "fail", "severity": "error", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "port80", "label": "Port 80", "status": "warn", "severity": "warn", "message": "", "details": {}, "duration_ms": 1},
+        ]
+
+    monkeypatch.setattr(router_mod, "run_checks", _fake_checks)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["meta"]["checks_total"] == 3
+    assert out["meta"]["checks_failed"] == 1
+    assert out["meta"]["checks_warn"] == 1
+    assert out["meta"]["correlation_id"]
+
+
+# ----------------------------------------------------------------------------
+# /events endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_missing_status_column_returns_partial_envelope(monkeypatch):
+    """Bulgu #95 — user_activity_logs lacks a `status` column.
+
+    Pre-fix, the SELECT pulled `status` directly and the endpoint
+    returned HTTP 500 for every order that had any correlated
+    user-activity rows. The Round-25 fix introspects the schema; here
+    we simulate a deployment with no `status` column AND a JOIN /
+    query that would otherwise crash — the endpoint must stay 200,
+    return whatever it could collect from acme_order_events, and
+    record the user_activity_logs section as degraded but recoverable.
+    """
+    conn = AsyncMock()
+
+    # _load_order
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    fetchval_calls = {"n": 0}
+
+    async def _fetchval(sql, *args):
+        fetchval_calls["n"] += 1
+        # First call: existence check for acme_order_events table
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    columns_no_status = [
+        {"column_name": "id"},
+        {"column_name": "user_id"},
+        {"column_name": "action"},
+        {"column_name": "resource_type"},
+        {"column_name": "resource_id"},
+        {"column_name": "details"},
+        {"column_name": "created_at"},
+        # NOTE: no "status" column — this is the canonical schema.
+    ]
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql and "user_activity_logs" in sql:
+            return columns_no_status
+        if "FROM acme_order_events" in sql:
+            return []  # empty timeline is fine for this test
+        if "FROM user_activity_logs" in sql:
+            # If the projection includes `status` we will fail loudly.
+            assert "status" not in sql, (
+                "SELECT must not include `status` when the column is absent; "
+                f"SQL was: {sql!r}"
+            )
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["count"] == 0
+    assert out["meta"]["correlation_id"]
+    # No section errors expected — schema-aware projection silently
+    # adapted, the panel just got an empty event list.
+    assert out["meta"]["errors"] == []
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_acme_order_events_query_crash_returns_partial(monkeypatch):
+    """A crash in the acme_order_events sub-query must NOT kill the
+    whole endpoint — the user_activity_logs section should still run
+    and the failure must appear in meta.errors."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+            ]
+        if "FROM acme_order_events" in sql:
+            raise RuntimeError("simulated acme_order_events index corruption")
+        if "FROM user_activity_logs" in sql:
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["count"] == 0
+    error_sections = [e["section"] for e in out["meta"]["errors"]]
+    assert "acme_order_events" in error_sections
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_load_order_404_still_raises(monkeypatch):
+    """The 404 HTTPException raised by `_load_order` for an unknown order
+    must remain a 404 — the Round-25 envelope is only for *unexpected*
+    failures, not for client-supplied invalid order IDs."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    conn.fetchrow.return_value = None  # no order found
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.get_order_events(order_id=9999, authorization="Bearer x")
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_setup_crash_returns_check_envelope(monkeypatch):
+    """The single-check re-run path must also envelope, never 500.
+
+    Pre-fix the rerun handler used the same bare ``try/finally`` shape
+    as the suite POST. If `_load_order` / `_enforce_rate_limit` raised,
+    the operator clicking the row's "Re-run" button got an opaque
+    toast and the row never updated. Now the rerun handler returns a
+    `fail` check shaped the same way the table already renders, so
+    the row updates in place with the cause + correlation_id.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB drop during rerun")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="dns", authorization="Bearer x",
+    )
+    assert out["order_id"] == 5
+    assert out["check"]["status"] == "fail"
+    assert out["check"]["id"] == "dns"
+    assert "simulated DB drop during rerun" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "setup"
+    assert out["meta"]["correlation_id"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_run_checks_crash_returns_check_envelope(monkeypatch):
+    """A crash inside run_checks during a re-run also stays 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated run_checks failure")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="agents", authorization="Bearer x",
+    )
+    assert out["check"]["id"] == "agents"
+    assert out["check"]["status"] == "fail"
+    assert "simulated run_checks failure" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "run_checks"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_invalid_check_id_still_400(monkeypatch):
+    """An unknown check_id must remain a 400, not an envelope. The
+    envelope is only for *unexpected* server-side failures, not for
+    client-supplied invalid identifiers."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5, check_id="bogus", authorization="Bearer x",
+        )
+    assert exc_info.value.status_code == 400
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_status_column_is_used_when_present(monkeypatch):
+    """If a deployment DID add a `status` column (e.g. via a private
+    schema extension), the projection picks it up and the resulting
+    severity reflects it."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    captured_sql = {"ua": None}
+    from datetime import datetime, timezone
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+                {"column_name": "status"},
+            ]
+        if "FROM acme_order_events" in sql:
+            return []
+        if "FROM user_activity_logs" in sql:
+            captured_sql["ua"] = sql
+            return [{
+                "id": 100,
+                "action": "letsencrypt.order.create",
+                "resource_type": "letsencrypt_order",
+                "resource_id": "5",
+                "details": '{"foo":"bar"}',
+                "created_at": datetime(2026, 5, 13, 19, 0, 0, tzinfo=timezone.utc),
+                "user_id": 7,
+                "status": "failure",
+            }]
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert "status" in captured_sql["ua"]
+    assert out["count"] == 1
+    ev = out["events"][0]
+    assert ev["source"] == "user_activity_log"
+    assert ev["severity"] == "warn"  # status="failure" → warn
+    assert ev["details"] == {"foo": "bar"}
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #96 (prod-canary follow-up): int4 overflow on order_id must NOT
+# leak the asyncpg DataError message ("invalid input for query argument
+# $1: ... value out of int32 range") into the operator-facing response
+# body. Same shape as the "row not found" path: clean HTTPException(404).
+# ----------------------------------------------------------------------------
+
+
+import asyncpg as _asyncpg  # noqa: E402 — late import so the test module
+# can still be collected even if asyncpg has changed its exception module.
+
+
+def _data_error(msg: str) -> _asyncpg.exceptions.DataError:
+    """Construct an asyncpg DataError that mirrors what Postgres returns
+    when a path-param order_id overflows int4. We can't easily build the
+    real instance from the binary protocol, so we synthesize one with the
+    same class so the router's `except asyncpg.exceptions.DataError`
+    branch is exercised."""
+    return _asyncpg.exceptions.DataError(msg)
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_diagnostics_int4_overflow_returns_clean_404(monkeypatch):
+    """``order_id`` outside the int4 range must surface as a clean 404,
+    NOT as a `diagnostics_unavailable` envelope leaking the asyncpg
+    DataError message ("query argument $1", "int32 range").
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 2147483648 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(
+            order_id=2_147_483_648,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    # The operator must see the canonical "not found" detail, NOT the
+    # raw asyncpg error message.
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+    assert "query argument" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_events_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the events endpoint — out-of-range order_id is a
+    clean 404, not a `meta.errors[]` envelope leaking SQL detail."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 9999999999 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.get_order_events(
+            order_id=9_999_999_999,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_rerun_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the per-check rerun endpoint — out-of-range
+    order_id is a clean 404, not a structured ``check.fail`` envelope
+    leaking the asyncpg DataError message."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 5000000000 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5_000_000_000,
+            check_id="dns",
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_load_order_dataerror_does_not_leak_correlation_envelope(monkeypatch):
+    """Belt-and-braces: even when the DataError carries other kinds of
+    invalid-input strings (e.g. type coercion failure on an int4
+    column), `_load_order` must still answer with the canonical 404
+    envelope and NOT route it through `_diagnostic_failure_envelope`
+    (which would surface the raw SQL detail to the UI)."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error("invalid integer literal: 'NaN'")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(order_id=123, authorization="Bearer x")
+
+    assert excinfo.value.status_code == 404
+    # No SQL detail leak
+    assert "invalid integer literal" not in str(excinfo.value.detail).lower()
+    assert "NaN" not in str(excinfo.value.detail)

backend/tests/test_acme_humanizer.py

@@ -81,6 +81,7 @@ def test_legacy_plain_string_with_brace_but_invalid_json_falls_back():
     ("urn:ietf:params:acme:error:rejectedIdentifier", "blacklisted",    "rejected"),
     ("urn:ietf:params:acme:error:serverInternal", "internal err",       "ACME server"),
     ("urn:ietf:params:acme:error:userActionRequired", "agree to ToS",   "User action"),
+    ("urn:ietf:params:acme:error:externalAccountRequired", "EAB required", "External Account Binding"),
 ])
 def test_known_problem_types_are_humanized(problem_type, detail_text, expected_title_contains):
     payload = json.dumps({"type": problem_type, "detail": detail_text, "status": 400})

backend/tests/test_acme_pydantic_validation.py

@@ -8,7 +8,31 @@ from pydantic import ValidationError
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from routers.letsencrypt import CertificateRequest
+from routers.letsencrypt import CertificateRequest, AccountCreate
+
+
+class TestAccountCreateEAB:
+    """Issue #35 follow-up: EAB HMAC key must be valid base64url; empty/None passes through
+    (falls back to global Settings) so non-EAB accounts (HTTP-01 / LE / Cloudflare) are unaffected."""
+
+    def test_no_eab_is_allowed(self):
+        acc = AccountCreate(email="a@b.com")
+        assert acc.eab_hmac_key is None and acc.eab_kid is None
+
+    def test_valid_base64url_hmac_accepted(self):
+        # urlsafe base64, unpadded and padded — both accepted.
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZGVmZ2g")
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZA==")
+
+    def test_invalid_base64_hmac_rejected(self):
+        # 5 base64 chars (count ≡ 1 mod 4) is undecodable — the exact shape that would otherwise
+        # make register_account's _b64url_decode raise a cryptic binascii error.
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="AAAAA")
+
+    def test_oversized_hmac_rejected(self):
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="A" * 600)
 
 
 class TestCertificateRequestDomains:

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_csr_import.py

@@ -0,0 +1,469 @@
+"""
+v1.9.0 CSR creation — unit tests for the signed-certificate import flow and
+config-version staging (pattern: test_ssl_service_extraction.py, AsyncMock conn).
+
+Pins the security-relevant invariants:
+- key match is a HARD gate: match=False → 400 before any INSERT, and
+  match=None (unverifiable) → 500, never a lenient pass (we generated the
+  key ourselves — deliberate divergence from create_cert_row's fallback).
+- the new cert row is cluster_id=NULL / last_config_status='PENDING' /
+  source='csr' (PENDING keeps it invisible to agents until Apply).
+- completing the CSR NULLs the private key copy.
+- staging reuses the exact `ssl-{id}-create-{ts}` version-name scheme.
+"""
+import json
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from fastapi import HTTPException
+
+from models.csr import SSLCSRImport
+from services.csr_service import (
+    assert_csr_name_available,
+    import_signed_certificate,
+    insert_csr_row,
+)
+from services.ssl_service import stage_ssl_config_versions
+
+
+_VALID_PARSE = {
+    "primary_domain": "www.example.com",
+    "all_domains": ["www.example.com"],
+    "expiry_date": datetime(2099, 1, 1, tzinfo=timezone.utc),
+    "issuer": "CN=Test CA",
+    "fingerprint": "AA:BB:CC",
+    "status": "valid",
+    "days_until_expiry": 365,
+}
+
+_FAKE_CERT = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+_FAKE_KEY = "-----BEGIN PRIVATE KEY-----\nY\n-----END PRIVATE KEY-----"
+
+
+def _csr_row(**overrides):
+    row = {
+        "id": 5,
+        "name": "csr-www",
+        "common_name": "www.example.com",
+        "subject": "{}",
+        "sans": json.dumps(["www.example.com"]),
+        "key_algorithm": "rsa-2048",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nZ\n-----END CERTIFICATE REQUEST-----",
+        "private_key_pem": _FAKE_KEY,
+        "status": "pending",
+        "ssl_certificate_id": None,
+    }
+    row.update(overrides)
+    return row
+
+
+def _mk_conn():
+    conn = AsyncMock()
+    # asyncpg's conn.transaction() is a SYNC call returning an async CM.
+    conn.transaction = MagicMock()
+    return conn
+
+
+def _import_payload(**overrides):
+    base = dict(
+        certificate_content=_FAKE_CERT,
+        chain_content=None,
+        usage_type="frontend",
+        is_global=False,
+        cluster_ids=[1, 2],
+        name=None,
+    )
+    base.update(overrides)
+    return SSLCSRImport(**base)
+
+
+@contextmanager
+def _patched(match=None, parse=None):
+    """Patch every parser touchpoint of the import path: the function-local
+    imports in csr_service (utils.ssl_parser.*) and the module-level imports
+    in ssl_service._prepare_cert_fields (services.ssl_service.*)."""
+    match_result = match if match is not None else {"match": True}
+    parse_result = dict(parse or _VALID_PARSE)
+    with patch("utils.ssl_parser.verify_certificate_key_match", return_value=match_result), \
+         patch("utils.ssl_parser.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.parse_ssl_certificate", return_value=dict(parse_result)), \
+         patch("services.ssl_service.validate_private_key", return_value=True), \
+         patch("services.ssl_service.validate_certificate_chain", return_value=True):
+        yield
+
+
+# ----------------------------------------------------------------------------
+# import_signed_certificate
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_import_happy_path_inserts_pending_csr_sourced_cert():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]  # FOR UPDATE row, no name clash
+    conn.fetchval.return_value = 42                 # INSERT ... RETURNING id
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert result["reactivated"] is False
+
+    # Concurrency invariants: everything runs inside a transaction and the
+    # CSR row is locked FOR UPDATE (serialises double-import and delete-races).
+    assert conn.transaction.call_count == 1
+    lock_sql = conn.fetchrow.call_args_list[0].args[0]
+    assert "FOR UPDATE" in lock_sql
+
+    insert_sql, *insert_args = conn.fetchval.call_args.args
+    assert "INSERT INTO ssl_certificates" in insert_sql
+    assert "NULL, 'PENDING'" in insert_sql, "cert must stay invisible to agents until Apply"
+    assert "'csr'" in insert_sql, "source column must record the CSR origin"
+    # The stored CSR key — not any request-supplied key — must be persisted.
+    assert _FAKE_KEY in insert_args
+
+    # One junction row per requested cluster.
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert len(junction_calls) == 2
+    assert {c.args[2] for c in junction_calls} == {1, 2}
+
+    # CSR completion must destroy the key copy.
+    completion_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_csrs" in c.args[0]
+    ]
+    assert len(completion_calls) == 1
+    assert "private_key_pem = NULL" in completion_calls[0].args[0]
+    assert "status = 'completed'" in completion_calls[0].args[0]
+    assert completion_calls[0].args[1] == 5   # csr_id
+    assert completion_calls[0].args[2] == 42  # cert_id
+
+
+@pytest.mark.asyncio
+async def test_import_global_creates_zero_junction_rows():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        await import_signed_certificate(
+            conn, 5, _import_payload(is_global=True, cluster_ids=None), user_id=7
+        )
+
+    junction_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "ssl_certificate_clusters" in c.args[0] and "INSERT" in c.args[0]
+    ]
+    assert junction_calls == [], "global cert = zero junction rows (existing convention)"
+
+
+@pytest.mark.asyncio
+async def test_import_key_mismatch_rejected_400_before_any_write():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": False, "reason": "public key mismatch"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "does not match" in exc_info.value.detail
+    assert not conn.fetchval.await_count, "nothing must be inserted on mismatch"
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_unverifiable_key_match_is_hard_error_not_lenient():
+    """match=None means OUR stored key is unreadable — integrity failure,
+    never the lenient pass create_cert_row historically allows."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(match={"match": None, "reason": "key could not be parsed"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 500
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_expired_certificate_rejected_400():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    expired = dict(_VALID_PARSE)
+    expired["status"] = "expired"
+    expired["days_until_expiry"] = -10
+    with _patched(parse=expired):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "expired" in exc_info.value.detail.lower()
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_malformed_certificate_rejected_400_not_500():
+    """A cert with PEM markers but unparseable content (truncated CA response)
+    is OPERATOR INPUT — it must get the manual flow's 400, not the 500 that
+    the strict key-match branch reserves for a corrupt STORED key."""
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row()]
+
+    with _patched(parse={"error": "Could not parse certificate"}):
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "Invalid SSL certificate" in exc_info.value.detail
+    assert not conn.fetchval.await_count
+    assert not conn.execute.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_completed_csr_conflicts_409():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(status="completed", ssl_certificate_id=42)]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 409
+    assert "already completed" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_import_missing_csr_404():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [None]
+
+    with pytest.raises(HTTPException) as exc_info:
+        await import_signed_certificate(conn, 999, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_import_active_name_collision_rejected_with_hint():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 9, "is_active": True}]
+
+    with _patched():
+        with pytest.raises(HTTPException) as exc_info:
+            await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert exc_info.value.status_code == 400
+    assert "already exists" in exc_info.value.detail
+    assert "name" in exc_info.value.detail  # points at the override escape hatch
+    assert not conn.fetchval.await_count
+
+
+@pytest.mark.asyncio
+async def test_import_name_override_is_used_for_the_cert_row():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), None]
+    conn.fetchval.return_value = 42
+
+    with _patched():
+        result = await import_signed_certificate(
+            conn, 5, _import_payload(name="renamed-cert"), user_id=7
+        )
+
+    assert result["certificate_name"] == "renamed-cert"
+    _, *insert_args = conn.fetchval.call_args.args
+    assert "renamed-cert" in insert_args
+    # And the collision check must have run against the override, not csr.name.
+    name_lookup = conn.fetchrow.call_args_list[1]
+    assert name_lookup.args[1] == "renamed-cert"
+
+
+@pytest.mark.asyncio
+async def test_import_reactivates_soft_deleted_name_and_warns():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [_csr_row(), {"id": 77, "is_active": False}]
+
+    with _patched():
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 77
+    assert result["reactivated"] is True
+    assert any("reactivated" in w for w in result["warnings"])
+    assert not conn.fetchval.await_count, "reactivation must UPDATE, not INSERT"
+    update_calls = [
+        c for c in conn.execute.call_args_list
+        if c.args and "UPDATE ssl_certificates" in c.args[0]
+    ]
+    assert len(update_calls) == 1
+    update_sql = update_calls[0].args[0]
+    assert "source = 'csr'" in update_sql
+    # The reactivated row must come back to life invisible to agents until
+    # Apply, with the row itself active again.
+    assert "last_config_status = 'PENDING'" in update_sql
+    assert "is_active = TRUE" in update_sql
+    # Old cluster bindings must be wiped before re-binding to the new scope.
+    junction_deletes = [
+        c for c in conn.execute.call_args_list
+        if c.args and "DELETE FROM ssl_certificate_clusters" in c.args[0]
+    ]
+    assert len(junction_deletes) == 1
+    assert junction_deletes[0].args[1] == 77
+    # …and the importer's requested clusters re-bound via the junction.
+    junction_inserts = [
+        c for c in conn.execute.call_args_list
+        if c.args and "INSERT INTO ssl_certificate_clusters" in c.args[0]
+    ]
+    assert {c.args[2] for c in junction_inserts} == {1, 2}
+
+
+@pytest.mark.asyncio
+async def test_import_san_drift_warns_but_succeeds():
+    conn = _mk_conn()
+    conn.fetchrow.side_effect = [
+        _csr_row(sans=json.dumps(["www.example.com", "api.example.com"])),
+        None,
+    ]
+    conn.fetchval.return_value = 42
+
+    drifted = dict(_VALID_PARSE)
+    drifted["all_domains"] = ["www.example.com", "cdn.example.com"]
+    with _patched(parse=drifted):
+        result = await import_signed_certificate(conn, 5, _import_payload(), user_id=7)
+
+    assert result["certificate_id"] == 42
+    assert any("added" in w and "cdn.example.com" in w for w in result["warnings"])
+    assert any("dropped" in w and "api.example.com" in w for w in result["warnings"])
+
+
+# ----------------------------------------------------------------------------
+# insert_csr_row / assert_csr_name_available
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_active_cert_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [11]  # active cert with the name exists
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "certificate" in exc_info.value.detail.lower()
+
+
+@pytest.mark.asyncio
+async def test_csr_name_taken_by_pending_csr_rejected():
+    conn = _mk_conn()
+    conn.fetchval.side_effect = [None, 12]  # no cert, but a pending CSR
+
+    with pytest.raises(HTTPException) as exc_info:
+        await assert_csr_name_available(conn, "taken")
+    assert exc_info.value.status_code == 400
+    assert "pending CSR" in exc_info.value.detail
+
+
+@pytest.mark.asyncio
+async def test_insert_csr_row_translates_unique_violation_to_400():
+    """The uq_ssl_csrs_name_pending partial index closes the create/create
+    race — the loser must get a clean 400, not a 500."""
+    import asyncpg as _asyncpg
+
+    conn = _mk_conn()
+    # availability checks pass, INSERT hits the unique index
+    conn.fetchval.side_effect = [
+        None, None, _asyncpg.exceptions.UniqueViolationError("dup"),
+    ]
+    payload = SimpleNamespace(
+        name="raced", common_name="www.example.com", key_algorithm="rsa-2048"
+    )
+    bundle = {"subject": {}, "sans": ["www.example.com"], "csr_pem": "PEM", "private_key_pem": "KEY"}
+
+    with pytest.raises(HTTPException) as exc_info:
+        await insert_csr_row(conn, payload, bundle, user_id=1)
+    assert exc_info.value.status_code == 400
+    assert "concurrent" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# router-level guards
+# ----------------------------------------------------------------------------
+
+
+def test_cluster_id_int32_guard_rejects_out_of_range_with_404():
+    """Body-supplied cluster ids must never reach asyncpg out of int4 range
+    (DataError → raw 500) — same Bulgu #96 hygiene as the csr_id path param."""
+    from routers.csr import _assert_valid_cluster_id
+
+    _assert_valid_cluster_id(1)
+    _assert_valid_cluster_id(2_147_483_647)
+    for bad in (0, -1, 2_147_483_648, 99_999_999_999):
+        with pytest.raises(HTTPException) as exc_info:
+            _assert_valid_cluster_id(bad)
+        assert exc_info.value.status_code == 404
+        assert "Cluster not found" in exc_info.value.detail
+
+
+# ----------------------------------------------------------------------------
+# stage_ssl_config_versions
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_stage_creates_one_pending_version_per_cluster_with_ssl_naming():
+    import re
+
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001  # config_versions INSERT RETURNING id
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(return_value="# cfg"),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert all(r["success"] for r in results)
+    assert [r["cluster_id"] for r in results] == [1, 2]
+
+    insert_calls = [
+        c for c in conn.fetchval.call_args_list
+        if c.args and "INSERT INTO config_versions" in c.args[0]
+    ]
+    assert len(insert_calls) == 2
+    for call in insert_calls:
+        sql = call.args[0]
+        assert "FALSE, 'PENDING'" in sql, "staged versions must be inactive + PENDING"
+        version_name = call.args[2]
+        # EXACT manual-flow scheme: Apply Management + has_pending_config
+        # LIKE-filters key off 'ssl-{id}-...'.
+        assert re.match(r"^ssl-42-create-\d+$", version_name), version_name
+        assert call.args[5] == 7  # created_by honours the importing user
+
+
+@pytest.mark.asyncio
+async def test_stage_reports_per_cluster_failure_without_raising():
+    conn = _mk_conn()
+    conn.fetchval.return_value = 1001
+
+    async def _gen(cluster_id):
+        if cluster_id == 2:
+            raise RuntimeError("config generation exploded")
+        return "# cfg"
+
+    with patch(
+        "services.haproxy_config.generate_haproxy_config_for_cluster",
+        new=AsyncMock(side_effect=_gen),
+    ):
+        results = await stage_ssl_config_versions(conn, 42, [1, 2], created_by=7)
+
+    assert len(results) == 2
+    assert results[0]["success"] is True
+    assert results[1]["success"] is False
+    assert "exploded" in results[1]["error"]

backend/tests/test_csr_key_encryption.py

@@ -0,0 +1,129 @@
+"""Issue #53 (v1.10.1) — at-rest encryption for the pending CSR private key.
+
+Pure logic: no DB, no network. Covers the round-trip, the backward-compatible read of rows
+written before this release, the unrecoverable-key path after a key rotation, and a static
+assertion that the write path can no longer store a raw PEM.
+"""
+import os
+import re
+from pathlib import Path
+
+import pytest
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-csr-encryption-unit-tests")
+
+from cryptography.fernet import Fernet
+
+from utils.csr_key_crypto import (
+    decrypt_csr_private_key,
+    encrypt_csr_private_key,
+    is_encrypted,
+    reset_fernet_for_tests,
+)
+
+_SAMPLE_PEM = (
+    "-----BEGIN PRIVATE KEY-----\n"
+    "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj\n"
+    "-----END PRIVATE KEY-----\n"
+)
+
+
+def test_roundtrip_and_ciphertext_does_not_contain_the_key():
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+    # The stored form must not be the PEM, and must not leak any recognisable fragment of it.
+    assert token != _SAMPLE_PEM
+    assert "-----BEGIN" not in token
+    assert "MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKj" not in token
+    assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+
+
+def test_is_encrypted_discriminates_token_from_legacy_pem():
+    reset_fernet_for_tests()
+    assert is_encrypted(encrypt_csr_private_key(_SAMPLE_PEM)) is True
+    assert is_encrypted(_SAMPLE_PEM) is False
+    assert is_encrypted("") is False
+    assert is_encrypted(None) is False
+
+
+def test_legacy_plaintext_row_is_read_unchanged():
+    # Rows written before v1.10.1 hold a raw PEM. They must keep working with NO data migration,
+    # otherwise upgrading would strand every CSR that is out for signature.
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(_SAMPLE_PEM) == _SAMPLE_PEM
+
+
+def test_empty_or_missing_value_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_csr_private_key(None) is None
+    assert decrypt_csr_private_key("") is None
+
+
+def test_key_rotation_makes_the_stored_key_unrecoverable_rather_than_wrong():
+    """After a rotation the caller must get None, never a silently wrong key."""
+    reset_fernet_for_tests()
+    token = encrypt_csr_private_key(_SAMPLE_PEM)
+
+    # Rotate: an explicit, different CSR_ENCRYPTION_KEY takes precedence over the derived one.
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    os.environ["CSR_ENCRYPTION_KEY"] = Fernet.generate_key().decode()
+    try:
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) is None
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_explicit_env_key_is_used_and_survives_reset():
+    previous = os.environ.get("CSR_ENCRYPTION_KEY")
+    key = Fernet.generate_key().decode()
+    os.environ["CSR_ENCRYPTION_KEY"] = key
+    try:
+        reset_fernet_for_tests()
+        token = encrypt_csr_private_key(_SAMPLE_PEM)
+        # Decryptable with the same explicit key from a fresh instance...
+        reset_fernet_for_tests()
+        assert decrypt_csr_private_key(token) == _SAMPLE_PEM
+        # ...and independently verifiable with the raw Fernet key.
+        assert Fernet(key.encode()).decrypt(token.encode()).decode() == _SAMPLE_PEM
+    finally:
+        if previous is None:
+            os.environ.pop("CSR_ENCRYPTION_KEY", None)
+        else:
+            os.environ["CSR_ENCRYPTION_KEY"] = previous
+        reset_fernet_for_tests()
+
+
+def test_derivation_uses_its_own_hkdf_info_string():
+    """Each secret class derives an independent key, so rotating one never affects another."""
+    src = (Path(__file__).resolve().parent.parent / "utils" / "csr_key_crypto.py").read_text()
+    assert b"csr-private-key-v1".decode() in src
+    # Must NOT reuse another class's info string.
+    for foreign in ("dns-provider-creds-v1", "vip-vrrp-secret-v1", "mfa-totp-secret-v1"):
+        assert foreign not in src, f"CSR key derivation must not reuse the {foreign} info string"
+
+
+def test_write_path_stores_the_encrypted_form_not_the_pem():
+    """Static pin: insert_csr_row must encrypt before the INSERT.
+
+    A future refactor that passed bundle['private_key_pem'] straight through would silently
+    reintroduce plaintext storage, and no unit test with a mocked connection would notice.
+    """
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    insert_fn = src[src.index("async def insert_csr_row("):]
+    insert_fn = insert_fn[: insert_fn.index("\nasync def ")]
+    assert "encrypt_csr_private_key(bundle['private_key_pem'])" in insert_fn
+    # The raw PEM must not be a bind parameter of the INSERT itself.
+    assert not re.search(r"^\s*bundle\['private_key_pem'\],\s*$", insert_fn, re.M)
+
+
+def test_import_path_decrypts_and_fails_closed_on_unrecoverable_key():
+    src = (Path(__file__).resolve().parent.parent / "services" / "csr_service.py").read_text()
+    fn = src[src.index("async def import_signed_certificate("):]
+    assert "decrypt_csr_private_key(row['private_key_pem'])" in fn
+    # A None decrypt must raise rather than fall through to the key-match comparison.
+    assert "cannot be decrypted" in fn

backend/tests/test_csr_migration.py

@@ -0,0 +1,104 @@
+"""
+v1.9.0 CSR creation — static source assertions (pattern: test_vip_purge.py).
+
+Guards the migration wiring that a unit test cannot exercise without a real
+database: the SCHEMA_VERSION bump (without it, deployed installs skip the
+whole migration run and the ssl_csrs table never appears), the migration
+registration, the security-relevant DDL, and the router registration.
+"""
+import os
+import re
+
+_BACKEND_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel_path: str) -> str:
+    with open(os.path.join(_BACKEND_DIR, rel_path), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_schema_version_bumped_to_10():
+    src = _read(os.path.join("database", "migrations.py"))
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert m, "SCHEMA_VERSION constant not found in migrations.py"
+    assert int(m.group(1)) >= 10, (
+        "SCHEMA_VERSION must be >= 10 for the v1.9.0 ssl_csrs table — "
+        "without the bump, existing installs (version >= 9) skip the whole "
+        "migration run and never gain the table."
+    )
+
+
+def test_ssl_csrs_migration_defined_and_registered():
+    src = _read(os.path.join("database", "migrations.py"))
+    assert "async def ensure_ssl_csrs_table" in src
+
+    inner = src.split("async def _run_all_migrations_inner", 1)[1]
+    inner = inner.split("\nasync def ", 1)[0]  # body of the runner only
+    assert "await ensure_ssl_csrs_table()" in inner, (
+        "ensure_ssl_csrs_table must be invoked from _run_all_migrations_inner"
+    )
+
+
+def test_ssl_csrs_ddl_essentials():
+    src = _read(os.path.join("database", "migrations.py"))
+    ddl_start = src.index("CREATE TABLE IF NOT EXISTS ssl_csrs")
+    ddl = src[ddl_start:ddl_start + 2500]
+
+    assert "private_key_pem TEXT" in ddl
+    assert "name VARCHAR(100) NOT NULL" in ddl, (
+        "ssl_csrs.name must align with ssl_certificates.name VARCHAR(100)"
+    )
+    assert "ssl_certificate_id INTEGER REFERENCES ssl_certificates(id) ON DELETE SET NULL" in ddl, (
+        "deleting the imported cert must not cascade into CSR history"
+    )
+    # Partial unique index: only PENDING CSRs reserve their target cert name.
+    assert "uq_ssl_csrs_name_pending" in src
+    assert re.search(
+        r"uq_ssl_csrs_name_pending\s+ON\s+ssl_csrs\(name\)\s+WHERE\s+status\s*=\s*'pending'",
+        src,
+    ), "name uniqueness must be scoped to pending CSRs (partial index)"
+
+
+def test_csr_router_registered_in_main():
+    src = _read("main.py")
+    assert "from routers.csr import router as csr_router" in src
+    assert "app.include_router(csr_router)" in src
+
+
+def test_csr_endpoint_permission_mapping():
+    """Pin which ssl.<action> permission each endpoint enforces: a regression
+    that dropped or weakened a _require() call would otherwise pass the
+    auth-rejection tests (they only assert 401/403 for unauthenticated calls)."""
+    src = _read(os.path.join("routers", "csr.py"))
+
+    def _handler_body(decorator):
+        start = src.index(decorator)
+        nxt = src.find("@router.", start + 1)
+        return src[start:nxt if nxt != -1 else len(src)]
+
+    expectations = [
+        ('@router.post("")', '"create"'),
+        ('@router.get("")', '"read"'),
+        ('@router.get("/{csr_id}")', '"read"'),
+        ('@router.post("/{csr_id}/import")', '"create"'),
+        ('@router.delete("/{csr_id}")', '"delete"'),
+    ]
+    for decorator, action in expectations:
+        body = _handler_body(decorator)
+        assert f"_require(authorization, {action})" in body, (
+            f"endpoint {decorator} must enforce ssl.{action.strip(chr(34))}"
+        )
+
+
+def test_csr_router_never_selects_private_key():
+    """The CSR endpoints must use the explicit column list — a bare
+    `SELECT *` into an API response is how the key would leak. The one place
+    SELECT * is allowed is the service-layer FOR UPDATE row (it needs the key
+    to pair with the cert); the router itself must not touch the column."""
+    src = _read(os.path.join("routers", "csr.py"))
+    code_only = re.sub(r"#.*", "", src)  # strip comments; the column name may
+    # legitimately appear there as documentation
+    assert "private_key_pem" not in code_only, (
+        "routers/csr.py must never reference private_key_pem in code"
+    )
+    assert "SELECT *" not in code_only, "routers/csr.py must use explicit column lists"

backend/tests/test_csr_models.py

@@ -0,0 +1,172 @@
+"""
+v1.9.0 CSR creation — Pydantic model validation tests (models/csr.py).
+
+The CSR name shares the SSL certificate name's path-traversal contract
+(Bulgu #21) with one deliberate tightening: max 100 chars, matching the
+ssl_certificates.name VARCHAR(100) column.
+"""
+import pytest
+from pydantic import ValidationError
+
+from models.csr import SSLCSRCreate, SSLCSRImport
+
+_CERT_PEM = "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+
+
+def _create(**overrides):
+    base = dict(name="my-csr", common_name="www.example.com")
+    base.update(overrides)
+    return SSLCSRCreate(**base)
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRCreate
+# ----------------------------------------------------------------------------
+
+
+def test_minimal_valid_create():
+    m = _create()
+    assert m.name == "my-csr"
+    assert m.common_name == "www.example.com"
+    assert m.key_algorithm == "rsa-2048"
+    assert m.sans == []
+
+
+@pytest.mark.parametrize("bad_name", [
+    "../../etc/cron.d/evil",   # path traversal
+    "a..b",                    # embedded ..
+    ".hidden",                 # hidden filename
+    "-flag",                   # CLI flag confusion
+    "has space",
+    "wild*card",
+    "",
+    "x" * 101,                 # VARCHAR(100) alignment — 200 is NOT allowed here
+])
+def test_name_rejects_unsafe_values(bad_name):
+    with pytest.raises(ValidationError):
+        _create(name=bad_name)
+
+
+def test_name_accepts_100_chars():
+    assert _create(name="x" * 100).name == "x" * 100
+
+
+def test_common_name_wildcard_accepted_and_lowercased():
+    m = _create(common_name="*.Example.COM")
+    assert m.common_name == "*.example.com"
+
+
+@pytest.mark.parametrize("bad_cn", [
+    "",
+    "under_score.example.com",      # _ is not LDH
+    "*.*.example.com",              # wildcard only as leftmost single label
+    "-leading.example.com",
+    "a" * 70 + ".example.com",      # label > 63
+    "cn-longer-than-64-chars-" + "x" * 45 + ".example.com",  # CN > 64 total
+])
+def test_common_name_rejects_invalid(bad_cn):
+    with pytest.raises(ValidationError):
+        _create(common_name=bad_cn)
+
+
+def test_sans_normalised_deduped_and_capped():
+    m = _create(sans=["API.example.com", "api.example.com", "cdn.example.com"])
+    assert m.sans == ["api.example.com", "cdn.example.com"]
+
+    with pytest.raises(ValidationError):
+        _create(sans=[f"h{i}.example.com" for i in range(101)])
+
+
+def test_country_normalised_or_rejected():
+    assert _create(country="tr").country == "TR"
+    assert _create(country=None).country is None
+    for bad in ("TUR", "T", "1A"):
+        with pytest.raises(ValidationError):
+            _create(country=bad)
+
+
+def test_subject_fields_reject_control_characters():
+    with pytest.raises(ValidationError):
+        _create(organization="Evil\x00Corp")
+    with pytest.raises(ValidationError):
+        _create(locality="line\nbreak")
+
+
+def test_subject_fields_reject_overlength():
+    with pytest.raises(ValidationError):
+        _create(organization="x" * 65)
+
+
+def test_key_algorithm_strict_enum():
+    for good in ("rsa-2048", "rsa-4096", "ecdsa-p256", "ecdsa-p384"):
+        assert _create(key_algorithm=good).key_algorithm == good
+    for bad in ("rsa-1024", "rsa-8192", "ed25519", "2048", ""):
+        with pytest.raises(ValidationError):
+            _create(key_algorithm=bad)
+
+
+def test_email_basic_validation():
+    assert _create(email="ops@example.com").email == "ops@example.com"
+    with pytest.raises(ValidationError):
+        _create(email="not-an-email")
+
+
+# ----------------------------------------------------------------------------
+# SSLCSRImport
+# ----------------------------------------------------------------------------
+
+
+def test_import_minimal_global():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True)
+    assert m.usage_type == "frontend"
+    assert m.name is None
+
+
+def test_import_requires_clusters_when_not_global():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[])
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=False, cluster_ids=[1])
+    assert m.cluster_ids == [1]
+
+
+def test_import_certificate_must_be_pem():
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="not a pem", is_global=True)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content="", is_global=True)
+
+
+def test_import_certificate_size_capped():
+    huge = _CERT_PEM + "A" * (64 * 1024 + 1)
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=huge, is_global=True)
+
+
+def test_import_chain_optional_but_validated():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, chain_content="  ")
+    assert m.chain_content is None
+    with pytest.raises(ValidationError):
+        SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, chain_content="garbage"
+        )
+
+
+def test_import_name_override_shares_the_name_contract():
+    m = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="renamed")
+    assert m.name == "renamed"
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="../evil")
+    # Empty override collapses to None (falls back to the CSR's own name).
+    m2 = SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, name="  ")
+    assert m2.name is None
+
+
+def test_import_usage_type_enum():
+    for good in ("frontend", "server"):
+        assert SSLCSRImport(
+            certificate_content=_CERT_PEM, is_global=True, usage_type=good
+        ).usage_type == good
+    with pytest.raises(ValidationError):
+        SSLCSRImport(certificate_content=_CERT_PEM, is_global=True, usage_type="both")

backend/tests/test_csr_router_auth.py

@@ -0,0 +1,66 @@
+"""
+v1.9.0 CSR creation — behavioral auth tests for /api/ssl/csrs endpoints
+(pattern: test_ssl_list_endpoint_auth.py).
+
+Every CSR endpoint must refuse unauthenticated / garbage-token requests.
+The CSR detail route additionally must never 200 without auth because it
+returns the CSR PEM; no endpoint ever returns the private key, but auth is
+the first line regardless.
+"""
+import pytest
+
+_VALID_CREATE_BODY = {
+    "name": "auth-test-csr",
+    "common_name": "www.example.com",
+}
+
+_VALID_IMPORT_BODY = {
+    "certificate_content": (
+        "-----BEGIN CERTIFICATE-----\nX\n-----END CERTIFICATE-----"
+    ),
+    "is_global": True,
+}
+
+_ENDPOINTS = [
+    ("get", "/api/ssl/csrs", None),
+    ("get", "/api/ssl/csrs/1", None),
+    ("post", "/api/ssl/csrs", _VALID_CREATE_BODY),
+    ("post", "/api/ssl/csrs/1/import", _VALID_IMPORT_BODY),
+    ("delete", "/api/ssl/csrs/1", None),
+]
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_unauthenticated_rejected(client, method, path, body):
+    """No Authorization header → endpoint must refuse the request."""
+    res = getattr(client, method)(path, json=body) if body is not None else getattr(client, method)(path)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} without Authorization returned "
+        f"{res.status_code} — anonymous access to CSR data must not be "
+        f"possible. Body: {res.text[:200]}"
+    )
+    if res.status_code == 200:  # defensive, mirrors the R18 test style
+        data = res.json()
+        assert not data, "CSR endpoint returned data without auth"
+
+
+@pytest.mark.parametrize("method,path,body", _ENDPOINTS)
+def test_csr_endpoint_invalid_token_rejected(client, method, path, body):
+    """Garbage token → endpoint must refuse the request."""
+    headers = {"Authorization": "Bearer not-a-valid-jwt"}
+    if body is not None:
+        res = getattr(client, method)(path, json=body, headers=headers)
+    else:
+        res = getattr(client, method)(path, headers=headers)
+    assert res.status_code in (401, 403, 422), (
+        f"{method.upper()} {path} with an invalid token returned {res.status_code}"
+    )
+
+
+def test_csr_routes_are_registered(client):
+    """The router must actually be mounted — a 404 would make the auth tests
+    above pass vacuously."""
+    res = client.get("/api/ssl/csrs")
+    assert res.status_code != 404, (
+        "GET /api/ssl/csrs returned 404 — csr_router is not registered in main.py"
+    )

backend/tests/test_csr_service_crypto.py

@@ -0,0 +1,171 @@
+"""
+v1.9.0 CSR creation — pure-crypto tests for services/csr_service.py.
+
+No mocks: every algorithm's output must parse with `cryptography` and the
+CSR's public key must match the generated private key (the property the
+whole import flow depends on).
+"""
+from types import SimpleNamespace
+
+import pytest
+
+from cryptography import x509
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import ec, rsa
+from cryptography.x509.oid import ExtensionOID, NameOID
+
+from services.csr_service import csr_row_to_dict, diff_domains, generate_csr_bundle
+
+
+def _payload(**overrides):
+    base = dict(
+        name="test-csr",
+        common_name="www.example.com",
+        organization=None,
+        organizational_unit=None,
+        locality=None,
+        state=None,
+        country=None,
+        email=None,
+        sans=[],
+        key_algorithm="rsa-2048",
+    )
+    base.update(overrides)
+    return SimpleNamespace(**base)
+
+
+def _spki(key):
+    return key.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+
+
+@pytest.mark.parametrize(
+    "algo,key_cls,key_check",
+    [
+        ("rsa-2048", rsa.RSAPrivateKey, lambda k: k.key_size == 2048),
+        ("rsa-4096", rsa.RSAPrivateKey, lambda k: k.key_size == 4096),
+        ("ecdsa-p256", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp256r1"),
+        ("ecdsa-p384", ec.EllipticCurvePrivateKey, lambda k: k.curve.name == "secp384r1"),
+    ],
+)
+def test_generate_bundle_all_algorithms(algo, key_cls, key_check):
+    bundle = generate_csr_bundle(_payload(key_algorithm=algo))
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    key = serialization.load_pem_private_key(
+        bundle["private_key_pem"].encode(), password=None
+    )
+
+    assert isinstance(key, key_cls)
+    assert key_check(key)
+    # The CSR must be signed by exactly this key.
+    csr_spki = csr.public_key().public_bytes(
+        serialization.Encoding.DER,
+        serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+    assert csr_spki == _spki(key)
+    assert csr.is_signature_valid
+    # PKCS8, unencrypted — the agent concatenates cert+key into one PEM and
+    # HAProxy cannot read passphrase-protected keys.
+    assert bundle["private_key_pem"].startswith("-----BEGIN PRIVATE KEY-----")
+
+
+def test_subject_contains_all_provided_fields():
+    bundle = generate_csr_bundle(_payload(
+        organization="Example Corp",
+        organizational_unit="IT",
+        locality="Istanbul",
+        state="Marmara",
+        country="TR",
+        email="ops@example.com",
+    ))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+
+    def _one(oid):
+        attrs = csr.subject.get_attributes_for_oid(oid)
+        return attrs[0].value if attrs else None
+
+    assert _one(NameOID.COMMON_NAME) == "www.example.com"
+    assert _one(NameOID.ORGANIZATION_NAME) == "Example Corp"
+    assert _one(NameOID.ORGANIZATIONAL_UNIT_NAME) == "IT"
+    assert _one(NameOID.LOCALITY_NAME) == "Istanbul"
+    assert _one(NameOID.STATE_OR_PROVINCE_NAME) == "Marmara"
+    assert _one(NameOID.COUNTRY_NAME) == "TR"
+    assert _one(NameOID.EMAIL_ADDRESS) == "ops@example.com"
+    assert bundle["subject"] == {
+        "O": "Example Corp", "OU": "IT", "L": "Istanbul",
+        "ST": "Marmara", "C": "TR", "emailAddress": "ops@example.com",
+    }
+
+
+def test_subject_omits_empty_fields():
+    bundle = generate_csr_bundle(_payload())
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    assert not csr.subject.get_attributes_for_oid(NameOID.ORGANIZATION_NAME)
+    assert bundle["subject"] == {}
+
+
+def test_sans_cn_first_and_deduped():
+    bundle = generate_csr_bundle(_payload(
+        common_name="www.example.com",
+        sans=["api.example.com", "www.example.com", "api.example.com", "cdn.example.com"],
+    ))
+    assert bundle["sans"] == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    dns_names = san_ext.value.get_values_for_type(x509.DNSName)
+    assert dns_names == ["www.example.com", "api.example.com", "cdn.example.com"]
+
+
+def test_wildcard_common_name_flows_into_san():
+    bundle = generate_csr_bundle(_payload(common_name="*.example.com"))
+    csr = x509.load_pem_x509_csr(bundle["csr_pem"].encode())
+    san_ext = csr.extensions.get_extension_for_oid(
+        ExtensionOID.SUBJECT_ALTERNATIVE_NAME
+    )
+    assert san_ext.value.get_values_for_type(x509.DNSName) == ["*.example.com"]
+
+
+def test_diff_domains_reports_added_and_dropped():
+    warnings = diff_domains(
+        ["www.example.com", "api.example.com"],
+        ["WWW.example.com", "cdn.example.com"],
+    )
+    assert len(warnings) == 2
+    added = next(w for w in warnings if "added" in w)
+    dropped = next(w for w in warnings if "dropped" in w)
+    assert "cdn.example.com" in added
+    assert "api.example.com" in dropped
+    # Case-insensitive: www must NOT be reported in either direction.
+    assert "www.example.com" not in added
+    assert "www.example.com" not in dropped
+
+
+def test_diff_domains_identical_sets_yield_no_warnings():
+    assert diff_domains(["a.example.com"], ["A.EXAMPLE.COM"]) == []
+    assert diff_domains([], []) == []
+
+
+def test_csr_row_to_dict_never_exposes_private_key():
+    row = {
+        "id": 1,
+        "name": "x",
+        "private_key_pem": "-----BEGIN PRIVATE KEY-----\nSECRET\n-----END PRIVATE KEY-----",
+        "csr_pem": "-----BEGIN CERTIFICATE REQUEST-----\nX\n-----END CERTIFICATE REQUEST-----",
+        "subject": '{"O": "Example"}',
+        "sans": '["a.example.com"]',
+    }
+    out = csr_row_to_dict(row)
+    assert "private_key_pem" not in out
+    assert "csr_pem" not in out          # lists exclude the PEM
+    assert out["subject"] == {"O": "Example"}
+    assert out["sans"] == ["a.example.com"]
+
+    detail = csr_row_to_dict(row, include_pem=True)
+    assert "private_key_pem" not in detail  # NEVER, even on detail
+    assert detail["csr_pem"].startswith("-----BEGIN CERTIFICATE REQUEST-----")

backend/tests/test_dns01.py

@@ -0,0 +1,615 @@
+"""Issue #35 — ACME DNS-01: focused unit tests for the pure logic (no DB/network).
+
+Covers the TXT-value math (RFC 8555 §8.4 — raw SHA-256 digest, base64url, NOT hex),
+the _acme-challenge record-name derivation (wildcard stripping), credential encryption
+round-trip + tamper handling, the DNS provider registry/allow-list, and (v1.10.0) the
+GoDaddy provider's zone-relative name derivation and additive RRset merge math.
+"""
+import base64
+import hashlib
+import os
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-dns01-unit-tests")
+
+from services.acme_service import ACMEService
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import (
+    encrypt_dns_credentials, decrypt_dns_credentials, reset_fernet_for_tests,
+)
+
+
+def _b64url(b: bytes) -> str:
+    return base64.urlsafe_b64encode(b).rstrip(b"=").decode("ascii")
+
+
+def test_dns_txt_value_is_raw_sha256_base64url():
+    key_auth = "token123.thumbprintABC"
+    expected = _b64url(hashlib.sha256(key_auth.encode("utf-8")).digest())
+    assert ACMEService._dns_txt_value(key_auth) == expected
+    # Must NOT be the (classic-mistake) base64url of the HEX digest.
+    hex_based = _b64url(hashlib.sha256(key_auth.encode("utf-8")).hexdigest().encode("utf-8"))
+    assert ACMEService._dns_txt_value(key_auth) != hex_based
+
+
+def test_challenge_dns_name_derivation():
+    assert ACMEService._challenge_dns_name("example.com") == "_acme-challenge.example.com"
+    # Wildcard: the '*.' is stripped, so apex + wildcard share the SAME record name.
+    assert ACMEService._challenge_dns_name("*.example.com") == "_acme-challenge.example.com"
+    assert ACMEService._challenge_dns_name("foo.bar.example.com") == "_acme-challenge.foo.bar.example.com"
+
+
+def test_credential_encryption_roundtrip():
+    reset_fernet_for_tests()
+    creds = {"api_token": "super-secret-token-value"}
+    token = encrypt_dns_credentials(creds)
+    assert token != "super-secret-token-value"
+    assert "super-secret-token-value" not in token  # ciphertext, not plaintext
+    assert decrypt_dns_credentials(token) == creds
+
+
+def test_decrypt_invalid_token_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_dns_credentials("not-a-valid-fernet-token") is None
+
+
+def test_provider_registry_and_allow_list():
+    names = {p["name"] for p in list_providers()}
+    assert {"manual", "cloudflare", "godaddy"} <= names
+    assert is_supported("manual") and is_supported("cloudflare") and is_supported("godaddy")
+    assert not is_supported("route53")  # not in the allow-list
+
+    assert get_provider("manual").automated is False
+    cf = get_provider("cloudflare", {"api_token": "x"})
+    assert cf.automated is True
+    assert any(f["key"] == "api_token" for f in cf.credential_fields)
+
+    raised = False
+    try:
+        get_provider("definitely-not-a-provider")
+    except ValueError:
+        raised = True
+    assert raised
+
+
+def test_cloudflare_token_sanitize():
+    # Issue #35 follow-up: a pasted token with quotes/spaces/control/unicode chars produced an
+    # invalid Authorization header (CF 6003 "Invalid request headers"). The sanitizer strips them.
+    from services.dns_providers.cloudflare import _sanitize_token, CloudflareDNSProvider
+
+    # Surrounding double quotes stripped.
+    assert _sanitize_token('"abc123-_def"') == 'abc123-_def'
+    # Interior spaces / tabs / newlines removed.
+    assert _sanitize_token('abc 123\tdef\n') == 'abc123def'
+    # A clean token68 string is unchanged (cannot corrupt a valid Cloudflare token).
+    clean = 'A1b2-_C3.d4~e5+f6/g7=='
+    assert _sanitize_token(clean) == clean
+    # Single quotes and a zero-width char removed.
+    assert _sanitize_token("'tok" + chr(0x200b) + "en'") == 'token'
+
+    # The provider constructor sanitizes into _token and keeps the raw input for diagnostics.
+    p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
+    assert p._token == 'my-token_123'
+    assert p._raw_token == '"my-token_123"'
+
+
+# --- v1.10.0: GoDaddy provider (pure logic only — no network, no DB) ---
+
+
+def test_godaddy_credential_fields_schema():
+    # Re-assert DnsCredentialsUpsert's validator rules directly against the declared schema, so the
+    # UI can never render a field whose submission the API would reject with a 422.
+    import re
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    fields = GoDaddyDNSProvider.credential_fields
+    assert [f["key"] for f in fields] == ["api_key", "api_secret"]
+    for f in fields:
+        assert re.match(r"^[a-zA-Z0-9_]{1,50}$", f["key"])         # DnsCredentialsUpsert key regex
+        assert f["type"] == "password"                              # renders Input.Password, not Input
+        assert isinstance(f["max_length"], int) and 0 < f["max_length"] <= 4000  # validator value cap
+        assert f["help"] and isinstance(f["help"], str)             # shown in the Form.Item `extra` slot
+    # api_secret is optional on purpose: leaving it blank is how a Personal Access Token is used
+    # (Bearer), which is the migration path off the sso-key scheme GoDaddy is retiring.
+    assert fields[0]["required"] is True and fields[1]["required"] is False
+    # Must not reuse Cloudflare's field name: the register modal's credential Form.Items are named
+    # cred_<key> in a SHARED form and are not cleared when the provider dropdown changes.
+    assert "api_token" not in {f["key"] for f in fields}
+
+
+def test_godaddy_provider_is_automated():
+    p = get_provider("godaddy", {"api_key": "k", "api_secret": "s"})
+    assert p.automated is True      # else the orchestrator takes the manual-confirm branch
+    assert p.name == "godaddy" and 1 <= len(p.name) <= 50  # dns_provider Field(min_length=1, max_length=50)
+    assert p.label == "GoDaddy"
+
+
+def test_godaddy_missing_credentials_returns_not_ok():
+    # verify_credentials must RETURN {"ok": False}, never raise: the router turns any non-
+    # DnsProviderError into the information-free generic 422 and the user never sees the reason.
+    import asyncio
+
+    for creds in ({}, {"api_secret": "s"}):  # blank UI fields arrive as MISSING keys, not ""
+        r = asyncio.run(get_provider("godaddy", creds).verify_credentials())
+        assert r["ok"] is False and r["detail"]
+    # Short-circuits before any request, so this touches no network.
+
+
+def test_godaddy_auth_header_formats_and_secret_never_leaks():
+    from services.dns_providers.godaddy import GoDaddyDNSProvider, _scrub
+
+    sentinel = "SENTINEL-SECRET-DO-NOT-LEAK"
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": sentinel})
+    # Literal prefix, one space, a single colon — no base64, no quoting.
+    assert p._auth_header() == f"sso-key KEY123:{sentinel}"
+    # No secret -> Personal Access Token. This one branch is the whole sso-key-sunset migration.
+    assert GoDaddyDNSProvider({"api_key": "PAT"})._auth_header() == "Bearer PAT"
+    # _scrub removes credential substrings from anything bound for a log or an order event.
+    assert sentinel not in _scrub(f"boom {sentinel} boom", "KEY123", sentinel)
+    assert "KEY123" not in _scrub("boom KEY123", "KEY123", sentinel)
+    assert _scrub("x" * 500, "KEY123") == "x" * 300  # bounded, so a huge body can't flood an event
+
+    # The channel that actually persists text: _http_error composes the message an order event and
+    # letsencrypt_orders.error_detail will carry, so it must scrub its own inputs — a caller that
+    # forgets to pre-scrub must not be able to leak. (Regression guard: scrubbing used to live at
+    # the single call site in _request instead of here.)
+    exc = p._http_error(403, f"DENIED_{sentinel}", f"token {sentinel} rejected", None)
+    assert sentinel not in str(exc) and "***" in str(exc)
+
+    # This module must not log at all — logging is the one channel _scrub cannot reach, since the
+    # arguments would be formatted by the logging framework rather than passed through it.
+    import inspect
+    import re as _re
+    from services.dns_providers import godaddy as gd_mod
+
+    assert not _re.search(r"\blogger\.\w+\(", inspect.getsource(gd_mod)), \
+        "godaddy.py must not log; surface everything through DnsProviderError so it is scrubbed"
+
+
+def test_godaddy_relative_record_name():
+    # GoDaddy names are RELATIVE to the zone with no trailing dot; the apex is the literal "@".
+    from services.dns_providers.godaddy import _relative_name
+
+    assert _relative_name("_acme-challenge.example.com", "example.com") == "_acme-challenge"
+    assert _relative_name("_acme-challenge.foo.bar.example.com", "example.com") == "_acme-challenge.foo.bar"
+    assert _relative_name("example.com", "example.com") == "@"          # never "" — see _rrset_path
+    assert _relative_name("_acme-challenge.example.com.", "example.com") == "_acme-challenge"
+    assert _relative_name("_ACME-Challenge.Example.COM", "example.com") == "_acme-challenge"
+    # Apex and wildcard produce the SAME relative name — which is exactly why the merge below
+    # has to be additive.
+    apex = ACMEService._challenge_dns_name("example.com")
+    wild = ACMEService._challenge_dns_name("*.example.com")
+    assert _relative_name(apex, "example.com") == _relative_name(wild, "example.com") == "_acme-challenge"
+
+
+def test_godaddy_rrset_merge_is_additive():
+    # THE critical test: GoDaddy's PUT REPLACES an entire RRset, so the merge math is the only thing
+    # keeping a wildcard+apex certificate's two coexisting TXT values alive.
+    from services.dns_providers.godaddy import _live_values, _merge_add, _merge_remove
+
+    def vals(body):
+        return sorted(r["data"] for r in body)
+
+    assert vals(_merge_add([{"data": "valueA", "ttl": 600}], "valueB")) == ["valueA", "valueB"]
+    assert _merge_add([{"data": "valueA"}], "valueA") is None      # idempotent; ACME retries land here
+    # Total, not an all()-over-a-computed-list (which passes vacuously on an empty result): the
+    # first publish at a fresh name must emit exactly one element, carrying the 600s TTL floor.
+    assert _merge_add([], "v") == [{"data": "v", "ttl": 600}]      # below 600 GoDaddy answers 422
+    # Tombstone rows ({"data": ""}) must never be echoed back — GoDaddy answers 422 INVALID_BODY.
+    assert _live_values([{"data": ""}, {"data": "x"}, {}]) == ["x"]
+    assert vals(_merge_add([{"data": ""}, {"data": "valueA"}], "valueB")) == ["valueA", "valueB"]
+
+    assert vals(_merge_remove([{"data": "valueA"}, {"data": "valueB"}], "valueB")) == ["valueA"]
+    assert _merge_remove([{"data": "valueA"}], "valueZ") is None    # already gone — tolerate
+    assert _merge_remove([], "valueZ") is None
+    # [] means "use DELETE": PUT with an empty array is rejected (422, "Records must be specified").
+    assert _merge_remove([{"data": "valueA"}], "valueA") == []
+    assert _merge_remove([{"data": ""}, {"data": "valueA"}], "valueA") == []
+
+
+def test_godaddy_never_builds_a_zone_wide_txt_path():
+    # A 3-segment path (.../records/TXT) is the endpoint that wipes EVERY TXT in the zone — SPF,
+    # DKIM, DMARC, domain verifications. An empty relative name must never be able to produce it.
+    from services.dns_providers.godaddy import _rrset_path
+
+    p = _rrset_path("example.com", "_acme-challenge")
+    assert p == "/domains/example.com/records/TXT/_acme-challenge"
+    assert p.count("/") == 5 and not p.endswith("/TXT")
+    assert _rrset_path("example.com", "@").endswith("/%40")  # apex percent-encoded for proxy safety
+    # "." and ".." survive quote() and are then normalized away by yarl when the URL is built, so
+    # ".../records/TXT/.." would resolve to the whole-zone endpoint. They must be refused too.
+    for bad in [("example.com", ""), ("", "_acme-challenge"), ("example.com", "."),
+                ("example.com", ".."), ("example.com", "...")]:
+        raised = False
+        try:
+            _rrset_path(*bad)
+        except DnsProviderError:
+            raised = True
+        assert raised, f"_rrset_path{bad} must refuse to build a zone-wide TXT path"
+    # And the only way to reach those inputs — a malformed domain — really does produce them.
+    from services.dns_providers.godaddy import _relative_name as _rel
+    assert _rel("..example.com", "example.com") == "."
+
+
+def test_godaddy_credential_encryption_roundtrip():
+    # The two-field credential dict rides the same Fernet blob as Cloudflare's single token.
+    reset_fernet_for_tests()
+    creds = {"api_key": "gd-key-plaintext", "api_secret": "gd-secret-plaintext"}
+    token = encrypt_dns_credentials(creds)
+    assert "gd-key-plaintext" not in token and "gd-secret-plaintext" not in token  # ciphertext
+    assert decrypt_dns_credentials(token) == creds
+    # This sorted key list is exactly what GET /dns-credentials exposes as credential_fields_present
+    # — names only, never values.
+    assert sorted(decrypt_dns_credentials(token).keys()) == ["api_key", "api_secret"]
+
+
+_GD_NS = "/domains/example.com/records/NS"
+_GD_TXT = "/domains/example.com/records/TXT/_acme-challenge"
+
+
+def _gd_provider(responses):
+    """A GoDaddy provider whose _request is replaced by a recorder.
+
+    The pure-merge tests above prove the MATH; this proves the WRITE PATH actually uses it. Without
+    it, replacing the merge with a single-value PUT — the mutation that silently destroys the
+    sibling value of every wildcard+apex certificate — leaves the whole suite green.
+
+    `responses` maps (method, path) -> value to return, or an Exception to raise. Unmapped calls
+    return None, which is how the zone suffix-walk's failed probes are modelled.
+    """
+    import types
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    calls = []
+
+    async def _fake_request(self, session, method, path, **kwargs):
+        calls.append((method, path, kwargs.get("json")))
+        result = responses.get((method, path))
+        if isinstance(result, Exception):
+            raise result
+        return result
+
+    p = GoDaddyDNSProvider({"api_key": "k", "api_secret": "s"})
+    p._request = types.MethodType(_fake_request, p)
+    return p, calls
+
+
+def _assert_never_zone_wide(calls):
+    # A write to .../records or .../records/TXT replaces every TXT (or every record) in the zone.
+    for method, path, _json in calls:
+        if method in ("PUT", "DELETE"):
+            assert not path.endswith("/records"), f"zone-wide write: {method} {path}"
+            assert not path.endswith("/records/TXT"), f"type-wide write: {method} {path}"
+
+
+def test_godaddy_add_write_path_merges_siblings():
+    import asyncio
+
+    # An existing sibling value at the same name — the apex half of an apex+wildcard certificate.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+
+    writes = [c for c in calls if c[0] in ("PUT", "PATCH", "DELETE")]
+    assert len(writes) == 1 and writes[0][0] == "PUT" and writes[0][1] == _GD_TXT
+    # BOTH values must be in the body: GoDaddy's PUT replaces the whole RRset.
+    assert sorted(r["data"] for r in writes[0][2]) == ["valueA", "valueB"]
+    _assert_never_zone_wide(calls)
+
+
+def test_godaddy_add_write_path_is_idempotent_and_fails_closed():
+    import asyncio
+
+    # Already published -> no write at all (this is where an ACME retry cycle lands).
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueB", "ttl": 600}],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    assert [c for c in calls if c[0] != "GET"] == []
+
+    # Unreadable RRset read (2xx whose body did not parse as a list) must FAIL, never be treated as
+    # an empty RRset — the PUT that follows would replace the sibling values with only ours.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): None,
+    })
+    raised = False
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    except DnsProviderError:
+        raised = True
+    assert raised, "an unreadable RRset read must not be coerced into an empty RRset"
+    assert [c for c in calls if c[0] != "GET"] == []
+
+
+def test_godaddy_remove_write_path_uses_delete_for_the_last_value():
+    import asyncio
+
+    # Two values -> PUT back the survivor only.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}, {"data": "valueB"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueB"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0][0] == "PUT"
+    assert [r["data"] for r in writes[0][2]] == ["valueA"]
+
+    # Last value -> DELETE. `PUT []` is rejected by GoDaddy (422 INVALID_BODY), so an empty PUT
+    # body would make every cleanup fail forever.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueA"))
+    writes = [c for c in calls if c[0] != "GET"]
+    assert len(writes) == 1 and writes[0] == ("DELETE", _GD_TXT, None)
+    assert not any(c[0] == "PUT" and c[2] == [] for c in calls)
+
+    # Value already gone -> no write, no error.
+    p, calls = _gd_provider({
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [{"data": "valueA"}],
+    })
+    asyncio.run(p.remove_txt_record("_acme-challenge.example.com", "valueZ"))
+    assert [c for c in calls if c[0] != "GET"] == []
+    _assert_never_zone_wide(calls)
+
+
+class _FakeGDResponse:
+    """Minimal stand-in for aiohttp's ClientResponse: status, headers, and json()."""
+
+    _NO_BODY = object()
+
+    def __init__(self, status, body=_NO_BODY, headers=None):
+        self.status = status
+        self._body = body
+        self.headers = headers or {}
+
+    async def json(self, content_type=None):
+        if self._body is _FakeGDResponse._NO_BODY:
+            raise ValueError("no body to decode")  # what an empty 204 does
+        return self._body
+
+
+class _FakeGDSession:
+    def __init__(self, response):
+        self._response = response
+        self.calls = []
+
+    def request(self, method, url, **kwargs):
+        self.calls.append((method, url, kwargs))
+        response = self._response
+
+        class _Ctx:
+            async def __aenter__(self_inner):
+                return response
+
+            async def __aexit__(self_inner, *exc):
+                return False
+
+        return _Ctx()
+
+
+def test_godaddy_request_status_handling():
+    import asyncio
+    from services.dns_providers.godaddy import GoDaddyDNSProvider
+
+    p = GoDaddyDNSProvider({"api_key": "KEY123", "api_secret": "SEC456"})
+
+    def call(response):
+        session = _FakeGDSession(response)
+        try:
+            return asyncio.run(p._request(session, "PUT", "/domains/example.com/records/TXT/x",
+                                          json=[{"data": "v", "ttl": 600}])), None, session
+        except DnsProviderError as exc:
+            return None, str(exc), session
+
+    # 204 with an EMPTY body is the normal answer to every GoDaddy write — it must not raise.
+    body, err, session = call(_FakeGDResponse(204))
+    assert body is None and err is None
+    # Redirects are deliberately not followed (aiohttp would forward the Authorization header), so
+    # a 3xx is a FAILED call. Treating it as success would report a redirected write as a no-op.
+    _kw = session.calls[0][2]
+    assert _kw["allow_redirects"] is False
+    assert _kw["headers"]["Authorization"] == "sso-key KEY123:SEC456"
+    assert _kw["headers"]["Accept"] == "application/json"
+    for status in (301, 302, 307):
+        body, err, _ = call(_FakeGDResponse(status))
+        assert body is None and err and str(status) in err, f"HTTP {status} must not read as success"
+
+    # 200 with a list is passed through verbatim.
+    body, err, _ = call(_FakeGDResponse(200, [{"data": "v"}]))
+    assert err is None and body == [{"data": "v"}]
+
+    # Error mapping: each message must name what the operator has to fix.
+    _, err, _ = call(_FakeGDResponse(401, {"code": "UNABLE_TO_AUTHENTICATE", "message": "nope"}))
+    assert "PRODUCTION" in err and "UNABLE_TO_AUTHENTICATE" in err
+    _, err, _ = call(_FakeGDResponse(403, {"code": "ACCESS_DENIED", "message": "not allowed"}))
+    assert "domains.dns:update" in err
+    # 429: Retry-After wins; the legacy body field is the fallback; absent both -> 60s default.
+    _, err, _ = call(_FakeGDResponse(429, None, {"Retry-After": "17"}))
+    assert "~17s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"retryAfterSec": 42}))
+    assert "~42s" in err
+    _, err, _ = call(_FakeGDResponse(429, {"Retry-After": "not-a-number"}))
+    assert "~60s" in err
+    # A non-dict error body must not crash the error mapper.
+    _, err, _ = call(_FakeGDResponse(500, "<html>gateway</html>"))
+    assert "500" in err
+
+    # A transport failure MID-READ must not be mistaken for "empty body". Only a decode error may
+    # be swallowed: a caller reading an RRset would otherwise see None and could take it for an
+    # empty record set, and the full-RRset PUT that follows would destroy the sibling values.
+    import aiohttp
+
+    class _TruncatedResponse(_FakeGDResponse):
+        async def json(self, content_type=None):
+            raise aiohttp.ClientPayloadError("connection closed mid-body")
+
+    body, err, _ = call(_TruncatedResponse(200))
+    assert body is None and err and "GoDaddy" in err
+
+
+def test_godaddy_zone_resolution_walks_suffixes_and_caches():
+    import asyncio
+    from services.dns_providers.godaddy import _GoDaddyHTTPError
+
+    # The deepest candidate is not a zone (404 = "not this zone"); the walk must continue to the
+    # registrable domain and then reuse it, so the second challenge at the same name costs no probe.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("nope", status=404, code="UNKNOWN_DOMAIN"),
+        ("GET", _GD_NS): [{"data": "ns1.domaincontrol.com"}],
+        ("GET", _GD_TXT): [],
+    })
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueA"))
+    asyncio.run(p.add_txt_record("_acme-challenge.example.com", "valueB"))
+    probes = [c for c in calls if c[1].endswith("/records/NS")]
+    assert len(probes) == 2, "the resolved zone must be cached for the life of the provider"
+    # Relative name derived from the RESOLVED zone, never from the deepest candidate.
+    assert all(c[1] == _GD_TXT for c in calls if "/records/TXT/" in c[1])
+
+    # A credential/eligibility failure during the walk must surface, not be swallowed as
+    # "no managed domain" — otherwise the operator chases a DNS problem that is really a bad key.
+    p, calls = _gd_provider({
+        ("GET", "/domains/_acme-challenge.example.com/records/NS"):
+            _GoDaddyHTTPError("denied", status=403, code="ACCESS_DENIED"),
+    })
+    raised = ""
+    try:
+        asyncio.run(p.add_txt_record("_acme-challenge.example.com", "v"))
+    except DnsProviderError as exc:
+        raised = str(exc)
+    assert "denied" in raised and "No managed GoDaddy domain" not in raised
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/tests/test_haproxy_validator_bulgu12.py

@@ -206,41 +206,38 @@ def test_module_level_validate_haproxy_config_forwards_partial_fragment():
 
 
 # ──────────────────────────────────────────────────────────────────────
-# Wizard Pydantic gate: ACL `-f` flag must be REJECTED at submit.
+# Issue #38 follow-up: ACL `-f <file>` pattern-file references are
+# ACCEPTED (the Bulgu #12 hard reject was removed — pattern files are
+# operator-managed host files, the agent's pre-reload `haproxy -c`
+# makes a missing file fail safely, and bulk import always accepted
+# `-f`). These tests pin the ACCEPT behaviour.
 # ──────────────────────────────────────────────────────────────────────
 
 
-def test_wizard_pydantic_rejects_acl_with_file_flag():
-    """Pre-fix the wizard's ACL string validator passed
-    `acl name path -i -m reg -f /path` straight through. Apply-time
-    HAProxy `-c` then failed with "failed to open pattern file".
-    Pin that the validator now rejects `-f` at submit.
+def test_wizard_pydantic_accepts_acl_with_file_flag():
+    """Issue #38 follow-up — the wizard's ACL string validator must
+    ACCEPT `-f <file>` pattern-file references (Bulgu #12 reject
+    removed). Operators with large host-managed IP blacklists rely
+    on this in production.
     """
     from models.site_wizard import FrontendStep
 
-    # Minimal valid wizard frontend kwargs — only the offending
-    # acl_rules entry should trigger the failure.
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=["acl1 path -i -m reg -f /path"],
     )
-    from pydantic import ValidationError
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(**fe_kwargs)
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Bulgu #12 regression: ACL -f flag must be rejected with a "
-        f"clear pattern-file error. Got: {msg}"
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"], (
+        "ACL `-f` rule must round-trip verbatim through the wizard model"
     )
 
 
 @pytest.mark.parametrize(
     "rule",
     [
-        # Various spacing / position variants the regex must catch.
+        # Various spacing / position variants must all be accepted.
         "acl1 path -f /etc/haproxy/list",
         "acl1 path -i -f /tmp/x.lst",
         "acl1 src -f /etc/haproxy/admins.lst",
@@ -249,23 +246,19 @@ def test_wizard_pydantic_rejects_acl_with_file_flag():
         "acl1 path -f",
     ],
 )
-def test_wizard_pydantic_rejects_acl_with_file_flag_variants(rule):
-    """Every spacing / position variant the operator might type must
-    be rejected. Pinned defensively so the regex never accidentally
-    relaxes to "only matches trailing -f".
-    """
+def test_wizard_pydantic_accepts_acl_with_file_flag_variants(rule):
+    """Every spacing / position variant must be accepted verbatim
+    (Issue #38 follow-up — no `-f` shape may be rejected)."""
     from models.site_wizard import FrontendStep
-    from pydantic import ValidationError
 
-    fe_kwargs = dict(
+    fe = FrontendStep(
         name="fe1",
         mode="http",
         bind_address="*",
         bind_port=80,
         acl_rules=[rule],
     )
-    with pytest.raises(ValidationError):
-        FrontendStep(**fe_kwargs)
+    assert fe.acl_rules == [rule]
 
 
 def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
@@ -291,42 +284,29 @@ def test_wizard_pydantic_does_not_falsely_match_dash_f_inside_token():
     assert len(fe.acl_rules) == 3
 
 
-def test_manual_frontend_validator_rejects_acl_with_file_flag():
-    """Parity check: the manual Frontend API
-    (`models/frontend.py::validate_acl_rules`) must apply the same
-    `-f` rejection. Operators see consistent behaviour from both the
-    wizard and the per-entity frontend page.
+def test_manual_frontend_validator_accepts_acl_with_file_flag():
+    """Parity check (Issue #38 follow-up): the manual Frontend API
+    (`models/frontend.py::validate_acl_rules`) must ACCEPT `-f`
+    pattern-file references, same as the wizard and bulk import.
     """
     from models.frontend import FrontendConfig
-    from pydantic import ValidationError
 
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendConfig(
-            name="fe1",
-            bind_port=80,
-            mode="http",
-            acl_rules=["acl1 path -i -m reg -f /path"],
-        )
-    msg = str(exc_info.value)
-    assert "-f" in msg or "pattern-file" in msg.lower(), (
-        f"Manual frontend API parity regression: ACL -f flag must be "
-        f"rejected. Got: {msg}"
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        acl_rules=["acl1 path -i -m reg -f /path"],
     )
+    assert fe.acl_rules == ["acl1 path -i -m reg -f /path"]
 
 
-def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
-    """Round-3 audit extension — structured redirect dicts (the
-    alternative shape that `models/site_wizard.py::_validate_redirect_rules`
-    accepts alongside legacy strings) also flow through to
-    `services/haproxy_config.py::_format_redirect_rule` and emit
-    their `condition` / `target` verbatim into the rendered HAProxy
-    directive. Without the dict-aware reject the visual builder's
-    `-f` block could be bypassed by hand-crafting a dict payload
-    against the API — recreating the same `failed to open pattern
-    file` failure at apply time.
+def test_wizard_pydantic_accepts_structured_redirect_dict_with_file_flag():
+    """Issue #38 follow-up — structured redirect dicts carrying `-f`
+    pattern-file references in `condition`/`target` are ACCEPTED
+    (the Bulgu #12 dict-aware reject was removed together with the
+    string-rule reject).
     """
-    from models.site_wizard import FrontendStep, BackendStep
-    from pydantic import ValidationError
+    from models.site_wizard import FrontendStep
 
     fe_kwargs = dict(
         name="fe1",
@@ -334,37 +314,32 @@ def test_wizard_pydantic_rejects_structured_redirect_dict_with_file_flag():
         mode="http",
     )
 
-    # `condition` carrying `-f` must be rejected.
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "scheme",
-                    "target": "https",
-                    "condition": "if { src -f /etc/haproxy/admins.lst }",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `condition` carrying `-f` is accepted.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "scheme",
+                "target": "https",
+                "condition": "if { src -f /etc/haproxy/admins.lst }",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["condition"] == "if { src -f /etc/haproxy/admins.lst }"
 
-    # `target` carrying `-f` must also be rejected (defence-in-depth
-    # for hand-crafted payloads).
-    with pytest.raises(ValidationError) as exc_info:
-        FrontendStep(
-            **fe_kwargs,
-            redirect_rules=[
-                {
-                    "type": "location",
-                    "target": "/foo -f /tmp/x.lst",
-                }
-            ],
-        )
-    msg = str(exc_info.value)
-    assert "pattern-file" in msg.lower() or "-f" in msg, msg
+    # `target` carrying `-f` is accepted too.
+    fe = FrontendStep(
+        **fe_kwargs,
+        redirect_rules=[
+            {
+                "type": "location",
+                "target": "/foo -f /tmp/x.lst",
+            }
+        ],
+    )
+    assert fe.redirect_rules[0]["target"] == "/foo -f /tmp/x.lst"
 
-    # Clean structured dict still passes — no false positive.
+    # Clean structured dict still passes.
     FrontendStep(
         **fe_kwargs,
         redirect_rules=[
@@ -667,8 +642,8 @@ def test_user_reported_wizard_config_emits_no_false_warnings():
     zero WARNINGs from the directives we expanded.
     """
     # Distilled from the user's bulk-site-create snapshot, minus the
-    # `-f` ACL (which the new Pydantic gate rejects before this
-    # validator ever runs).
+    # `-f` ACL (accepted since the Issue #38 follow-up, but irrelevant
+    # to the directive-expansion warnings this test pins).
     config = """# ─── Wizard candidate fragment (dry-run preview) ───
 frontend fe-site1
     bind *:80
@@ -5328,7 +5303,22 @@ def test_bulgu62_update_path_grandfathers_unchanged_rule():
         fe, grandfathered_signatures=grand,
     )
     assert len(warnings) == 1
-    assert "grandfathered" in warnings[0].lower() or "pre-dated" in warnings[0].lower()
+    # Bulgu #83 (round-23 audit) — the warning was reworded from
+    # "Grandfathered ... pre-dated this validation" to a clearer
+    # "rule was not modified by this edit" phrasing that also
+    # echoes the verbatim rule body. Accept either the legacy
+    # markers or the new ones so the contract is signal-not-
+    # wording.
+    w_lower = warnings[0].lower()
+    assert (
+        "not modified by this edit" in w_lower
+        or "grandfathered" in w_lower
+        or "pre-dated" in w_lower
+    ), warnings[0]
+    # The verbatim rule must appear in the warning body so the
+    # operator can identify the offending entry without opening
+    # the ACL Builder cards.
+    assert stale in warnings[0], warnings[0]
 
 
 def test_bulgu62_update_path_still_rejects_new_contradiction():
@@ -6267,3 +6257,623 @@ def test_bulgu82_generate_install_script_validates_cluster():
     window = src[fn_start:fn_start + 5000]
     assert "validate_user_cluster_access" in window
     assert "Bulgu #82 (round-22 audit)" in window
+
+
+# ---- Bulgu #83 — grandfathered contradiction warning wording ----
+
+
+def test_bulgu83_warning_includes_verbatim_rule_text():
+    """The PUT-path grandfather warning must echo the verbatim
+    offending rule string so the operator can identify the
+    offender without opening the ACL Builder cards. Pre-fix the
+    warning only said "1 legacy rule" via the FE toast and
+    "Grandfathered <label> entry contains a self-contradictory
+    X !X condition that pre-dated this validation" via the server
+    JSON payload — both omitted the actual rule body, forcing
+    the operator to open the modal and hunt for the dead-code
+    entry."""
+    from models.frontend import FrontendConfig
+    from routers.frontend import (
+        _enforce_routing_rule_contradictions,
+        _rule_to_signature,
+    )
+
+    stale = "be-x if acl1 !acl1"
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        use_backend_rules=[stale],
+    )
+    warnings = _enforce_routing_rule_contradictions(
+        fe, grandfathered_signatures={_rule_to_signature(stale)},
+    )
+    assert len(warnings) == 1
+    # Verbatim rule body appears in the warning.
+    assert stale in warnings[0]
+    # The wording no longer uses the operator-unfriendly
+    # "Grandfathered" lead-in; the message states what the
+    # branch actually knows ("not modified by this edit").
+    assert "not modified by this edit" in warnings[0].lower()
+
+
+def test_bulgu83_dict_redirect_warning_includes_signature_snippet():
+    """Dict-shaped redirect rules with a contradictory `condition`
+    must still surface the offending signature in the warning
+    (truncated to 160 chars to bound the toast length). Pin the
+    truncation contract so a future refactor doesn't accidentally
+    grow the warning into a multi-KB blob."""
+    from models.frontend import FrontendConfig
+    from routers.frontend import (
+        _enforce_routing_rule_contradictions,
+        _rule_to_signature,
+    )
+
+    # Note: the redirect condition itself carries the contradiction;
+    # the dict wrapper is what the wizard emits.
+    bad_dict = {
+        "type": "scheme",
+        "scheme": "https",
+        "code": 301,
+        "condition": "if acl1 !acl1",
+    }
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        redirect_rules=[bad_dict],
+    )
+    warnings = _enforce_routing_rule_contradictions(
+        fe, grandfathered_signatures={_rule_to_signature(bad_dict)},
+    )
+    assert len(warnings) == 1
+    # Truncated signature (up to 160 chars) must appear in body.
+    assert "acl1" in warnings[0]
+    assert "!acl1" in warnings[0]
+    assert "redirect_rules" in warnings[0]
+
+
+def test_bulgu83_static_marker_in_fe_warning_toast():
+    """Front-end pin — the FrontendManagement.js toast no longer
+    calls these rules "legacy" and now lists each offending rule
+    body. Static-source check so a refactor that re-introduces
+    the misleading wording or drops the rule snippets is caught.
+
+    Skipped automatically when the test runs inside the backend
+    Dockerfile build context (which only copies `backend/` and
+    therefore has no `frontend/` tree next to it). This mirrors
+    the guard already used by every other front-end static-source
+    pin in this file (e.g. lines 583-592, 795-810, 833-845, etc.)
+    — Bulgu #83's pin was missing it, which broke the corporate
+    CI's `RUN python -m pytest` step in the backend Docker build
+    immediately after Round-23 went live.
+    """
+    fm_path = (
+        _BACKEND_DIR.parent
+        / "frontend"
+        / "src"
+        / "components"
+        / "FrontendManagement.js"
+    )
+    if not fm_path.exists():
+        pytest.skip(
+            f"frontend not present at {fm_path}; running in backend-only "
+            "container is expected — skip JS source pin"
+        )
+    fm_src = fm_path.read_text()
+    assert "Bulgu #83 (round-23 audit)" in fm_src
+    # No more `legacy routing/redirect rule(s)` wording.
+    assert "legacy routing/redirect rule(s) with a self-contradictory" not in fm_src
+    # The new toast wires the offending rule snippets into the
+    # message body via `ruleSnippets`.
+    assert "ruleSnippets" in fm_src
+    # And it re-surfaces server-emitted warnings as a safety net.
+    assert "serverWarnings" in fm_src
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #84 + #85 (round-23 audit — comprehensive Site Wizard live-test pass)
+# ----------------------------------------------------------------------------
+
+
+def test_bulgu84_acme_diagnostics_uses_last_seen_column():
+    """Bulgu #84 (round-23 audit) — duplicate pin in this file so the
+    full bulgu-pin suite has a one-stop reference; the original sits
+    in `test_acme_diagnostics.py::test_bulgu84_check_agents_uses_
+    last_seen_column` and asserts the AsyncMock call shape.
+
+    This duplicate is a STATIC-SOURCE check that's cheaper to grep:
+    we open the service file and verify the wrong column name is
+    not back in the SQL, plus the Bulgu marker is present so a
+    future refactor that drops the comment also fails.
+    """
+    svc_path = _BACKEND_DIR / "services" / "acme_diagnostics.py"
+    src = svc_path.read_text()
+    assert "Bulgu #84 (round-23 audit)" in src, (
+        "Bulgu #84 marker missing from acme_diagnostics.py — the "
+        "fix comment was removed but the underlying SQL change "
+        "may have been reverted too. Re-verify check_agents."
+    )
+    # Strip comments before scanning for the wrong column — the fix
+    # comment legitimately mentions ``a.last_heartbeat`` as the
+    # pre-fix symptom. We only care about NON-comment occurrences.
+    non_comment_lines = "\n".join(
+        ln for ln in src.splitlines() if not ln.lstrip().startswith("#")
+    )
+    # The canonical SELECT inside check_agents must use a.last_seen.
+    assert "SELECT a.id, a.hostname, a.status, a.last_seen" in non_comment_lines, (
+        "acme_diagnostics.check_agents must SELECT a.last_seen "
+        "(the canonical agents-table column). Pre-fix this used "
+        "a non-existent a.last_heartbeat and every preflight-acme "
+        "call 500'd in production."
+    )
+    assert "a.last_heartbeat" not in non_comment_lines, (
+        "acme_diagnostics still references a.last_heartbeat in non-"
+        "comment code — this column does NOT exist on the agents "
+        "table. Use a.last_seen."
+    )
+
+
+def test_bulgu87_wizard_create_endpoint_rate_limited():
+    """Bulgu #87 (round-23 audit) — `POST /api/sites` must be
+    rate-limited, matching the existing 5/min caps on the
+    /preview and /preflight-acme sibling endpoints.
+
+    Pre-fix the wizard's actual create endpoint had NO rate cap:
+      - /api/sites/preflight-acme  → rate-limited (site_acme_preflight)
+      - /api/sites/preview         → rate-limited (site_previewed)
+      - /api/sites (POST, create)  → UNLIMITED ← gap
+
+    Round-3 live testing fired 10 wizard creates inside one
+    second against demo-cluster1; the response codes were
+    200/409/409/...409/200/409... — no 429 ever fired. With
+    Bulgu #86 fixed (version_name collision), the UNIQUE-error
+    side-effect that ACCIDENTALLY rate-limited rapid-fire creates
+    disappears too, so an authenticated user/script can now
+    create entities until the cluster apply queue clogs.
+
+    The fix calls `_enforce_rate_limit(conn, user_id,
+    "wizard_create_site")` early in create_site (after auth +
+    cluster-access checks). The action name must match what the
+    router itself logs (`wizard_create_site` — see
+    `_log_user_activity(... action="wizard_create_site", ...)`
+    in the same router); the activity_logger middleware
+    intentionally skips this endpoint to avoid double-logging,
+    so the rate-limit must reference the router-emitted action.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #87 (round-23 audit)" in sw_src, (
+        "Bulgu #87 marker missing — the wizard create rate-limit "
+        "may have been reverted."
+    )
+    # The create_site function body must call _enforce_rate_limit
+    # with the canonical create action name.
+    import re as _re
+    m = _re.search(
+        r"async def create_site\([^)]*\)[^{]*:(.*?)(?=\nasync def |\n@router\.|\Z)",
+        sw_src,
+        _re.DOTALL,
+    )
+    assert m, "could not locate create_site function body in site_wizard.py"
+    fn_body = m.group(1)
+    assert (
+        '_enforce_rate_limit(conn, user_id, "wizard_create_site")' in fn_body
+        or "_enforce_rate_limit(conn, user_id, 'wizard_create_site')" in fn_body
+    ), (
+        "create_site must call _enforce_rate_limit(... "
+        '"wizard_create_site") so rapid-fire POST /api/sites '
+        "calls hit a 429 cap; pre-fix the endpoint was unlimited."
+    )
+
+
+def test_bulgu86_wizard_version_name_has_unique_suffix():
+    """Bulgu #86 (round-23 audit) — pin the version-name generation so
+    two wizard POST /api/sites calls submitted within the SAME epoch
+    second cannot collide on `config_versions(cluster_id, version_name)`.
+
+    Pre-fix `version_name = f"bulk-site-create-{int(time.time())}"`
+    had seconds resolution. Round-3 live testing reproduced the
+    failure with a back-to-back wizard automation: the second call
+    returned 409 with the misleading
+
+        "A wizard entity with this name already exists on the
+         cluster (UNIQUE constraint). Pick a different name."
+
+    fall-through detail, even though the operator-supplied
+    backend / frontend / SSL names were genuinely unique — the
+    real collision was on the auto-generated version name. The
+    operator who tried to rename the wizard inputs would still
+    hit the same error and have no way to make progress until the
+    epoch second ticked over.
+
+    Fix: append a 6-hex-char UUID suffix
+    (`bulk-site-create-{ts}-{uuid.uuid4().hex[:6]}`) so the
+    suffix space is 16M and birthday-collision-proof at any
+    realistic per-second request rate. The `bulk-site-create-`
+    prefix is preserved so reject_pending_changes /
+    restore-from-prior-version paths (cluster.py:4179 prefix scan)
+    keep working.
+
+    Static-source pin so any future refactor that strips the
+    suffix fails this test.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #86 (round-23 audit)" in sw_src, (
+        "Bulgu #86 marker missing — the version_name uniqueness "
+        "suffix may have been reverted."
+    )
+    # The fixed form references uuid.uuid4().hex[:6] (or similar
+    # length-bounded random suffix) appended to the ts. A pure
+    # `f"bulk-site-create-{ts}"` line (without any suffix
+    # interpolation after ts) is the regressed form.
+    import re as _re
+    # Find the wizard's version_name assignment. We allow flexible
+    # spacing / quoting around the f-string but require the suffix
+    # interpolation token immediately after `{ts}-`.
+    has_suffix = bool(_re.search(
+        r'version_name\s*=\s*f"bulk-site-create-\{ts\}-\{[^}]+\}"',
+        sw_src,
+    ))
+    has_regressed = bool(_re.search(
+        r'version_name\s*=\s*f"bulk-site-create-\{ts\}"\s*$',
+        sw_src,
+        _re.MULTILINE,
+    ))
+    assert has_suffix, (
+        "wizard's version_name must append a unique suffix "
+        "(e.g. uuid.uuid4().hex[:6]) after `{ts}` so back-to-back "
+        "wizard creates within the same epoch second cannot "
+        "collide on config_versions UNIQUE."
+    )
+    assert not has_regressed, (
+        "wizard's version_name regressed to seconds-only resolution "
+        "— this re-introduces Bulgu #86 (UNIQUE collision on rapid "
+        "back-to-back creates)."
+    )
+
+
+def test_bulgu85_unique_violation_handler_extracts_constraint_name():
+    """Bulgu #85 (round-23 audit) — the wizard's UniqueViolationError
+    handler must surface the offending constraint name in the operator-
+    visible 409 detail so debugging a "wizard entity with this name
+    already exists" toast doesn't require shell access to the server
+    logs.
+
+    Pre-fix the handler had THREE branches:
+      * backends_name_cluster_id_key  → "backend with this name"
+      * frontends_name_cluster_id_key → "frontend with this name"
+      * else                          → generic "wizard entity"
+    Any other constraint (SSL cert UNIQUE, backend_servers UNIQUE,
+    config_versions UNIQUE) fell through to the generic message,
+    leaving the operator with NO actionable hint and requiring an
+    on-call engineer to ssh into the API pod and tail `logger.info`
+    output to find ``constraint=<name>`` in the asyncpg traceback.
+
+    The fix:
+      1. Echoes the entity NAME (body.backend.name, body.frontend.name,
+         body.ssl.name) so the operator can match toast to wizard input.
+      2. Adds dedicated branches for `ssl_certificates_*` and
+         `backend_servers_*` constraints with actionable hints.
+      3. In the fall-through ELSE branch, echoes the constraint name
+         from `uve.constraint_name` (or substring scan as fallback)
+         so an unknown constraint still gives the engineer a stable
+         schema identifier to grep.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #85 (round-23 audit)" in sw_src, (
+        "Bulgu #85 marker missing from site_wizard.py UniqueViolation "
+        "handler — the constraint-echoing fix may have been reverted."
+    )
+    # Entity-name echoing in branch detail bodies.
+    assert "body.backend.name" in sw_src and (
+        "A backend named '{body.backend.name}'" in sw_src
+        or "A backend named '\"{body.backend.name}\"" in sw_src
+        or "A backend named '" in sw_src and "body.backend.name" in sw_src
+    )
+    assert "A frontend named '{body.frontend.name}'" in sw_src or (
+        "A frontend named '" in sw_src and "body.frontend.name" in sw_src
+    )
+    # SSL cert and backend_servers branches present.
+    assert "ssl_certificates" in sw_src
+    assert "backend_servers" in sw_src
+    # Fall-through echoes constraint name.
+    assert "constraint=" in sw_src or "constraint_name" in sw_src
+    # The asyncpg attribute is preferred over str(uve) scanning.
+    assert 'getattr(uve, "constraint_name"' in sw_src, (
+        "Fix should prefer asyncpg's structured constraint_name "
+        "attribute over substring scanning str(uve), which is the "
+        "human-formatted DETAIL line and may vary across server "
+        "versions / locales."
+    )
+
+
+# ----- Bulgu #88 / #89 (round-24 audit) ----------------------------------
+#
+# `POST /api/sites/preview` skipped the SSL-cert cluster-RBAC gate that
+# `POST /api/sites` (create) enforces via `select_existing_cert()`. The
+# `select_existing_cert()` helper encodes "global cert OR junction-bound
+# to <cluster_id>"; pre-fix the preview accepted ssl_certificate_id values
+# bound to OTHER clusters and returned the full rendered `would_create`
+# envelope — a tenant-boundary information leak for the cert id and the
+# rendered HAProxy config snippet. The fix mirrors the create-time check
+# in the preview path: same predicate, same 400 status, same error text.
+
+def test_bulgu88_preview_enforces_ssl_cert_cluster_rbac():
+    """Preview must reject `ssl.ssl_certificate_id` referencing a cert
+    that is bound to a DIFFERENT cluster — same RBAC predicate the
+    create endpoint enforces. Static source check: the preview body
+    invokes `select_existing_cert` with `body.ssl.ssl_certificate_id`
+    and raises 400 with a "not found / inactive / not bound" hint."""
+    import os
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+
+    # Locate the preview function.
+    assert "async def preview_create(" in sw_src
+    preview_start = sw_src.index("async def preview_create(")
+    # Bound the slice at the next top-level `async def` so we don't
+    # match the create_site copy (which is at line ~1900 and would
+    # cause this test to trivially pass even if preview was unfixed).
+    next_def = sw_src.index("\nasync def ", preview_start + 1)
+    preview_body = sw_src[preview_start:next_def]
+
+    # The preview must call select_existing_cert.
+    assert "select_existing_cert(" in preview_body, (
+        "Preview path must mirror create_site's cluster-RBAC gate by "
+        "invoking select_existing_cert(conn, ssl_certificate_id, "
+        "cluster_id). Pre-fix preview returned 200 with the full "
+        "rendered would_create envelope when the cert belonged to a "
+        "different cluster — a tenant-boundary information leak."
+    )
+
+    # The check must be wired to the body.ssl.ssl_certificate_id field
+    # (not some unrelated cert reference).
+    assert "body.ssl.ssl_certificate_id" in preview_body, (
+        "Preview RBAC check must reference body.ssl.ssl_certificate_id"
+    )
+
+    # Must raise 400 (consistent with create-time message).
+    assert "status_code=400" in preview_body
+    assert "not found / inactive / not bound" in preview_body or (
+        "not found / inactive" in preview_body
+        and "not bound" in preview_body
+    ), (
+        "Preview should raise 400 with the same 'not found / inactive "
+        "/ not bound to this cluster' hint create_site emits, so wizard "
+        "clients can match the message uniformly."
+    )
+
+
+def test_bulgu89_preview_enforces_per_server_ca_bundle_cluster_rbac():
+    """Preview must also enforce per-server `ssl_certificate_id` (CA
+    bundle) cluster-RBAC. Pre-fix only `create_site` checked this — a
+    direct API caller could submit a preview with
+    `servers[i].ssl_certificate_id=<cert from another cluster>` and
+    receive the full rendered backend block back, including the
+    `ca-file` directive bound to a cert id the caller's cluster has
+    no junction row for."""
+    import os
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+    preview_start = sw_src.index("async def preview_create(")
+    next_def = sw_src.index("\nasync def ", preview_start + 1)
+    preview_body = sw_src[preview_start:next_def]
+
+    # Must iterate body.servers and check per-server ssl_certificate_id.
+    assert "body.servers" in preview_body, (
+        "Preview must iterate body.servers to validate each "
+        "server.ssl_certificate_id reference"
+    )
+    assert "ssl_certificate_id" in preview_body
+    # The per-server message must be field-qualified so operators see
+    # WHICH server triggered the gate.
+    assert "servers[" in preview_body and "ssl_certificate_id=" in preview_body, (
+        "Per-server preview error must include the array index "
+        "(servers[i].ssl_certificate_id=<id>) so operators can find "
+        "the offending row in their wizard payload."
+    )
+
+
+# ----- Bulgu #90 (round-24 audit) ----------------------------------------
+#
+# `backend.cookie_name`, `backend.cookie_options`, and `server.cookie_value`
+# only rejected newline characters pre-fix. HAProxy's directive parser
+# tokenises by whitespace and treats `;` as an inline-comment, so a value
+# like `SESS'; DROP TABLE backends; --` rendered as
+# `cookie SESS'; DROP TABLE backends; -- insert indirect nocache` which
+# HAProxy parsed as `cookie SESS'` + comment — silently truncating the
+# operator's persistence options and producing a malformed Set-Cookie
+# header that browsers may drop. Tighten to the conservative
+# alphanumeric+`_.-` set (cookie_name / cookie_value) and the keyword/
+# `=`-bearing set (cookie_options).
+
+def test_bulgu90_cookie_name_rejects_haproxy_parser_hostile_chars():
+    """cookie_name must reject `;` (HAProxy inline comment), whitespace
+    (token delimiter), and other punctuation that breaks the rendered
+    `cookie <name>` directive."""
+    from models.site_wizard import BackendStep
+    import pydantic
+    # Semicolon (HAProxy comment) — used to be silently accepted.
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name="SESS'; DROP TABLE backends; --")
+    # Space (token delimiter) — splits the cookie directive.
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name="SESS ID")
+    # Quote (legal in RFC 6265 but produces ugly Set-Cookie headers
+    # and confuses log scrapers).
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name='SESS"id')
+    # Common-case valid input continues to work.
+    BackendStep(name="be1", cookie_name="SESS_ID-1.app")
+
+
+def test_bulgu90_cookie_options_rejects_parser_hostile_chars():
+    """cookie_options must reject `;` and quoting metacharacters but
+    still accept legitimate `attr SameSite=Lax` style values."""
+    from models.site_wizard import BackendStep
+    import pydantic
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(
+            name="be1", cookie_name="SESS",
+            cookie_options="insert indirect; rm -rf /",
+        )
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(
+            name="be1", cookie_name="SESS",
+            cookie_options='insert "indirect"',
+        )
+    # Valid HAProxy cookie options continue to parse.
+    BackendStep(
+        name="be1", cookie_name="SESS",
+        cookie_options="insert indirect nocache attr SameSite=Lax",
+    )
+
+
+def test_bulgu90_server_cookie_value_rejects_rfc6265_disallowed_chars():
+    """server.cookie_value must reject `;`, whitespace, and other
+    chars RFC 6265 disallows in cookie-values."""
+    from models.site_wizard import ServerStep
+    import pydantic
+    with pytest.raises(pydantic.ValidationError):
+        ServerStep(
+            server_name="s1", server_address="10.0.0.1",
+            server_port=8080, cookie_value="srv1; secure",
+        )
+    with pytest.raises(pydantic.ValidationError):
+        ServerStep(
+            server_name="s1", server_address="10.0.0.1",
+            server_port=8080, cookie_value="srv 1",
+        )
+    # Valid cookie values continue to work.
+    ServerStep(
+        server_name="s1", server_address="10.0.0.1",
+        server_port=8080, cookie_value="srv-1.app_2",
+    )
+
+
+# ----- Bulgu #93 (round-24 audit) ----------------------------------------
+#
+# `GET /api/sites/suggest` produced backend/frontend name suggestions
+# using `c.isalnum()` over the first domain label. Python's `isalnum()`
+# is Unicode-aware and returns True for non-ASCII letters (ü, é, ñ, …),
+# so an operator typing `bücher.example.com` received
+# `backend_name='be-bücher'`. The wizard CREATE path then rejected the
+# very name the SUGGEST endpoint returned, because the entity-name
+# regex (`^[a-zA-Z][a-zA-Z0-9_-]{0,63}$`) and the domain validator are
+# both ASCII-only. The fix converts non-ASCII labels through IDN/
+# punycode FIRST and only then applies ASCII-only sanitisation, so the
+# resulting name is one the operator can submit unchanged.
+
+def test_bulgu93_suggest_idn_unicode_produces_ascii_safe_slug():
+    """`suggest` must produce slugs that the backend/frontend name
+    regex `^[a-zA-Z][a-zA-Z0-9_-]{0,63}$` accepts. Static source check:
+    the suggest function uses an ASCII-only predicate
+    (`c.isascii() and c.isalnum()` etc.) and routes IDN labels through
+    `encode('idna')` to preserve the operator's intent in punycode."""
+    import os, re
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+
+    # Locate the suggest_defaults function body.
+    assert "async def suggest_defaults(" in sw_src
+    start = sw_src.index("async def suggest_defaults(")
+    end = sw_src.index("\n@router.", start + 1)
+    suggest_body = sw_src[start:end]
+
+    # The fix must use IDN/punycode encoding when the input is non-ASCII.
+    assert 'encode("idna")' in suggest_body or "encode('idna')" in suggest_body, (
+        "suggest_defaults must route non-ASCII labels through IDN/"
+        "punycode so the rendered slug matches the ASCII-only entity-"
+        "name regex the create endpoint enforces."
+    )
+
+    # The sanitiser must use an ASCII-only predicate, not Unicode-aware
+    # `isalnum()` alone (which is the pre-fix bug).
+    assert "isascii()" in suggest_body, (
+        "Sanitisation step must explicitly bound to ASCII (via "
+        "`c.isascii()`) so Unicode alphabetics get mapped to '-' rather "
+        "than retained verbatim."
+    )
+
+    # The first occurrence of the Unicode-only `c.isalnum()` (without an
+    # adjacent isascii() guard) must no longer exist in the slug-build
+    # path. We sanity-check by counting the matches of the bare
+    # `c.isalnum()` token within the function body — pre-fix there was
+    # exactly one such occurrence in the slug builder.
+    bare = re.findall(r"\bc\.isalnum\(\)\b", suggest_body)
+    assert all(
+        # Each occurrence must be paired with an `isascii()` guard on
+        # the same line — confirm by inspecting the line context.
+        any("isascii()" in ln for ln in suggest_body.splitlines() if "c.isalnum()" in ln)
+        for _ in bare
+    ), (
+        "Every `c.isalnum()` predicate in the slug builder must be "
+        "AND'd with `c.isascii()` so non-ASCII letters cannot leak "
+        "into the suggested entity names."
+    )
+
+
+def test_bulgu93_suggest_function_behavioural_smoke():
+    """Behavioural check: the same logic, invoked directly, should
+    produce an ASCII-only slug for a Unicode input."""
+    # Replicate the post-fix slug builder inline so the test does not
+    # require an event loop / FastAPI fixture. The point is to confirm
+    # the algorithmic shape matches what the source check above
+    # enforces.
+    def _build_slug(domain: str) -> str:
+        first_label = (
+            domain.replace("*.", "").split(".")[0]
+            if domain.replace("*.", "")
+            else ""
+        )
+        ascii_label = first_label
+        if first_label and not first_label.isascii():
+            try:
+                ascii_label = first_label.encode("idna").decode("ascii")
+            except (UnicodeError, UnicodeDecodeError):
+                ascii_label = "".join(
+                    c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                    else "-"
+                    for c in first_label
+                )
+        slug = (ascii_label or "newhost")[:32].lower()
+        slug = "".join(
+            c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+            else "-"
+            for c in slug
+        )
+        if slug.startswith("_"):
+            slug = "h-" + slug.lstrip("_")
+        if not slug or not slug[0].isalpha():
+            slug = "h-" + slug
+        return slug
+
+    # Unicode IDN — should round-trip through punycode.
+    s = _build_slug("bücher.example.com")
+    # punycode of 'bücher' is 'xn--bcher-kva'
+    assert s == "xn--bcher-kva", f"expected xn--bcher-kva got {s!r}"
+    # The slug must satisfy the entity-name regex.
+    import re
+    assert re.fullmatch(r"^[a-zA-Z][a-zA-Z0-9_-]{0,63}$", s), s
+
+    # Plain ASCII domain still produces the obvious slug.
+    assert _build_slug("example.com") == "example"
+
+    # Uppercase normalises to lowercase.
+    assert _build_slug("EXAMPLE.COM") == "example"
+
+    # Trailing dot (FQDN absolute) stripped.
+    assert _build_slug("example.com.") == "example"
+
+    # Empty / dots-only input falls back to 'newhost'.
+    assert _build_slug("") == "newhost"
+    assert _build_slug("...") == "newhost"

backend/tests/test_keepalive_fault_state.py

@@ -0,0 +1,21 @@
+"""v1.7.6 — guard: the agent must surface keepalived FAULT state.
+
+A VIP whose interface has no usable IPv4 (or whose track-script fails) puts keepalived into
+FAULT — the virtual IP is NOT held. Previously get_keepalive_state only grepped (MASTER|BACKUP),
+so a FAULT'd VIP reported as BACKUP — misleading (looks healthy-ish). It must report FAULT so the
+UI shows it red. Both get_keepalive_state copies (installer + SKIP_TO_DAEMON daemon) must include
+FAULT. Pure source guard."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def test_keepalive_state_detects_fault_in_both_copies():
+    with open(os.path.join(ROOT, "utils", "agent_scripts", "linux_install.sh"), encoding="utf-8") as f:
+        s = f.read()
+    # FAULT added to the state grep in both copies (installer + daemon), both detection methods.
+    assert s.count("MASTER|BACKUP|FAULT") >= 2
+    # and the misleading MASTER|BACKUP-only grep is gone.
+    assert 'grep -oE "(MASTER|BACKUP)"' not in s

backend/tests/test_keepalived_config.py

@@ -0,0 +1,129 @@
+"""Issue #27 (v1.7.0) — unit tests for the keepalived config generator + secret crypto.
+
+Pure-function tests; no DB. Validates the rendered keepalived.conf for MASTER/BACKUP,
+unicast peers, the failover weight arithmetic, the script-security requirements, the
+ownership marker, and Fernet round-trip.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from services import keepalived_config as kc  # noqa: E402
+
+
+VIP = {
+    "id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+    "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True,
+}
+MEMBERS = [
+    {"role": "MASTER", "priority": 150, "network_interface": "eth0", "agent_id": 1, "ip_address": "10.0.0.11"},
+    {"role": "BACKUP", "priority": 100, "network_interface": "eth0", "agent_id": 2, "ip_address": "10.0.0.12"},
+]
+
+
+def _render(this_idx, auth="s3cr3t"):
+    this_agent = MEMBERS[this_idx]
+    peers = [m["ip_address"] for m in MEMBERS if m["agent_id"] != this_agent["agent_id"]]
+    return kc.render_keepalived_conf(vip=VIP, members=MEMBERS, this_agent=this_agent,
+                                     peer_ips=peers, auth_pass_plain=auth)
+
+
+class TestRender:
+    def test_master_state_priority_iface_vrid(self):
+        conf = _render(0)
+        assert "state MASTER" in conf
+        assert "priority 150" in conf
+        assert "interface eth0" in conf
+        assert "virtual_router_id 51" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+    def test_backup_state(self):
+        conf = _render(1)
+        assert "state BACKUP" in conf
+        assert "priority 100" in conf
+
+    def test_unicast_peers(self):
+        # MASTER's config lists the BACKUP as its unicast peer (and its own src ip).
+        conf = _render(0)
+        assert "unicast_src_ip 10.0.0.11" in conf
+        assert "unicast_peer" in conf
+        assert "10.0.0.12" in conf
+
+    def test_script_security_block(self):
+        conf = _render(0)
+        assert "enable_script_security" in conf
+        assert "script_user root" in conf
+
+    def test_ownership_marker(self):
+        assert kc.OWNERSHIP_MARKER in _render(0)
+
+    def test_weight_makes_failed_master_lose(self):
+        # On HAProxy failure the master's effective priority must drop below the backup.
+        conf = _render(0)
+        weight_line = [l for l in conf.splitlines() if l.strip().startswith("weight ")][0]
+        weight = int(weight_line.strip().split()[1])
+        assert 150 + weight < 100, "failed master must fall below every backup"
+        assert "track_script" in conf and "chk_haproxy" in conf
+
+    def test_track_disabled_omits_script(self):
+        vip = {**VIP, "track_haproxy": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain=None)
+        assert "vrrp_script" not in conf
+        assert "track_script" not in conf
+
+    def test_no_auth_when_secret_absent(self):
+        conf = _render(0, auth=None)
+        assert "auth_pass" not in conf
+
+    def test_multicast_omits_unicast(self):
+        vip = {**VIP, "use_unicast": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain="x")
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+
+    def test_single_node_omits_unicast_block(self):
+        # Single-node VIP (no peers): even with use_unicast=True we must NOT emit a bare
+        # `unicast_src_ip`/`unicast_peer` — keepalived treats a unicast keyword with no peers
+        # as deprecated, warns, and falls back to multicast (and `keepalived -t` flags it).
+        # Omitting the block yields a clean multicast config that holds the VIP solo.
+        only = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                 "agent_id": 1, "ip_address": "10.0.0.11"}]
+        conf = kc.render_keepalived_conf(vip=VIP, members=only, this_agent=only[0],
+                                         peer_ips=[], auth_pass_plain=None)
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+        assert "state MASTER" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+
+class TestCheckScript:
+    def test_default_process_name(self):
+        s = kc.build_haproxy_check_script()
+        assert "pidof haproxy" in s
+        assert kc.OWNERSHIP_MARKER in s
+
+    def test_process_name_from_bin_path(self):
+        s = kc.build_haproxy_check_script(bin_path="/opt/hap/sbin/haproxy-ent")
+        assert "pidof haproxy-ent" in s
+
+    def test_malicious_bin_path_falls_back(self):
+        s = kc.build_haproxy_check_script(bin_path="/x/haproxy; rm -rf /")
+        assert "rm -rf" not in s
+        assert "pidof haproxy" in s
+
+
+class TestSecretCrypto:
+    def test_roundtrip(self):
+        kc.reset_fernet_for_tests()
+        token = kc.encrypt_vrrp_secret("s3cr3t")
+        assert token != "s3cr3t"
+        assert kc.decrypt_vrrp_secret(token) == "s3cr3t"
+
+    def test_decrypt_garbage_returns_none(self):
+        kc.reset_fernet_for_tests()
+        assert kc.decrypt_vrrp_secret("not-a-fernet-token") is None

backend/tests/test_keepalived_parser.py

@@ -0,0 +1,341 @@
+"""Issue #27 follow-up (v1.10.4) — unit tests for parsing an EXISTING keepalived.conf so a
+hand-maintained VIP can be adopted.
+
+Pure-function tests; no DB, no network. The parser exists because the heartbeat carries only
+the VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs eleven
+fields — and because adoption REPLACES the operator's file, so anything our renderer cannot
+reproduce has to be reported as a blocker rather than silently dropped.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-keepalived-parser-tests")
+
+from services import keepalived_config as kc  # noqa: E402
+from services.keepalived_parser import (  # noqa: E402
+    KeepalivedParseError, analyse_keepalived_conf, build_adoption_candidate,
+    parse_keepalived_conf,
+)
+
+
+# A realistic hand-maintained config: two nodes, unicast VRRP, password auth, HAProxy check.
+HANDWRITTEN = """\
+! Configuration File for keepalived
+global_defs {
+    enable_script_security
+    script_user root
+}
+
+vrrp_script chk_haproxy {
+    script "/etc/keepalived/check_haproxy.sh"
+    interval 2
+    weight -21
+}
+
+vrrp_instance VI_1 {
+    state MASTER
+    interface eth0        # public leg
+    virtual_router_id 51
+    priority 150
+    advert_int 1
+    authentication {
+        auth_type PASS
+        auth_pass s3cr3t
+    }
+    unicast_src_ip 10.0.0.11
+    unicast_peer {
+        10.0.0.12
+    }
+    virtual_ipaddress {
+        10.0.0.100/24 dev eth0
+    }
+    track_script {
+        chk_haproxy
+    }
+}
+"""
+
+
+def _only_candidate(text):
+    parsed = parse_keepalived_conf(text)
+    assert len(parsed["instances"]) == 1
+    return build_adoption_candidate(parsed, parsed["instances"][0])
+
+
+def test_parses_a_handwritten_config_into_model_fields():
+    cand = _only_candidate(HANDWRITTEN)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["blockers"] == []
+    assert cand["vip"] == {
+        "virtual_ip": "10.0.0.100",
+        "prefix_length": 24,
+        "virtual_router_id": 51,
+        "advert_int": 1,
+        "use_unicast": True,
+        "track_haproxy": True,
+        "auth_pass": "s3cr3t",
+    }
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["peers"] == ["10.0.0.12"] and cand["unicast_src_ip"] == "10.0.0.11"
+    assert cand["defaulted"] == []          # every value came from the file, nothing assumed
+
+
+def test_comment_and_layout_variants():
+    # `!` and `#` both start comments; a block may open and close on one line; a quoted
+    # script path keeps its spaces. None of this may change the parse.
+    text = """\
+#!/not/a/shebang — this whole line is a comment
+vrrp_script chk { script "/opt/my scripts/chk.sh" }
+vrrp_instance VI_1 { state BACKUP
+    interface eth1   ! trailing bang comment
+    virtual_router_id 7
+    priority 90
+    virtual_ipaddress { 192.168.5.9/32 dev eth1 }
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert parsed["scripts"]["chk"]["script"] == "/opt/my scripts/chk.sh"
+    inst = parsed["instances"][0]
+    assert inst["state"] == "BACKUP" and inst["interface"] == "eth1"
+    assert inst["virtual_router_id"] == 7 and inst["priority"] == 90
+    assert inst["virtual_ips"] == [
+        {"address": "192.168.5.9", "prefix_length": 32, "dev": "eth1", "extra": []}
+    ]
+
+
+def test_documented_defaults_are_applied_and_flagged():
+    # keepalived's own defaults for absent directives. Applying them re-renders the same
+    # behaviour, so they are allowed — but the UI must be able to say they were assumed.
+    text = """\
+vrrp_instance VI_1 {
+    interface eth0
+    virtual_router_id 12
+    virtual_ipaddress { 10.1.1.5/24 dev eth0 }
+}
+"""
+    cand = _only_candidate(text)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["member"]["role"] == "BACKUP" and cand["member"]["priority"] == 100
+    assert cand["vip"]["advert_int"] == 1
+    assert sorted(cand["defaulted"]) == ["advert_int", "priority", "state"]
+    # No authentication block and no track_script — both legal, both faithfully represented.
+    assert cand["vip"]["auth_pass"] is None and cand["vip"]["track_haproxy"] is False
+    assert cand["vip"]["use_unicast"] is False
+
+
+def _blockers_for(text):
+    return " | ".join(_only_candidate(text)["blockers"])
+
+
+def test_directives_we_cannot_render_block_adoption():
+    # THE central safety property: adoption overwrites the file, so a failover hook we do not
+    # render would be destroyed. It must stop the flow, not warn.
+    text = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _blockers_for(text)
+    assert "notify_master" in blockers and "would delete it" in blockers
+    assert _only_candidate(text)["adoptable"] is False
+
+
+def test_multiple_addresses_in_one_instance_block_adoption():
+    text = HANDWRITTEN.replace("        10.0.0.100/24 dev eth0",
+                               "        10.0.0.100/24 dev eth0\n        10.0.0.101/24 dev eth0")
+    blockers = _blockers_for(text)
+    assert "2 addresses" in blockers and "10.0.0.101" in blockers
+
+
+def test_missing_vrid_blocks_adoption_with_the_split_brain_reason():
+    text = HANDWRITTEN.replace("    virtual_router_id 51\n", "")
+    blockers = _blockers_for(text)
+    assert "no virtual_router_id" in blockers and "separate VRRP domains" in blockers
+
+
+def test_missing_prefix_blocks_adoption():
+    # Our renderer always writes an explicit prefix; guessing one would change the netmask of a
+    # live VIP, so the operator has to state it.
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _blockers_for(text)
+    assert "no explicit prefix length" in blockers
+
+
+def test_address_on_a_different_dev_blocks_adoption():
+    text = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100/24 dev eth1")
+    blockers = _blockers_for(text)
+    assert "dev eth1" in blockers and "interface eth0" in blockers
+
+
+def test_foreign_track_script_blocks_adoption():
+    text = HANDWRITTEN.replace("        chk_haproxy", "        chk_custom")
+    blockers = _blockers_for(text)
+    assert "chk_custom" in blockers and "replaced by OpenManager" in blockers
+
+
+def test_unsupported_auth_type_blocks_adoption():
+    text = HANDWRITTEN.replace("auth_type PASS", "auth_type AH")
+    assert "auth_type AH" in _blockers_for(text)
+
+
+def test_fractional_advert_int_blocks_adoption():
+    # Rounding 0.5s to 1s changes VRRP timing, so adopt-and-alter is not acceptable.
+    text = HANDWRITTEN.replace("advert_int 1", "advert_int 0.5")
+    blockers = _blockers_for(text)
+    assert "advert_int 0.5" in blockers and "fractional" in blockers
+
+
+def test_half_configured_unicast_blocks_adoption():
+    text = HANDWRITTEN.replace("    unicast_peer {\n        10.0.0.12\n    }\n", "")
+    assert "unicast_src_ip without unicast_peer" in _blockers_for(text)
+
+
+def test_sync_group_and_lvs_sections_block_adoption():
+    text = HANDWRITTEN + """
+vrrp_sync_group VG1 {
+    group {
+        VI_1
+    }
+}
+virtual_server 10.0.0.100 80 {
+    lb_algo rr
+}
+"""
+    parsed = parse_keepalived_conf(text)
+    assert [g["name"] for g in parsed["sync_groups"]] == ["VG1"]
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "vrrp_sync_group VG1" in directives and "virtual_server" in directives
+    # Both are top-level, so EVERY candidate in the file is blocked — a sync group changes
+    # failover semantics for the instances it groups.
+    cand = build_adoption_candidate(parsed, parsed["instances"][0])
+    assert cand["adoptable"] is False
+
+
+def test_extra_global_defs_are_reported_as_losses():
+    text = HANDWRITTEN.replace("    script_user root",
+                               "    script_user root\n    router_id LVS_DEVEL")
+    parsed = parse_keepalived_conf(text)
+    directives = " ".join(d["directive"] for d in parsed["unsupported"])
+    assert "global_defs/router_id LVS_DEVEL" in directives
+    assert parsed["global_defs"]["router_id"] == "LVS_DEVEL"
+
+
+def test_multiple_instances_yield_one_candidate_each():
+    text = HANDWRITTEN + """
+vrrp_instance VI_2 {
+    state BACKUP
+    interface eth0
+    virtual_router_id 52
+    priority 100
+    advert_int 1
+    virtual_ipaddress { 10.0.0.200/24 dev eth0 }
+}
+"""
+    analysed = analyse_keepalived_conf(text)
+    assert analysed["instance_count"] == 2
+    names = [c["instance_name"] for c in analysed["candidates"]]
+    assert names == ["VI_1", "VI_2"]
+    assert [c["vip"]["virtual_ip"] for c in analysed["candidates"]] == ["10.0.0.100", "10.0.0.200"]
+    assert all(c["adoptable"] for c in analysed["candidates"])
+
+
+def test_unbalanced_braces_raise():
+    for bad in ("vrrp_instance VI_1 {\n    state MASTER\n", "}\n"):
+        raised = False
+        try:
+            parse_keepalived_conf(bad)
+        except KeepalivedParseError:
+            raised = True
+        assert raised, f"should have raised for {bad!r}"
+
+
+def test_our_own_render_round_trips_with_zero_blockers():
+    """The invariant that keeps the parser honest: a config WE generated must parse back into
+    the same model with nothing unsupported. If a future change to render_keepalived_conf emits
+    a directive the parser does not know, this fails — instead of adoption silently reporting
+    that OpenManager's own output is unadoptable."""
+    vip = {"id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+           "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True}
+    members = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                "agent_id": 1, "ip_address": "10.0.0.11"},
+               {"role": "BACKUP", "priority": 100, "network_interface": "eth0",
+                "agent_id": 2, "ip_address": "10.0.0.12"}]
+    rendered = kc.render_keepalived_conf(
+        vip=vip, members=members, this_agent=members[0],
+        peer_ips=["10.0.0.12"], auth_pass_plain="s3cr3t")
+
+    cand = _only_candidate(rendered)
+    assert cand["adoptable"] is True, cand["blockers"]
+    assert cand["vip"]["virtual_ip"] == vip["virtual_ip"]
+    assert cand["vip"]["prefix_length"] == vip["prefix_length"]
+    assert cand["vip"]["virtual_router_id"] == vip["virtual_router_id"]
+    assert cand["vip"]["track_haproxy"] is True and cand["vip"]["use_unicast"] is True
+    assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
+    assert cand["vip"]["auth_pass"] == "s3cr3t"
+
+    # And the same for the no-auth / multicast / untracked shape, which renders fewer blocks.
+    plain = kc.render_keepalived_conf(
+        vip={**vip, "use_unicast": False, "track_haproxy": False},
+        members=members, this_agent=members[1], peer_ips=[], auth_pass_plain=None)
+    cand2 = _only_candidate(plain)
+    assert cand2["adoptable"] is True, cand2["blockers"]
+    assert cand2["vip"]["use_unicast"] is False and cand2["vip"]["track_haproxy"] is False
+    assert cand2["vip"]["auth_pass"] is None
+
+
+# --- v1.10.4 adoption gate: which blockers an operator may resolve --------------------------
+
+
+def test_only_prefix_and_data_loss_are_waivable():
+    from services.keepalived_parser import remaining_blockers
+
+    loss = "line 9: `notify_master \"/x.sh\"` — OpenManager's renderer cannot reproduce this, so adopting would delete it"
+    prefix = "`10.0.0.5` has no explicit prefix length; state it during adoption so the netmask cannot change on takeover"
+    hard_vrid = "no virtual_router_id — it cannot be guessed: a wrong VRID puts the nodes in separate VRRP domains"
+    hard_auth = "auth_type AH is not supported (only PASS is rendered)"
+    all_four = [loss, prefix, hard_vrid, hard_auth]
+
+    # Nothing waived: everything survives.
+    assert remaining_blockers(all_four) == all_four
+    # A supplied prefix resolves ONLY the prefix blocker.
+    assert remaining_blockers(all_four, prefix_supplied=True) == [loss, hard_vrid, hard_auth]
+    # Accepting data loss resolves ONLY the loss blocker.
+    assert remaining_blockers(all_four, accept_data_loss=True) == [prefix, hard_vrid, hard_auth]
+    # Both together still cannot wave through an impossibility — this is the property that stops
+    # a UI flag from destroying a VIP whose VRID or auth_type we could not reproduce.
+    assert remaining_blockers(all_four, prefix_supplied=True, accept_data_loss=True) == \
+        [hard_vrid, hard_auth]
+    # And an adoptable candidate stays adoptable.
+    assert remaining_blockers([]) == []
+
+
+def test_waiver_markers_match_the_messages_the_parser_actually_emits():
+    # The gate matches on substrings of the blocker prose, so a reworded message would silently
+    # stop being waivable. Pin both directions against real parser output.
+    from services.keepalived_parser import remaining_blockers
+
+    no_prefix = HANDWRITTEN.replace("10.0.0.100/24 dev eth0", "10.0.0.100 dev eth0")
+    blockers = _only_candidate(no_prefix)["blockers"]
+    assert blockers, "expected a prefix blocker"
+    assert remaining_blockers(blockers, prefix_supplied=True) == []
+
+    with_hook = HANDWRITTEN.replace(
+        "    track_script {", '    notify_master "/usr/local/bin/promote.sh"\n    track_script {')
+    blockers = _only_candidate(with_hook)["blockers"]
+    assert blockers, "expected a data-loss blocker"
+    assert remaining_blockers(blockers, accept_data_loss=True) == []
+
+
+def test_auth_pass_masking_leaves_no_trace_of_the_secret():
+    # The discovered config is stored and served to the UI, so the ingest endpoint masks the VRRP
+    # password. Reuse the router's own regex so the test breaks if it is loosened.
+    from routers.agent import _AUTH_PASS_MASK_RE
+
+    secret = "s3cr3t with spaces"
+    text = HANDWRITTEN.replace("auth_pass s3cr3t", f"auth_pass {secret}")
+    masked = _AUTH_PASS_MASK_RE.sub(r"\1********", text)
+    assert secret not in masked and "s3cr3t" not in masked
+    assert "auth_pass ********" in masked
+    # Everything else survives, so the preview is still useful.
+    assert "virtual_router_id 51" in masked and "10.0.0.100/24 dev eth0" in masked

backend/tests/test_mfa_backwards_compat.py

@@ -0,0 +1,150 @@
+"""Backwards-compatibility regression tests for the MFA rollout (Issue #18).
+
+These tests don't hit a real database — they exercise the authoritative
+contract surfaces (login response shape, auth_middleware behaviour) using
+mocks where needed so the suite stays fast and deterministic.
+"""
+from __future__ import annotations
+
+import os
+import sys
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+class TestAuthMiddlewareUnchanged:
+    """auth_middleware MUST NOT look for MFA claims — Madde 2 of the plan."""
+
+    def test_decoder_imports_without_mfa_dependencies(self):
+        import auth_middleware
+        # The middleware's verification function exists and is callable.
+        assert callable(getattr(auth_middleware, "get_current_user_from_token", None))
+
+    def test_middleware_source_has_no_mfa_claim_check(self):
+        """The middleware source must not reference ``mfa`` claims directly."""
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "auth_middleware.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        # Allow incidental occurrences (e.g. comments); but never a claim lookup.
+        assert "payload.get('mfa'" not in source
+        assert 'payload.get("mfa"' not in source
+        assert "claims['mfa'" not in source
+        assert 'claims["mfa"' not in source
+
+
+class TestMfaModelsCoexistWithUserModels:
+    def test_models_user_module_unchanged_pydantic_shape(self):
+        from models import user as user_mod
+        # Ensure that User / UserUpdate / LoginRequest still load and still
+        # don't expose mfa-related fields (kept in models.mfa).
+        for cls in (user_mod.User, user_mod.UserUpdate, user_mod.LoginRequest):
+            fields = set(cls.model_fields.keys())
+            assert not {"mfa_enabled", "mfa_required", "mfa_token"} & fields, (
+                f"{cls.__name__} unexpectedly exposes MFA field; should stay byte-identical."
+            )
+
+    def test_models_mfa_module_exposes_expected_models(self):
+        from models import mfa as mfa_mod
+        for name in (
+            "MfaVerifyRequest",
+            "MfaEnrollStartResponse",
+            "MfaEnrollConfirmRequest",
+            "MfaEnrollConfirmResponse",
+            "MfaDisableRequest",
+            "MfaRegenerateBackupRequest",
+            "MfaRegenerateBackupResponse",
+            "MfaAdminResetRequest",
+            "MfaAdminResetAllRequest",
+            "MfaStatusResponse",
+        ):
+            assert hasattr(mfa_mod, name), f"Missing model: {name}"
+
+
+class TestRouterIncluded:
+    def test_main_includes_mfa_router(self):
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "main.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        assert "from routers.mfa import router as mfa_router" in source
+        assert "app.include_router(mfa_router)" in source
+
+
+class TestLoginResponseShapeForNonMfaUser:
+    """When MFA columns are missing or mfa_enabled=FALSE, /login returns the
+    pre-MFA response shape — no ``mfa_required`` / ``mfa_token`` keys leak through.
+    """
+
+    def test_login_without_mfa_returns_legacy_shape(self):
+        from fastapi.testclient import TestClient
+        from main import app
+
+        client = TestClient(app)
+
+        async def _fetch_mfa_state_none(conn, user_id):
+            return None
+
+        async def _no_log(*args, **kwargs):
+            return None
+
+        fake_user = {
+            "id": 1,
+            "username": "admin",
+            "email": "admin@example.com",
+            "password_hash": "$2b$12$placeholder",
+            "is_active": True,
+            "role": "admin",
+            "created_at": None,
+            "updated_at": None,
+            "last_login_at": None,
+        }
+
+        mock_conn = MagicMock()
+        mock_conn.fetchrow = AsyncMock(return_value=fake_user)
+        mock_conn.fetch = AsyncMock(return_value=[])
+        mock_conn.execute = AsyncMock(return_value=None)
+
+        async def _get_conn():
+            return mock_conn
+
+        async def _close(conn):
+            return None
+
+        with patch(
+            "routers.auth.get_database_connection", _get_conn
+        ), patch("routers.auth.close_database_connection", _close), patch(
+            "routers.auth._fetch_mfa_state", _fetch_mfa_state_none
+        ), patch("routers.auth.log_user_activity", _no_log), patch(
+            "bcrypt.checkpw", return_value=True
+        ):
+            resp = client.post(
+                "/api/auth/login",
+                json={"username": "admin", "password": "anything"},
+            )
+
+        assert resp.status_code == 200, resp.text
+        body = resp.json()
+        assert "access_token" in body
+        assert "token_type" in body
+        assert "expires_in" in body
+        assert "user" in body
+        assert "roles" in body
+        assert "permissions" in body
+        # CRITICAL — pre-MFA contract must not be polluted with MFA fields.
+        assert "mfa_required" not in body
+        assert "mfa_token" not in body
+        assert "methods" not in body

backend/tests/test_mfa_rate_limit_key.py

@@ -0,0 +1,200 @@
+"""Tests for middleware.mfa_rate_limit_key — user-aware + ingress-aware key."""
+import importlib
+from datetime import datetime, timedelta
+from typing import Dict, Optional
+
+import pytest
+from fastapi import Request
+from jose import jwt
+
+
+def _make_request(
+    headers: Optional[Dict[str, str]] = None,
+    peer: str = "127.0.0.1",
+) -> Request:
+    """Tiny ASGI scope shim — enough for the key_func surface."""
+    raw_headers = []
+    if headers:
+        raw_headers = [
+            (k.encode("latin-1"), v.encode("latin-1")) for k, v in headers.items()
+        ]
+    scope = {
+        "type": "http",
+        "headers": raw_headers,
+        "client": (peer, 12345),
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    return Request(scope)
+
+
+@pytest.fixture
+def reload_key(monkeypatch):
+    """Reload the key module so MFA_TRUSTED_PROXY_CIDRS is re-parsed."""
+
+    def _reload(**env):
+        for var in ("MFA_TRUSTED_PROXY_CIDRS",):
+            monkeypatch.delenv(var, raising=False)
+        for k, v in env.items():
+            monkeypatch.setenv(k, v)
+        from middleware import mfa_rate_limit_key as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+# ---------------------------------------------------------------------------
+# User-aware key extraction
+# ---------------------------------------------------------------------------
+
+
+def _mint_jwt(user_id, claim: str = "user_id") -> str:
+    """Mint a test JWT. Note: RFC 7519 says ``sub`` is a StringOrURI,
+    and python-jose validates that type when decoding, so callers that use
+    ``claim='sub'`` must pass a string user_id (matches production behavior
+    where auth_middleware also accepts string ``sub``)."""
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        claim: user_id,
+        "exp": datetime.utcnow() + timedelta(minutes=10),
+    }
+    return jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+
+def test_user_aware_via_user_id_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt(42, claim="user_id")
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:42"
+
+
+def test_user_aware_via_sub_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt("7", claim="sub")  # JWT spec: sub is a string
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:7"
+
+
+def test_no_auth_header_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_missing_bearer_prefix_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(headers={"authorization": "abc.def.ghi"}, peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_bearer_null_or_undefined_falls_back_to_ip(reload_key):
+    m = reload_key()
+    for bogus in ("null", "undefined", "", "   "):
+        req = _make_request(
+            headers={"authorization": f"Bearer {bogus}"}, peer="198.51.100.9"
+        )
+        assert m.mfa_rate_limit_key(req) == "ip:198.51.100.9"
+
+
+def test_tampered_jwt_falls_back_to_ip(reload_key):
+    """A token with a forged signature must NOT be honored — fallback to IP."""
+    m = reload_key()
+    bad = "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjogMTIzfQ.NOT_A_VALID_SIGNATURE"
+    req = _make_request(headers={"authorization": f"Bearer {bad}"}, peer="10.1.2.3")
+    assert m.mfa_rate_limit_key(req) == "ip:10.1.2.3"
+
+
+def test_expired_jwt_falls_back_to_ip(reload_key):
+    m = reload_key()
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        "user_id": 9,
+        "exp": datetime.utcnow() - timedelta(minutes=5),
+    }
+    expired = jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+    req = _make_request(headers={"authorization": f"Bearer {expired}"}, peer="10.0.0.7")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.7"
+
+
+# ---------------------------------------------------------------------------
+# Trusted-proxy X-Forwarded-For handling
+# ---------------------------------------------------------------------------
+
+
+def test_untrusted_peer_xff_is_ignored(reload_key):
+    """X-Forwarded-For from an untrusted client cannot move buckets."""
+    m = reload_key()  # no trusted CIDRs
+    req = _make_request(
+        headers={"x-forwarded-for": "1.2.3.4"},
+        peer="203.0.113.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_trusted_peer_xff_is_honored(reload_key):
+    """Peer in trusted CIDR → first XFF hop becomes the bucket."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(
+        headers={"x-forwarded-for": "203.0.113.42, 10.0.0.99"},
+        peer="10.0.0.99",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.42"
+
+
+def test_trusted_cidr_multiple_ranges(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8, 172.16.0.0/12")
+    req = _make_request(
+        headers={"x-forwarded-for": "198.51.100.4"},
+        peer="172.16.5.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:198.51.100.4"
+
+
+def test_trusted_peer_no_xff_falls_back_to_peer(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(peer="10.0.0.99")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.99"
+
+
+def test_invalid_cidr_in_env_is_logged_and_ignored(reload_key, caplog):
+    import logging
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limit_key"):
+        m = reload_key(MFA_TRUSTED_PROXY_CIDRS="not-a-cidr, 10.0.0.0/8")
+    assert any("ignoring invalid CIDR" in r.message for r in caplog.records)
+    # The valid one is still effective.
+    req = _make_request(
+        headers={"x-forwarded-for": "9.9.9.9"},
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:9.9.9.9"
+
+
+def test_user_bucket_wins_over_xff(reload_key):
+    """Auth always wins, even from a trusted proxy."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    token = _mint_jwt(99)  # integer user_id claim
+    req = _make_request(
+        headers={
+            "authorization": f"Bearer {token}",
+            "x-forwarded-for": "1.1.1.1",
+        },
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "user:99"
+
+
+def test_no_client_in_scope_does_not_crash(reload_key):
+    m = reload_key()
+    scope = {
+        "type": "http",
+        "headers": [],
+        "client": None,
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    req = Request(scope)
+    # Whatever it returns, it must be deterministic and not raise.
+    out = m.mfa_rate_limit_key(req)
+    assert out.startswith("ip:")

backend/tests/test_mfa_rate_limits.py

@@ -0,0 +1,93 @@
+"""Tests for middleware.mfa_rate_limits — env-driven MFA rate-limit config."""
+import importlib
+import logging
+
+import pytest
+
+
+@pytest.fixture
+def reload_module(monkeypatch):
+    """Helper: reload the module after env mutation so dataclass defaults
+    pick up the new values."""
+
+    def _reload(**env):
+        for key in list(globals().get('_OVERRIDDEN_ENVS', set())):
+            monkeypatch.delenv(key, raising=False)
+        for key, value in env.items():
+            monkeypatch.setenv(key, value)
+        from middleware import mfa_rate_limits as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+def test_defaults_when_no_env(reload_module, monkeypatch):
+    """No env var set → secure defaults applied (user-aware key assumption)."""
+    for key in (
+        "MFA_RATE_LIMIT_ENROLL_START",
+        "MFA_RATE_LIMIT_ENROLL_CONFIRM",
+        "MFA_RATE_LIMIT_DISABLE",
+        "MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES",
+        "MFA_RATE_LIMIT_ADMIN_RESET",
+        "MFA_RATE_LIMIT_ADMIN_RESET_ALL",
+    ):
+        monkeypatch.delenv(key, raising=False)
+    m = reload_module()
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert m.MFA_LIMITS.enroll_confirm == "10/minute"
+    assert m.MFA_LIMITS.disable == "10/minute"
+    assert m.MFA_LIMITS.regenerate_backup_codes == "5/hour"
+    assert m.MFA_LIMITS.admin_reset == "60/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "1/day"
+
+
+def test_env_override_per_endpoint(reload_module):
+    m = reload_module(
+        MFA_RATE_LIMIT_ENROLL_START="100/hour",
+        MFA_RATE_LIMIT_ADMIN_RESET_ALL="3/day",
+    )
+    assert m.MFA_LIMITS.enroll_start == "100/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "3/day"
+    # Untouched values still default.
+    assert m.MFA_LIMITS.disable == "10/minute"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "totally-bogus",
+        "5/lightyear",
+        "abc/minute",
+        "5",
+        "/minute",
+        "5//minute",
+        "",
+    ],
+)
+def test_invalid_format_falls_back_to_default(reload_module, caplog, bad):
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limits"):
+        m = reload_module(MFA_RATE_LIMIT_ENROLL_START=bad)
+    # Falls back to the secure default for enroll_start.
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert any("not a valid slowapi limit string" in r.message for r in caplog.records)
+
+
+def test_whitespace_around_value_is_tolerated(reload_module):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE="  30/minute  ")
+    assert m.MFA_LIMITS.disable == "30/minute"
+
+
+@pytest.mark.parametrize(
+    "valid",
+    ["1/second", "100/minute", "1000/hour", "10/day"],
+)
+def test_all_valid_periods_accepted(reload_module, valid):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE=valid)
+    assert m.MFA_LIMITS.disable == valid
+
+
+def test_dataclass_is_frozen(reload_module):
+    """Frozen dataclass guards against accidental mutation after import."""
+    m = reload_module()
+    with pytest.raises((AttributeError, Exception)):
+        m.MFA_LIMITS.disable = "999/second"  # type: ignore[misc]

backend/tests/test_mfa_service.py

@@ -0,0 +1,223 @@
+"""Unit tests for the MFA service layer (Issue #18, v1.6.0).
+
+These tests cover the pure-Python side of MFA — no DB, no FastAPI.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+import re
+import sys
+import time
+
+import pytest
+
+# Repo path setup (mirrors other tests in this folder).
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import pyotp  # noqa: E402
+
+from services import mfa_service  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# TOTP
+# ---------------------------------------------------------------------------
+
+
+class TestTotpSecret:
+    def test_secret_is_base32(self):
+        secret = mfa_service.generate_totp_secret()
+        # pyotp.random_base32() returns 32-character base32 strings.
+        assert len(secret) == 32
+        assert re.fullmatch(r"[A-Z2-7]+", secret), "secret must be valid base32"
+
+    def test_secrets_are_unique(self):
+        secrets = {mfa_service.generate_totp_secret() for _ in range(50)}
+        assert len(secrets) == 50
+
+
+class TestVerifyTotp:
+    def setup_method(self):
+        self.secret = mfa_service.generate_totp_secret()
+        self.totp = pyotp.TOTP(self.secret, digits=6, interval=30, digest="sha1")
+
+    def test_happy_path(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        assert step == int(time.time()) // 30
+
+    def test_invalid_code_format_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "abc", None)
+        assert ok is False and step is None
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "12345", None)
+        assert ok is False and step is None
+
+    def test_tolerance_minus_30s(self):
+        now = int(time.time())
+        previous_step = (now // 30) - 1
+        prev_code = self.totp.at(previous_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, prev_code, None)
+        assert ok is True
+        assert step == previous_step
+
+    def test_tolerance_plus_30s(self):
+        now = int(time.time())
+        next_step = (now // 30) + 1
+        next_code = self.totp.at(next_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, next_code, None)
+        assert ok is True
+        assert step == next_step
+
+    def test_replay_rejected(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        # Submit again with the previously-consumed step: must be rejected.
+        ok2, step2 = mfa_service.verify_totp_with_replay_guard(self.secret, code, step)
+        assert ok2 is False
+        assert step2 is None
+
+    def test_wrong_code_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "000000", None)
+        assert ok is False
+        assert step is None
+
+
+# ---------------------------------------------------------------------------
+# Fernet + key resolution
+# ---------------------------------------------------------------------------
+
+
+class TestFernet:
+    def setup_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def teardown_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def test_encrypt_decrypt_roundtrip_with_env_key(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        key = Fernet.generate_key().decode()
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", key)
+        mfa_service.reset_fernet_for_tests()
+
+        secret = "JBSWY3DPEHPK3PXP" * 2
+        token = mfa_service.encrypt_secret(secret)
+        assert token and token != secret
+        recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+
+    def test_decrypt_invalid_token_returns_none(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", Fernet.generate_key().decode())
+        mfa_service.reset_fernet_for_tests()
+
+        assert mfa_service.decrypt_secret("not-a-valid-fernet-token") is None
+
+    def test_hkdf_fallback_when_env_unset(self, monkeypatch, caplog):
+        monkeypatch.delenv("MFA_ENCRYPTION_KEY", raising=False)
+        mfa_service.reset_fernet_for_tests()
+
+        with caplog.at_level("WARNING"):
+            secret = "JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP"
+            token = mfa_service.encrypt_secret(secret)
+            recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+        assert any("MFA_ENCRYPTION_KEY" in r.message for r in caplog.records), (
+            "expected a WARN log when falling back to SECRET_KEY derivation"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+class TestBackupCodes:
+    def test_generate_count_and_format(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(codes) == 10
+        # 31-char alphabet: A-H J K M N P-Z 2-9 (excludes I, L, O, 0, 1).
+        for code in codes:
+            assert re.fullmatch(r"[A-HJKM-NP-Z2-9]{4}-[A-HJKM-NP-Z2-9]{4}", code), code
+
+    def test_alphabet_excludes_confusing_characters(self):
+        # Generate enough codes to virtually guarantee any forbidden char would surface.
+        for _ in range(20):
+            codes = mfa_service.generate_backup_codes()
+            for code in codes:
+                for ch in code.replace("-", ""):
+                    assert ch not in "0O1IL", f"forbidden char {ch!r} in {code!r}"
+
+    def test_codes_are_unique(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(set(codes)) == len(codes)
+
+    def test_normalize_strips_case_dash_space(self):
+        assert mfa_service.normalize_backup_code("abcd-efgh") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code(" ab cd-ef gh ") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code("") == ""
+        assert mfa_service.normalize_backup_code(None) == ""  # type: ignore[arg-type]
+
+    def test_hash_and_check_async(self):
+        async def _run():
+            plain = mfa_service.generate_backup_codes()[:1]
+            hashes = await mfa_service.hash_backup_codes(plain)
+            assert len(hashes) == 1
+            assert await mfa_service.check_backup_code(plain[0], hashes[0]) is True
+            assert await mfa_service.check_backup_code("WRONG-CODE", hashes[0]) is False
+            # Case + dash normalization
+            assert await mfa_service.check_backup_code(plain[0].lower(), hashes[0]) is True
+            assert (
+                await mfa_service.check_backup_code(plain[0].replace("-", ""), hashes[0])
+                is True
+            )
+
+        asyncio.run(_run())
+
+
+# ---------------------------------------------------------------------------
+# otpauth URI
+# ---------------------------------------------------------------------------
+
+
+class TestOtpAuthUri:
+    def test_uri_shape(self):
+        uri = mfa_service.build_otpauth_uri("alice@example.com", "JBSWY3DPEHPK3PXP")
+        assert uri.startswith("otpauth://totp/")
+        assert "secret=JBSWY3DPEHPK3PXP" in uri
+        assert "issuer=" in uri
+        assert "algorithm=SHA1" in uri
+        assert "digits=6" in uri
+        assert "period=30" in uri
+
+    def test_account_label_env_override(self, monkeypatch):
+        monkeypatch.setenv("MFA_ACCOUNT_LABEL_DOMAIN", "ops.example.com")
+        label = mfa_service.build_account_label("alice", hostname_hint="ignored.com")
+        assert label == "alice@ops.example.com"
+
+    def test_account_label_hostname_hint(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint="api.local")
+        assert label == "alice@api.local"
+
+    def test_account_label_fallback(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint=None)
+        assert label == "alice@haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Challenge token
+# ---------------------------------------------------------------------------
+
+
+class TestChallengeToken:
+    def test_length_and_uniqueness(self):
+        tokens = {mfa_service.generate_challenge_token() for _ in range(50)}
+        assert len(tokens) == 50
+        for t in tokens:
+            assert len(t) == 64
+            assert re.fullmatch(r"[0-9a-f]{64}", t)

backend/tests/test_router_path_shadowing.py

@@ -0,0 +1,141 @@
+"""
+v1.10.6 — literal API paths must never be declared after a parameterised one.
+
+Found in production on the v1.10.4 VIP-adoption feature: `@router.get("/discoveries")`
+sat at the bottom of routers/vip.py, below `@router.get("/{vip_id}")`. FastAPI matches
+routes in DECLARATION order, so every `GET /api/vip/discoveries` was answered by the
+`/{vip_id}` handler, which declares `vip_id: int` and therefore rejected the request with
+422 before `list_vip_discoveries` ever ran.
+
+Nothing about that failure was visible. The agents reported their discoveries correctly,
+the rows landed in `vip_discoveries`, and the HA/VIP page treats any non-OK response as
+"nothing to show" — so the adoption feature simply did not exist as far as the UI was
+concerned, with no error anywhere.
+
+These tests are a STATIC source scan on purpose: no imports, no app construction, no DB.
+They therefore also cover routers that cannot be imported in a bare test environment, and
+they keep covering routes added in the future.
+"""
+import pathlib
+import re
+
+import pytest
+
+ROUTERS_DIR = pathlib.Path(__file__).resolve().parents[1] / "routers"
+
+# `@router.get("/x")`, `@some_router.post("/x", ...)` — the path is the first string arg.
+_DECORATOR = re.compile(r'^@(?:\w+)\.(get|post|put|delete|patch)\(\s*[\'"]([^\'"]*)[\'"]')
+
+
+def _routes(source: str):
+    """[(line_no, verb, path)] in declaration order."""
+    out = []
+    for line_no, line in enumerate(source.splitlines(), 1):
+        match = _DECORATOR.match(line)
+        if match:
+            out.append((line_no, match.group(1), match.group(2)))
+    return out
+
+
+def _shadows(earlier: str, later: str) -> bool:
+    """True if `earlier` (declared first) swallows the literal path `later`.
+
+    Only literal paths can be silently swallowed, and only by a path that has the same
+    number of segments where every non-placeholder segment matches. The collection route
+    ("" or "/") is its own path and never collides.
+    """
+    if not later.strip("/") or not earlier.strip("/"):
+        return False
+    if "{" in later:
+        return False
+    if "{" not in earlier:
+        return False
+    earlier_segments = earlier.strip("/").split("/")
+    later_segments = later.strip("/").split("/")
+    if len(earlier_segments) != len(later_segments):
+        return False
+    return all(
+        e.startswith("{") or e == l
+        for e, l in zip(earlier_segments, later_segments)
+    )
+
+
+def _shadowed_routes(path: pathlib.Path):
+    routes = _routes(path.read_text())
+    found = []
+    for index, (line_no, verb, route_path) in enumerate(routes):
+        for prior_line, prior_verb, prior_path in routes[:index]:
+            if prior_verb == verb and _shadows(prior_path, route_path):
+                found.append(
+                    f"{path.name}:{line_no} {verb.upper()} {route_path} is swallowed by "
+                    f"{prior_path} declared at line {prior_line}"
+                )
+    return found
+
+
+# ----------------------------------------------------------------------------
+# 1. The specific regression: /api/vip/discoveries must outrank /{vip_id}
+# ----------------------------------------------------------------------------
+
+def test_vip_discoveries_declared_before_vip_id():
+    routes = _routes((ROUTERS_DIR / "vip.py").read_text())
+    get_paths = [path for _line, verb, path in routes if verb == "get"]
+
+    assert "/discoveries" in get_paths, "the discoveries endpoint disappeared"
+    assert "/{vip_id}" in get_paths, "the get-one endpoint disappeared"
+    assert get_paths.index("/discoveries") < get_paths.index("/{vip_id}"), (
+        "GET /discoveries is declared after GET /{vip_id}; FastAPI will route "
+        "/api/vip/discoveries into get_vip and answer 422, silently emptying the "
+        "adoption panel"
+    )
+
+
+# ----------------------------------------------------------------------------
+# 2. The general guard: no literal path anywhere is shadowed
+# ----------------------------------------------------------------------------
+
+def test_no_literal_route_is_shadowed_in_any_router():
+    problems = []
+    for router_file in sorted(ROUTERS_DIR.glob("*.py")):
+        problems.extend(_shadowed_routes(router_file))
+
+    assert not problems, (
+        "literal route(s) declared after a parameterised route that swallows them:\n  "
+        + "\n  ".join(problems)
+    )
+
+
+# ----------------------------------------------------------------------------
+# 3. The detector itself must actually detect (guards against a vacuous pass)
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize(
+    "earlier,later,expected",
+    [
+        ("/{vip_id}", "/discoveries", True),        # the v1.10.4 bug
+        ("/{vip_id}", "/{other}", False),           # two placeholders never shadow
+        ("/{vip_id}", "", False),                   # collection route is its own path
+        ("/{vip_id}/apply", "/adopt", False),       # different segment counts
+        ("/{vip_id}/apply", "/adopt/now", False),   # literal mismatch in segment 2
+        ("/{vip_id}/{action}", "/adopt/now", True), # both segments placeheld
+        ("/vips", "/discoveries", False),           # literal never shadows a literal
+    ],
+)
+def test_shadow_detector_semantics(earlier, later, expected):
+    assert _shadows(earlier, later) is expected
+
+
+def test_detector_flags_the_original_declaration_order():
+    """A synthetic file in the pre-fix order must be reported, so a future refactor that
+    breaks the detector cannot make the guard above pass vacuously."""
+    source = (
+        '@router.get("")\n'
+        "async def list_vips(): ...\n"
+        '@router.get("/{vip_id}")\n'
+        "async def get_vip(vip_id: int): ...\n"
+        '@router.get("/discoveries")\n'
+        "async def list_vip_discoveries(): ...\n"
+    )
+    routes = _routes(source)
+    assert [verb for _l, verb, _p in routes] == ["get", "get", "get"]
+    assert _shadows(routes[1][2], routes[2][2]) is True

backend/tests/test_security_advisories_auth.py

@@ -0,0 +1,223 @@
+"""Regression tests for the 2026-07 security advisories.
+
+Covers:
+- GHSA-7rhv-c5pc-69r8 (CRITICAL RCE): agent script-template management must
+  require the agents.version permission, not merely authentication.
+- GHSA-3p5c-m5m4-mjpx (missing auth): agent data-plane endpoints must require a
+  valid X-API-Key, and operator/UI endpoints must require a JWT. An anonymous
+  caller must never get a 200 with sensitive data.
+
+These are behavioral assertions via FastAPI's TestClient. The auth checks were
+deliberately moved ahead of any DB access, so an unauthenticated request is
+rejected without needing a database — the same approach as the existing
+test_ssl_list_endpoint_auth.py. Accepted rejection statuses are 401/403/422
+(never 200-with-data).
+"""
+import re
+import os
+import pytest
+
+REJECT = (401, 403, 422)
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: agent data-plane endpoints must reject a missing X-API-Key
+# --------------------------------------------------------------------------
+
+def test_agent_config_requires_api_key(client):
+    """GET /api/agents/{name}/config leaked the full haproxy.cfg without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/config")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: agent config served without X-API-Key ({res.status_code})"
+    )
+
+
+def test_agent_ssl_certificates_requires_api_key(client):
+    """GET /api/agents/{name}/ssl-certificates leaked SSL private keys without a key."""
+    res = client.get("/api/agents/prod-haproxy-1/ssl-certificates")
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: SSL certs (private keys!) served without X-API-Key ({res.status_code})"
+    )
+    if res.status_code == 200:
+        assert "private_key_content" not in res.text
+
+
+def test_agent_upgrade_status_requires_api_key(client):
+    res = client.get("/api/agents/prod-haproxy-1/upgrade-status")
+    assert res.status_code in REJECT
+
+
+def test_agent_pending_requests_requires_api_key(client):
+    res = client.get("/api/configuration/agents/prod-haproxy-1/pending-requests")
+    assert res.status_code in REJECT
+
+
+def test_agent_heartbeat_by_name_requires_api_key(client):
+    res = client.post("/api/agents/heartbeat", json={"name": "rogue-poc"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless heartbeat/auto-register accepted ({res.status_code})"
+    )
+
+
+def test_agent_heartbeat_by_id_requires_api_key(client):
+    res = client.post("/api/agents/1/heartbeat", json={"name": "spoofed"})
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c regression: keyless by-id heartbeat state-spoof accepted ({res.status_code})"
+    )
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c: operator/UI endpoints must reject a missing JWT
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_requires_jwt(client):
+    """GET /api/agents (RCE read-back channel) was served without a JWT."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", ["/api/health/deep", "/api/health/agents", "/api/health/clusters"])
+def test_detailed_health_requires_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, f"{path} served without a JWT ({res.status_code})"
+
+
+def test_simple_health_stays_public(client):
+    """The liveness probe endpoint (/api/health) must remain UNAUTHENTICATED.
+
+    It reports 200 when healthy and 503 when the DB is unreachable (as in this
+    no-DB test env); what matters for the k8s probe is that it is never gated
+    behind auth (401/403). We only added auth to /api/health/{deep,agents,clusters}.
+    """
+    res = client.get("/api/health")
+    assert res.status_code not in (401, 403), (
+        f"Regression: /api/health liveness probe now requires auth ({res.status_code}) — "
+        f"this breaks k8s liveness/readiness"
+    )
+
+
+def test_dashboard_stats_requires_jwt(client):
+    res = client.get("/api/dashboard-stats/stats?cluster_id=1")
+    assert res.status_code in REJECT
+
+
+def test_ssl_config_versions_requires_jwt(client):
+    res = client.get("/api/ssl/certificates/1/config-versions")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-7rhv (CRITICAL RCE): script-template management
+# --------------------------------------------------------------------------
+
+def test_script_template_write_requires_auth(client):
+    """Anonymous POST must be rejected outright."""
+    res = client.post("/api/agents/script-templates/linux",
+                      json={"script_content": "#!/bin/bash\nid", "version": "9.9.9"})
+    assert res.status_code in REJECT
+
+
+def test_script_template_read_requires_auth(client):
+    res = client.get("/api/agents/script-templates/linux")
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): sibling endpoints exposing the SAME class of data
+# (found during post-merge review — must also require a JWT)
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/haproxy-cluster-pools/1/agents",   # full agent inventory — same class as GET /api/agents
+    "/api/pools",
+    "/api/haproxy-cluster-pools",
+    "/api/dashboard/stats",
+    "/api/dashboard/overview",               # optional-auth pattern — leaked stats/names/alerts anonymously
+    "/api/haproxy/stats",
+    "/api/waf/rules",
+    "/api/health/errors",
+])
+def test_sibling_inventory_endpoints_require_jwt(client, path):
+    res = client.get(path)
+    assert res.status_code in REJECT, (
+        f"GHSA-3p5c (round 2) regression: {path} served without a JWT ({res.status_code}) — "
+        f"anonymous access to inventory/topology/WAF/error data"
+    )
+
+
+def test_pool_agents_no_anonymous_inventory_leak(client):
+    """The richest bypass: /api/haproxy-cluster-pools/{id}/agents must not leak inventory."""
+    res = client.get("/api/haproxy-cluster-pools/1/agents")
+    assert res.status_code in REJECT
+    if res.status_code == 200:
+        assert "ip_address" not in res.text and "hostname" not in res.text
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): agent webhooks must return 401 (not a 200 error body)
+# for anonymous callers — the auth raise must propagate, not be swallowed.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/agents/some-agent/config-applied",
+    "/api/agents/some-agent/config-validation-failed",
+    "/api/agents/some-agent/config-sync",
+])
+def test_agent_webhooks_reject_anonymous_with_401(client, path):
+    res = client.post(path, json={})
+    assert res.status_code in REJECT, (
+        f"{path} returned {res.status_code} for an anonymous caller — the auth "
+        f"rejection must be a 401/403, not a swallowed 200 error body"
+    )
+    # Specifically must NOT be a 200 "status: error" body.
+    assert res.status_code != 200
+
+
+# --------------------------------------------------------------------------
+# GHSA-3p5c (round 2): GET /api/agents must accept EITHER a JWT OR an agent
+# X-API-Key. Anonymous (neither) is still rejected — agents send a key, so a
+# JWT-only gate would break them (verified end-to-end in the localtest smoke).
+# --------------------------------------------------------------------------
+
+def test_agents_inventory_still_rejects_fully_anonymous(client):
+    """No JWT and no X-API-Key -> 401 (the agent-key accept path needs a valid key)."""
+    res = client.get("/api/agents")
+    assert res.status_code in REJECT
+
+
+def test_generate_uninstall_script_requires_auth(client):
+    """Agent-management endpoint must not be anonymously reachable (JWT or agent key)."""
+    res = client.get("/api/agents/generate-uninstall-script/linux")
+    assert res.status_code in REJECT
+
+
+@pytest.mark.parametrize("path", [
+    "/api/config/validate",
+    "/api/config/optimize",
+    "/api/config/templates/default/generate",
+])
+def test_config_compute_endpoints_require_auth(client, path):
+    """Config compute endpoints (run a HAProxy validator on caller input) were
+    optional-auth; now require a JWT. The dependency rejects before body parsing."""
+    res = client.post(path, json={})
+    assert res.status_code in REJECT
+
+
+def test_script_template_write_enforces_agents_version_permission():
+    """Static guarantee: the write handler checks agents.version (not just authN).
+
+    A behavioral 403-for-viewer test would need a seeded DB + a minted viewer JWT;
+    instead we assert the permission gate is present in source, mirroring the
+    existing audit-style source tests. This is the core RCE fix (GHSA-7rhv).
+    """
+    src_path = os.path.join(os.path.dirname(__file__), "..", "routers", "agent.py")
+    with open(src_path, "r") as f:
+        src = f.read()
+    # Isolate the save_agent_script_template handler body.
+    m = re.search(r"async def save_agent_script_template\(.*?\n(.*?)\n@router\.", src, re.DOTALL)
+    assert m, "save_agent_script_template handler not found"
+    body = m.group(1)
+    assert 'check_user_permission' in body and '"agents", "version"' in body, (
+        "GHSA-7rhv regression: script-template WRITE no longer enforces the "
+        "agents.version permission — any JWT holder could poison the root install script"
+    )

backend/tests/test_site_wizard_round11.py

@@ -276,14 +276,16 @@ def test_categorize_routes_directives_correctly():
 
 def test_emit_buckets_flushed_in_canonical_order():
     """The flush block at end of frontend processing must list buckets
-    in: prelude → stick → tcp_req → acl → http_req → http_resp →
+    in: prelude → filter → stick → tcp_req → acl → http_req → http_resp →
     redirect → use_be → default_be. Pre-fix `http-request` rules
     interleaved with `use_backend` rules in source order, producing
-    HAProxy parser warnings."""
+    HAProxy parser warnings. (Issue #38 added the `filter` bucket, flushed
+    right after `prelude` so SPOE `filter` lines precede `send-spoe-group`.)"""
     src = _gen_src()
     flush_match = re.search(
         r'for\s+_bucket_key\s+in\s+\(\s*'
         r'"prelude"\s*,\s*'
+        r'"filter"\s*,\s*'
         r'"stick"\s*,\s*'
         r'"tcp_req"\s*,\s*'
         r'"acl"\s*,\s*'

backend/tests/test_spoe_filter_import.py

@@ -0,0 +1,203 @@
+"""
+Issue #38 regression tests: HAProxy SPOE `filter` + frontend `log-format` support.
+
+Bug: the bulk-config parser recognised only a fixed set of frontend directives,
+so `filter spoe engine coraza config ...` and `log-format ...` were silently
+dropped on import / manual edit. This regenerated a config missing the SPOE
+engine definition, so HAProxy failed with
+"unable to find SPOE engine 'coraza' used by the send-spoe-group 'coraza-req'".
+
+These tests verify the end-to-end fix without requiring a database:
+1. parser captures `filter` + `log-format` into the new ParsedFrontend fields;
+2. `http-request send-spoe-group` is still preserved (regression guard);
+3. the generator's directive categoriser + bucket flush order emit `filter`
+   BEFORE the `http-request send-spoe-group` rules and keep `log-format`;
+4. reject/rollback restores the new columns;
+5. a non-SPOE frontend is completely unaffected (zero-impact).
+"""
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.haproxy_config_parser import parse_haproxy_config, ParsedFrontend
+from services.haproxy_config import _categorize_haproxy_directive
+from models.frontend import FrontendConfig
+
+
+# The exact frontend/backend config reported in Issue #38 (Coraza-SPOA).
+ISSUE_38_CONFIG = r"""
+frontend web-frontend
+    bind *:8073
+    mode http
+        log-format "%ci:%cp\ [%t]\ %ft\ %b/%s\ %ST\ %B\ %{+Q}r\ %[var(txn.coraza.id)]\ waf-hit:\ %[var(txn.coraza.fail)]"
+        filter spoe engine coraza config /etc/haproxy/coraza.cfg
+      http-request set-var(txn.coraza.app) str(haproxy_waf)
+     http-request send-spoe-group coraza coraza-req
+    http-request deny if { var(txn.coraza.fail) -m int eq 1 }
+    default_backend web-backend
+
+backend web-backend
+    balance roundrobin
+    mode http
+    server server1 192.168.1.10:443 weight 100 ssl verify none
+
+backend coraza-spoa
+    mode tcp
+    option spop-check
+    server coraza_spoa 192.168.12.21:9000
+"""
+
+
+def _get_frontend(parse_result, name):
+    for fe in parse_result.frontends:
+        if fe.name == name:
+            return fe
+    return None
+
+
+class TestParserCapturesSpoe:
+    def test_filter_and_log_format_captured(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe is not None, "web-frontend should be parsed and kept"
+        assert fe.filters is not None
+        assert "filter spoe engine coraza config /etc/haproxy/coraza.cfg" in fe.filters
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format")
+        # the escaped/quoted format string must be preserved verbatim
+        assert "%[var(txn.coraza.fail)]" in fe.log_format
+
+    def test_send_spoe_group_still_preserved(self):
+        # Regression guard: http-request rules (incl. send-spoe-group) must
+        # still be collected into request_headers as before.
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        fe = _get_frontend(result, "web-frontend")
+        assert fe.request_headers is not None
+        assert "send-spoe-group coraza coraza-req" in fe.request_headers
+
+    def test_multiple_filters_preserved_in_order(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    filter compression
+    filter spoe engine coraza config /etc/haproxy/coraza.cfg
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        lines = fe.filters.split("\n")
+        assert lines == [
+            "filter compression",
+            "filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+        ]
+
+    def test_log_format_sd_variant_captured(self):
+        cfg = """
+frontend f1
+    bind *:80
+    mode http
+    log-format-sd "[exampleSDID@1234 field=value]"
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "f1")
+        assert fe.log_format is not None
+        assert fe.log_format.startswith("log-format-sd")
+
+
+class TestGeneratorOrderingContract:
+    """The generator routes directives into ordered buckets. Verify SPOE
+    correctness at the (pure) categoriser + documented flush-order level."""
+
+    def test_filter_routes_to_filter_bucket(self):
+        assert _categorize_haproxy_directive("    filter spoe engine coraza config /x.cfg") == "filter"
+
+    def test_send_spoe_group_routes_to_http_req(self):
+        assert _categorize_haproxy_directive("    http-request send-spoe-group coraza coraza-req") == "http_req"
+
+    def test_log_format_routes_to_prelude(self):
+        assert _categorize_haproxy_directive('    log-format "%ci:%cp"') == "prelude"
+        assert _categorize_haproxy_directive('    log-format-sd "[x]"') == "prelude"
+
+    def test_flush_order_places_filter_before_http_req(self):
+        # The bucket flush order is the single source of truth for emission
+        # ordering. Assert `filter` is flushed before `http_req` (and after
+        # `prelude`), guaranteeing `filter ...` renders before
+        # `http-request send-spoe-group ...`.
+        src = _read_source("services/haproxy_config.py")
+        m = re.search(r"for _bucket_key in \((.*?)\):", src, re.DOTALL)
+        assert m, "bucket flush loop not found"
+        order = re.findall(r'"(\w+)"', m.group(1))
+        assert "filter" in order, "new 'filter' bucket missing from flush order"
+        assert order.index("prelude") < order.index("filter") < order.index("http_req")
+
+
+class TestModelAndRollback:
+    def test_model_has_passthrough_fields(self):
+        fc = FrontendConfig(
+            name="f", bind_port=80,
+            filters="filter spoe engine coraza config /etc/haproxy/coraza.cfg",
+            log_format='log-format "%ci"',
+        )
+        assert fc.filters.startswith("filter spoe")
+        assert fc.log_format.startswith("log-format")
+
+    def test_dataclass_defaults_none(self):
+        fe = ParsedFrontend(name="f")
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_rollback_restores_new_columns(self):
+        # Reject/rollback of a frontend UPDATE must restore the new columns,
+        # otherwise the rejected (new) filters/log_format would persist.
+        src = _read_source("utils/entity_snapshot.py")
+        assert "log_format = $" in src
+        assert "filters = $" in src
+        assert "old_values.get('log_format')" in src
+        assert "old_values.get('filters')" in src
+
+
+class TestZeroImpact:
+    def test_non_spoe_frontend_unaffected(self):
+        cfg = """
+frontend plain
+    bind *:80
+    mode http
+    option httplog
+    default_backend b1
+
+backend b1
+    mode http
+    server s1 10.0.0.1:80
+"""
+        fe = _get_frontend(parse_haproxy_config(cfg), "plain")
+        # No filter / log-format present → new fields stay None (no behaviour change)
+        assert fe.filters is None
+        assert fe.log_format is None
+
+    def test_spop_check_backend_roundtrips_without_warning(self):
+        result = parse_haproxy_config(ISSUE_38_CONFIG)
+        be = next((b for b in result.backends if b.name == "coraza-spoa"), None)
+        assert be is not None, "coraza-spoa backend should import"
+        assert be.mode == "tcp"
+        assert be.options and "option spop-check" in be.options
+        # spop-check is now a known option → no spurious 'unknown option' warning
+        assert not any(
+            "coraza-spoa" in w and "spop-check" in w and "Unknown" in w
+            for w in result.warnings
+        )
+
+
+def _read_source(relpath):
+    base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    with open(os.path.join(base, relpath), "r", encoding="utf-8") as fh:
+        return fh.read()

backend/tests/test_ssrf_guard.py

@@ -0,0 +1,68 @@
+"""Unit tests for the SSRF guard (GHSA-3vh4-gvxx-wm2p).
+
+The guard protects server-side fetches of ACME `directory_url` values. This
+project uses only public ACME CAs, so every non-public IP must be rejected.
+Tests avoid real network/DNS by using IP literals and scheme checks.
+"""
+import asyncio
+import pytest
+
+from utils.ssrf_guard import is_public_ip, assert_public_url, SSRFValidationError
+
+
+# ---- is_public_ip -----------------------------------------------------------
+
+@pytest.mark.parametrize("ip", [
+    "8.8.8.8", "1.1.1.1", "93.184.216.34",  # public
+])
+def test_public_ips_allowed(ip):
+    assert is_public_ip(ip) is True
+
+
+@pytest.mark.parametrize("ip", [
+    "127.0.0.1",            # loopback
+    "10.0.0.5",             # RFC1918
+    "172.19.0.1",           # RFC1918 (the SSRF PoC docker gateway)
+    "192.168.1.1",          # RFC1918
+    "169.254.169.254",      # link-local / cloud metadata
+    "0.0.0.0",              # unspecified
+    "::1",                  # IPv6 loopback
+    "fe80::1",              # IPv6 link-local
+    "::ffff:127.0.0.1",     # IPv4-mapped IPv6 loopback (R18c bypass)
+    "::ffff:169.254.169.254",  # IPv4-mapped metadata
+    "not-an-ip",            # garbage
+])
+def test_non_public_ips_rejected(ip):
+    assert is_public_ip(ip) is False
+
+
+# ---- assert_public_url ------------------------------------------------------
+
+def _raises(url):
+    with pytest.raises(SSRFValidationError):
+        asyncio.run(assert_public_url(url))
+
+
+def test_rejects_non_https_scheme():
+    # The SSRF PoC used http:// against an internal listener.
+    _raises("http://172.19.0.1:2121/internal-secret")
+    _raises("http://8.8.8.8/")            # even a public IP over http is refused
+    _raises("file:///etc/passwd")
+    _raises("gopher://8.8.8.8/")
+
+
+def test_rejects_private_ip_literals():
+    _raises("https://127.0.0.1/")
+    _raises("https://10.0.0.5/")
+    _raises("https://169.254.169.254/latest/meta-data/")
+    _raises("https://[::1]/")
+
+
+def test_rejects_empty_or_hostless():
+    _raises("")
+    _raises("https://")
+
+
+def test_allows_public_ip_literal_https():
+    # A public IP literal over https must pass (no DNS needed).
+    asyncio.run(assert_public_url("https://8.8.8.8/directory"))

backend/tests/test_version_consistency.py

@@ -0,0 +1,71 @@
+"""v1.8.7: app version is single-source and cannot silently drift.
+
+The UI shows the version via GET /api/version, which returns main.py's `_version_info`. That MUST be
+sourced from the one canonical file backend/version.json (co-located with main.py so `COPY . .`
+bakes it into every image, regardless of pipeline). main.py's in-code fallback must NOT be a real
+version, otherwise it drifts when version.json is bumped but the constant is forgotten — exactly
+what left the UI reporting 1.8.4 after 1.8.5/1.8.6 shipped. These checks fail loudly on regression.
+"""
+import ast
+import json
+import os
+import re
+
+import pytest
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))  # backend/
+_VERSION_JSON = os.path.join(_BACKEND, "version.json")
+_MAIN = os.path.join(_BACKEND, "main.py")
+_SEMVER = re.compile(r"^\d+\.\d+\.\d+$")
+
+
+def test_canonical_version_file_exists_and_valid():
+    assert os.path.exists(_VERSION_JSON), "backend/version.json (single source of truth) is missing"
+    with open(_VERSION_JSON) as f:
+        data = json.load(f)
+    assert _SEMVER.match(data.get("version", "")), \
+        f"backend/version.json version is not semver: {data.get('version')!r}"
+    assert data.get("releaseName"), "backend/version.json must have a releaseName"
+
+
+def _main_fallback_version_info():
+    """The literal dict assigned to _version_info in main.py (the in-code fallback)."""
+    with open(_MAIN) as f:
+        tree = ast.parse(f.read())
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign) and isinstance(node.value, ast.Dict):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == "_version_info":
+                    return ast.literal_eval(node.value)
+    return None
+
+
+def test_main_has_no_hardcoded_real_version():
+    fb = _main_fallback_version_info()
+    assert fb is not None, "could not find the _version_info fallback literal in main.py"
+    # Must be a neutral marker, never a real version that can drift out of sync.
+    assert not _SEMVER.match(str(fb.get("version", ""))), (
+        f"main.py hardcodes a real version {fb.get('version')!r}; it must be a neutral marker "
+        f"(e.g. 'unknown') so the version stays single-source in backend/version.json"
+    )
+
+
+def test_main_loads_the_canonical_file_first():
+    # The first candidate path main.py reads must resolve to the co-located backend/version.json,
+    # so the correct version is available in every image (not dependent on CI staging).
+    with open(_MAIN) as f:
+        src = f.read()
+    assert 'os.path.dirname(__file__), "version.json"' in src, \
+        "main.py must read version.json co-located with the module (backend/version.json)"
+
+
+def test_frontend_package_json_matches_when_present():
+    # Only meaningful in a full-repo checkout; the backend image build context (./backend) has no frontend/.
+    pkg = os.path.join(os.path.dirname(_BACKEND), "frontend", "package.json")
+    if not os.path.exists(pkg):
+        pytest.skip("frontend/package.json not in this context (e.g. backend-only image build)")
+    with open(_VERSION_JSON) as f:
+        canonical = json.load(f)["version"]
+    with open(pkg) as f:
+        fe = json.load(f)["version"]
+    assert fe == canonical, f"frontend/package.json {fe!r} != backend/version.json {canonical!r}"

backend/tests/test_vip_adoption_instance.py

@@ -0,0 +1,293 @@
+"""
+v1.10.8 — the four adoption defects found while testing v1.10.4 on a live HA pair.
+
+B1  The Apply Management "View Change" diff did not recognise the `adopt` action, so the
+    version fell through to the generic HAProxy diff and rendered the cluster's whole
+    haproxy.cfg as removed.
+
+B2  `vip_discoveries.adopted_vip_id` is write-once and nothing clears it, while a VIP is only
+    ever SOFT-deleted — so `ON DELETE SET NULL` never fires. Rejecting an adoption therefore
+    hid the node from the panel permanently: the VIP was gone from the VIP list too, and the
+    agent does not re-report a file whose hash has not changed. Adoptability is now derived
+    from whether the linked VIP is still active.
+
+B3  Adoption took only the node that was clicked. On a two-node pair that meant: adopting the
+    BACKUP produced a VIP whose apply fails ("exactly one member must be MASTER"), adopting the
+    MASTER left the peer unmanaged, and adopting the peer afterwards hit the VRID-collision
+    guard with 409. The pair could never be completed from the panel.
+
+B4  Worst of the four. `render_keepalived_conf` emits the unicast block only when it has peer
+    addresses, so a single-member adoption of a UNICAST instance silently dropped it and
+    keepalived fell back to multicast on that node while its peer stayed unicast — they stop
+    seeing each other and BOTH claim the VIP.
+
+B3 and B4 share one root and one fix: adoption now resolves the whole VRRP instance, keyed on
+(virtual_router_id, virtual address) exactly as keepalived groups nodes.
+
+These are source-level and unit tests: the adoption endpoint needs a live database, so the
+behaviour that can be exercised without one is pinned here, and the SQL/flow invariants are
+pinned by reading the module.
+"""
+import pathlib
+import re
+
+import pytest
+
+BACKEND = pathlib.Path(__file__).resolve().parents[1]
+VIP_ROUTER = (BACKEND / "routers" / "vip.py").read_text()
+CLUSTER_ROUTER = (BACKEND / "routers" / "cluster.py").read_text()
+RENDERER = (BACKEND / "services" / "keepalived_config.py").read_text()
+
+
+def _adopt_body() -> str:
+    start = VIP_ROUTER.index("async def adopt_vip")
+    return VIP_ROUTER[start:]
+
+
+# ----------------------------------------------------------------------------
+# B1 — the diff must recognise `adopt`
+# ----------------------------------------------------------------------------
+
+def test_view_change_diff_recognises_the_adopt_action():
+    match = re.search(r"vip_match = re\.search\(r'vip-\(\\d\+\)-\(([^)]+)\)'", CLUSTER_ROUTER)
+    assert match, "the vip version regex moved; re-point this test"
+    actions = set(match.group(1).split("|"))
+    assert actions == {"create", "update", "delete", "adopt"}, (
+        f"the View Change diff recognises {sorted(actions)}. An action missing here does not "
+        f"degrade gracefully: the version falls through to the generic HAProxy diff and shows "
+        f"the cluster's whole haproxy.cfg as removed."
+    )
+
+
+def test_every_staged_vip_action_is_covered_by_the_diff_regex():
+    """Whatever _stage_vip_version can be called with must be in that alternation."""
+    staged = set(re.findall(r'_stage_vip_version\(conn, vip_id, "(\w+)"', VIP_ROUTER))
+    match = re.search(r"vip_match = re\.search\(r'vip-\(\\d\+\)-\(([^)]+)\)'", CLUSTER_ROUTER)
+    recognised = set(match.group(1).split("|"))
+    assert staged, "no _stage_vip_version call sites found; re-point this test"
+    assert staged <= recognised, (
+        f"staged action(s) {sorted(staged - recognised)} are not recognised by the View Change "
+        f"diff regex {sorted(recognised)}"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B2 — adoptability follows the VIP's liveness, not the bare link
+# ----------------------------------------------------------------------------
+
+def test_discovery_api_exposes_whether_the_adopted_vip_still_stands():
+    assert "adopted_vip_active" in VIP_ROUTER, (
+        "the discovery payload no longer reports whether the adopted VIP is still active; the "
+        "UI would go back to hiding a rejected adoption forever"
+    )
+    assert re.search(r"LEFT JOIN vip_instances av ON av\.id = d\.adopted_vip_id", VIP_ROUTER), (
+        "the discovery query must join the linked VIP to report its is_active"
+    )
+
+
+def test_adopt_refuses_only_while_the_previous_adoption_still_stands():
+    body = _adopt_body()
+    assert re.search(r'if disc\["adopted_vip_id"\] and disc\["adopted_vip_active"\]', body), (
+        "adopt must refuse only when the linked VIP is still ACTIVE — refusing on the bare link "
+        "makes a rejected adoption impossible to retry, because nothing ever clears the column"
+    )
+
+
+def test_nothing_clears_adopted_vip_id_so_the_derivation_is_load_bearing():
+    """If a future change starts clearing the column, this test should be revisited rather than
+    silently left in place — the derived flag is what makes reject recoverable today."""
+    writes = re.findall(r"UPDATE vip_discoveries SET adopted_vip_id = (\S+)", VIP_ROUTER)
+    assert writes, "no adopted_vip_id write found; re-point this test"
+    assert all(w != "NULL" for w in writes), (
+        "adopted_vip_id is now cleared somewhere — re-check that the adopted_vip_active "
+        "derivation and this test still describe reality"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B3 — the whole VRRP instance is adopted, not one node
+# ----------------------------------------------------------------------------
+
+def test_participants_are_resolved_by_vrid_and_address():
+    assert "async def _collect_instance_participants" in VIP_ROUTER
+    start = VIP_ROUTER.index("async def _collect_instance_participants")
+    end = VIP_ROUTER.index("@router.post(\"/adopt\")", start)
+    body = VIP_ROUTER[start:end]
+    assert 'v.get("virtual_router_id") == vrid' in body and 'v.get("virtual_ip") == virtual_ip' in body, (
+        "instance identity must be (VRID, address) — the same key keepalived uses to decide two "
+        "nodes are one VRRP group"
+    )
+    assert 'if r["parse_error"]' in body, "a node whose config failed to parse must not become a member"
+    assert 'r["adopted_vip_id"] and r["adopted_vip_active"]' in body, (
+        "a node already held by a STANDING VIP must not be pulled into a second one"
+    )
+
+
+def test_adopt_inserts_one_member_per_participant():
+    body = _adopt_body()
+    insert = body.index("INSERT INTO vip_members")
+    preceding = body[:insert]
+    assert "for p in participants:" in preceding, (
+        "members must be inserted in a loop over the resolved participants; a single insert is "
+        "the half-adoption bug"
+    )
+    assert 'p["config_hash"]' in body[insert:insert + 800], (
+        "each member must carry ITS OWN takeover hash — the one-shot takeover guard is per node"
+    )
+
+
+def test_adopt_requires_exactly_one_master_across_the_instance():
+    body = _adopt_body()
+    assert 'roles.count("MASTER") != 1' in body, (
+        "adoption must reject an instance that does not have exactly one MASTER, instead of "
+        "letting apply fail later with 'exactly one member must be MASTER'"
+    )
+
+
+def test_adopt_enforces_one_active_vip_per_agent():
+    body = _adopt_body()
+    assert "already a member of VIP" in body, (
+        "adoption must enforce the one-active-VIP-per-agent rule that create/update enforce via "
+        "_validate_members_against_pool; a second membership never converges"
+    )
+
+
+# ----------------------------------------------------------------------------
+# B4 — a unicast instance can never be half-adopted
+# ----------------------------------------------------------------------------
+
+def test_renderer_only_emits_unicast_when_it_has_peers():
+    """The property that makes B4 dangerous. Pinned so the guard below keeps its reason."""
+    assert re.search(r"if use_unicast and peer_ips:", RENDERER), (
+        "the renderer no longer gates the unicast block on having peers; re-derive whether the "
+        "adoption guard is still needed"
+    )
+
+
+def test_adopt_refuses_a_unicast_peer_that_is_not_being_adopted():
+    body = _adopt_body()
+    assert "declared_peers" in body and "member_ips" in body, (
+        "adoption must verify every declared unicast peer is among the nodes being adopted"
+    )
+    assert "fall back to multicast" in body, (
+        "the refusal must explain the consequence — silently dropping a peer puts both nodes in "
+        "MASTER state on the same address"
+    )
+
+
+def test_adopt_refuses_to_strand_any_node_that_references_the_address():
+    """The half-adoption hole that instance resolution alone does not close.
+
+    Participant resolution can only match a node it can READ, that is ENABLED and that is in the
+    SAME pool. Each of those is a door a real member leaves through silently, and the nodes that
+    remain get rewritten while it keeps serving the same address unmanaged. Seen for real: one
+    node of a pair had a missing closing brace, so it parsed to nothing while its partner parsed
+    cleanly. Rather than guard each door, the endpoint asks whether ANY reported config mentions
+    this address and is not among the nodes being adopted.
+    """
+    body = _adopt_body()
+    assert "raw_config_masked LIKE" in body, (
+        "the check must be scoped to configs that reference THIS virtual address, so an unrelated "
+        "file elsewhere in the fleet does not block every adoption"
+    )
+    assert "NOT (a.id = ANY($2::int[]))" in body, (
+        "the guard must catch every node that is NOT a participant, not just the unparseable "
+        "ones — a disabled agent and a peer in another pool are stranded exactly the same way"
+    )
+    assert "COALESCE(av.is_active, FALSE) = FALSE" in body and "d.is_managed = FALSE" in body, (
+        "a node already under management is not stranded and must not block adoption"
+    )
+    for reason in ("could not be parsed", "agent is disabled", "different agent pool"):
+        assert reason in body, f"the refusal must be able to explain '{reason}'"
+
+
+@pytest.mark.parametrize("field,label", [
+    ("prefix_length", "prefix length"),
+    ("use_unicast", "unicast/multicast mode"),
+    ("track_haproxy", "HAProxy tracking"),
+])
+def test_adopt_requires_agreement_on_shared_vip_fields(field, label):
+    """These live on the VIP row and are re-rendered onto EVERY member, so taking them from the
+    node that happened to be clicked imposes its settings on the others. prefix_length is the
+    sharpest: the design refuses to guess a netmask for a live VIP, and copying one node's
+    netmask onto another is that same change by another name."""
+    body = _adopt_body()
+    assert f'("{field}", "{label}")' in body, (
+        f"{field} is written to the VIP row from one node's report; adoption must refuse when the "
+        f"nodes disagree about it"
+    )
+
+
+def test_adopt_requires_one_shared_vrrp_password():
+    body = _adopt_body()
+    assert "decrypt_vrrp_secret(enc)" in body, (
+        "Fernet is non-deterministic, so the per-node tokens cannot be compared as ciphertext — "
+        "they must be decrypted and compared as plaintext"
+    )
+    assert "do not share one VRRP password" in body, (
+        "adoption stores ONE secret and renders it onto every member, so a mismatch must be "
+        "refused rather than silently normalised"
+    )
+    assert "cannot be decrypted" in body, (
+        "a token we cannot decrypt must be an error, not silently treated as equal to another"
+    )
+
+
+def test_adopt_requires_reported_ips_before_trusting_the_peer_check():
+    body = _adopt_body()
+    assert "have not reported an IP address yet" in body, (
+        "the unicast peer check compares against member IPs, so a member without a reported IP "
+        "must block the check rather than silently pass it"
+    )
+
+
+# ----------------------------------------------------------------------------
+# The takeover authorisation is genuinely one-shot
+# ----------------------------------------------------------------------------
+
+def test_takeover_authorisation_is_retired_once_the_node_acks_our_config():
+    """`takeover_expected_hash` is the permission to overwrite a keepalived.conf that does NOT
+    carry our ownership marker. It was written at adoption and never cleared, so the release
+    notes' "authorises exactly ONE takeover" was only true as "for exactly that file content,
+    indefinitely" — restoring the pre-adoption file would have been silently overwritten again
+    with no fresh human approval."""
+    agent_router = (BACKEND / "routers" / "agent.py").read_text()
+    assert "takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL" in agent_router, (
+        "the status ack must retire the takeover authorisation once the node confirms our config"
+    )
+    assert "ELSE takeover_expected_hash END" in agent_router, (
+        "a non-matching ack must LEAVE the authorisation in place — dropping it on a partial or "
+        "failed deploy would leave the VIP unable to converge"
+    )
+
+
+def test_takeover_still_requires_the_pinned_hash_to_match_on_disk():
+    """The guard that stops an edit between adoption and Apply from being overwritten."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    guard = '[[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]'
+    assert script.count(guard) == 2, (
+        f"the takeover guard must be present in BOTH daemon copies (found {script.count(guard)}); "
+        f"a self-upgraded agent runs the in-script copy, a freshly installed one the heredoc"
+    )
+
+
+# ----------------------------------------------------------------------------
+# Backward compatibility
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize("guard", [
+    "version_name NOT LIKE 'vip-%'",     # bulk apply/reject still skip VIP versions
+])
+def test_vip_versions_stay_excluded_from_the_haproxy_apply_flow(guard):
+    assert guard in CLUSTER_ROUTER, (
+        "vip-* versions must stay out of the HAProxy apply/reject sweep; they are owned by the "
+        "VIP endpoints and are never served as haproxy.cfg"
+    )
+
+
+def test_vip_version_transition_matches_any_action():
+    """_transition_vip_versions must key on the VIP id alone, or a new action's PENDING row
+    would be stranded in Apply Management after apply/reject."""
+    assert 'f"vip-{vip_id}-%"' in VIP_ROUTER, (
+        "the PENDING -> APPLIED/REJECTED transition must match every action for the VIP"
+    )

backend/tests/test_vip_delete_approval.py

@@ -0,0 +1,69 @@
+"""Issue #27 safety (v1.7.2) — source-level guards for APPROVAL-GATED VIP deletion.
+
+The critical invariant: an agent must NEVER tear a VIP down without an explicit human approval.
+We enforce that by keeping the VIP is_active=TRUE (so the agent keeps serving it) when a delete
+is merely *requested*; only an APPROVE (apply) flips is_active=FALSE. These guards lock in that
+wiring so a future edit can't silently make delete immediate again. Pure (no DB)."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_delete_stages_for_approval_keeps_vip_running():
+    s = _read("routers/vip.py")
+    # A running (applied) VIP's delete sets pending_delete=TRUE and stages a delete version —
+    # it must NOT flip is_active=FALSE in delete_vip (that only happens on approval in apply_vip).
+    assert "pending_delete=TRUE" in s
+    assert '_stage_vip_version(conn, vip_id, "delete"' in s
+    # The staged-delete branch (running VIP) must not contain an is_active=FALSE soft-delete;
+    # only the "never applied" branch may remove immediately (guarded by applied_snapshot IS NULL).
+    assert 'v["applied_snapshot"] is None' in s
+
+
+def test_apply_performs_delete_only_on_approval():
+    s = _read("routers/vip.py")
+    # apply_vip short-circuits on pending_delete and only THEN flips is_active=FALSE.
+    assert 'if v["pending_delete"]:' in s
+    assert "is_active=FALSE, pending_delete=FALSE" in s
+
+
+def test_reject_cancels_delete_as_noop():
+    s = _read("routers/vip.py")
+    # reject_vip clears the staged delete + purge intent; the VIP keeps running (is_active never
+    # touched here) — the "nothing happened" path.
+    assert "pending_delete=FALSE, purge_on_teardown=FALSE" in s
+    assert "Deletion rejected" in s
+
+
+def test_agent_delivery_teardown_only_on_inactive():
+    # The agent is told to tear down ONLY when is_active=FALSE; a pending-delete VIP is still
+    # is_active=TRUE, so the delivery returns 'available' and the agent keeps serving it.
+    s = _read("routers/agent.py")
+    assert "if not row['is_active']:" in s
+    assert '"status": "teardown"' in s
+
+
+def test_migration_adds_pending_delete_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "pending_delete BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7
+
+
+def test_list_visibility_backward_compat_gates_on_applied():
+    # A VIP soft-deleted under the OLD immediate-delete (pre-1.7.2: is_active=FALSE,
+    # last_config_status='PENDING') must NOT reappear in the list as DELETING — the
+    # teardown-tracking clause is gated on last_config_status='APPLIED' (new-flow approved
+    # deletes only). Locks the backward-compat fix.
+    s = _read("routers/vip.py")
+    assert "v.last_config_status = 'APPLIED' AND EXISTS" in s

backend/tests/test_vip_diff_minimal.py

@@ -0,0 +1,34 @@
+"""v1.7.5 — guard for the HA/VIP "View Change" diff.
+
+Editing a VIP (e.g. priority + virtual IP) must show a REAL line diff (only the changed lines)
+against the previous applied vip-* config — not the whole keepalived.conf marked as "added", and
+without the doubled "+ +" prefix (the line must be stored WITHOUT a +/- prefix; the UI adds it
+from `type`, exactly like the standard haproxy diff). Pure source guard (the behavioural diff
+test needs a DB); locks the wiring so it can't regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _cluster_src() -> str:
+    with open(os.path.join(ROOT, "routers", "cluster.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_vip_diff_uses_real_difflib_against_previous_applied():
+    s = _cluster_src()
+    # The previous APPLIED vip-* config for this VIP is fetched as the diff baseline.
+    assert "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1" in s
+    # And the diff is a real unified_diff of old vs new (not "everything added").
+    assert "difflib.unified_diff(old_content.split('\\n'), new_content.split('\\n')" in s
+
+
+def test_vip_diff_stores_lines_without_prefix():
+    s = _cluster_src()
+    # The old vip bug prepended "+ {line}", doubling the UI prefix ("+ +"). The vip diff now
+    # stores the stripped diff line (dl[1:]) — the UI adds the +/- from `type`. `dl` is unique
+    # to the vip branch (the standard haproxy diff uses `line`), so this targets the vip fix
+    # only and does not touch the (separate, out-of-scope) SSL diff branch.
+    assert '"line": dl[1:], "line_number": line_number' in s

backend/tests/test_vip_discoveries_scope.py

@@ -0,0 +1,87 @@
+"""
+v1.10.6 — GET /api/vip/discoveries resolves to its own handler and honours cluster_id.
+
+Two defects, one endpoint, found in that order on a live fleet:
+
+1. The route was declared after `GET /{vip_id}`, so FastAPI matched it there and answered
+   422 ("discoveries" is not an int) before the handler ran. The static declaration-order
+   guard lives in test_router_path_shadowing.py; this file pins the observable behaviour,
+   because a 422 is what the browser actually saw.
+
+2. Once reachable, it returned every discovery in the fleet regardless of the cluster
+   selected in the header, so a multi-cluster install saw one undifferentiated list. The
+   endpoint now takes the same optional `cluster_id` the VIP list takes.
+
+The auth tests here deliberately assert `!= 422`: the repo's generic endpoint-auth tests
+accept 401/403/422 together, which is precisely why defect 1 slipped through them.
+"""
+import pathlib
+import re
+
+import pytest
+
+VIP_ROUTER = pathlib.Path(__file__).resolve().parents[1] / "routers" / "vip.py"
+
+
+# ----------------------------------------------------------------------------
+# 1. The route reaches its own handler (defect 1)
+# ----------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path", [
+    "/api/vip/discoveries",
+    "/api/vip/discoveries?cluster_id=7",
+])
+def test_discoveries_route_is_not_captured_by_the_vip_id_route(client, path):
+    res = client.get(path)
+    assert res.status_code != 422, (
+        f"GET {path} returned 422 — the request was routed into the get-one-VIP handler, "
+        f"which parses the path segment as an int. Declaration order regressed. "
+        f"Body: {res.text[:200]}"
+    )
+    assert res.status_code in (401, 403), (
+        f"GET {path} without a token should be refused by the vip.read gate, got "
+        f"{res.status_code}. Body: {res.text[:200]}"
+    )
+
+
+def test_get_one_vip_still_parses_a_numeric_id(client):
+    """Moving /discoveries above /{vip_id} must not shadow the numeric route itself."""
+    res = client.get("/api/vip/12")
+    assert res.status_code in (401, 403), res.text[:200]
+
+
+# ----------------------------------------------------------------------------
+# 2. cluster_id is accepted and actually scopes the query (defect 2)
+# ----------------------------------------------------------------------------
+
+def test_handler_accepts_cluster_id():
+    from routers.vip import list_vip_discoveries
+    import inspect
+
+    params = inspect.signature(list_vip_discoveries).parameters
+    assert "cluster_id" in params, (
+        "list_vip_discoveries no longer takes cluster_id; the HA/VIP page would show every "
+        "cluster's nodes at once again"
+    )
+    assert params["cluster_id"].default is None, (
+        "cluster_id must stay optional — omitting it returns the whole fleet, which is what "
+        "a caller that predates the parameter expects"
+    )
+
+
+def test_discovery_query_scopes_by_the_cluster_pool():
+    """The filter must resolve cluster -> pool the same way the VIP list does, and must be a
+    no-op when the parameter is absent."""
+    source = VIP_ROUTER.read_text()
+    start = source.index("async def list_vip_discoveries")
+    end = source.index("def _find_candidate", start)
+    body = source[start:end]
+
+    assert "FROM vip_discoveries" in body, "the discovery query moved; re-point this test"
+    assert re.search(r"a\.pool_id\s*=\s*\(\s*SELECT\s+pool_id\s+FROM\s+haproxy_clusters", body), (
+        "the cluster filter must map cluster -> pool via haproxy_clusters, matching list_vips"
+    )
+    assert "IS NULL" in body, (
+        "the filter must short-circuit when cluster_id is absent, so an unscoped call still "
+        "returns the whole fleet"
+    )

backend/tests/test_vip_models.py

@@ -0,0 +1,91 @@
+"""Issue #27 (v1.7.0) — unit tests for VIP request-model validation (pure, no DB)."""
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from models.vip import VIPCreate, VIPMemberIn  # noqa: E402
+
+
+def _members(master_prio=150, backup_prio=100, n_backup=1):
+    members = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": master_prio}]
+    for i in range(n_backup):
+        members.append({"agent_id": 2 + i, "network_interface": "eth0", "role": "BACKUP", "priority": backup_prio})
+    return members
+
+
+def _create(**over):
+    base = dict(name="web-vip", pool_id=1, virtual_ip="10.0.0.100", members=_members())
+    base.update(over)
+    return VIPCreate(**base)
+
+
+class TestField:
+    def test_ok(self):
+        v = _create()
+        assert v.virtual_ip == "10.0.0.100"
+        assert v.use_unicast is True  # cloud-safe default
+
+    def test_ipv6_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="fd00::1")
+
+    def test_bad_ip_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="not-an-ip")
+
+    def test_auth_pass_max_8(self):
+        _create(auth_pass="12345678")
+        with pytest.raises(ValueError):
+            _create(auth_pass="123456789")
+
+    def test_interface_forbidden_char(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0; rm -rf /", role="BACKUP", priority=100)
+
+    def test_priority_range(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0", role="BACKUP", priority=255)
+
+    def test_role_normalized(self):
+        m = VIPMemberIn(agent_id=1, network_interface="eth0", role="master", priority=150)
+        assert m.role == "MASTER"
+
+    def test_vrid_range(self):
+        with pytest.raises(ValueError):
+            _create(virtual_router_id=300)
+
+
+class TestMembers:
+    def test_single_node_allowed(self):
+        # A single-node VIP (one MASTER, no BACKUP) is valid: a keepalived-managed floating
+        # IP without failover (Issue #27 follow-up — relaxed from >=2 members to >=1, e.g. a
+        # one-box cluster that wants a stable VIP, or before a 2nd node is added for real HA).
+        v = VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5",
+                      members=[{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150}])
+        assert len(v.members) == 1 and v.members[0].role == "MASTER"
+
+    def test_needs_at_least_one(self):
+        # An empty membership is still rejected — a VIP must have at least one node.
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=[])
+
+    def test_exactly_one_master(self):
+        bad = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 2, "network_interface": "eth0", "role": "MASTER", "priority": 140}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=bad)
+
+    def test_master_must_be_highest(self):
+        with pytest.raises(ValueError):
+            _create(members=_members(master_prio=100, backup_prio=120))
+
+    def test_no_duplicate_agent(self):
+        dup = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 1, "network_interface": "eth1", "role": "BACKUP", "priority": 100}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=dup)

backend/tests/test_vip_purge.py

@@ -0,0 +1,51 @@
+"""Issue #27 follow-up (v1.7.2) — source-level guards for the opt-in keepalived package
+uninstall + the "we installed it" marker. Pure (no DB), mirroring the project's other
+source-assertion tests: they lock in the *safe defaults* so a future edit can't silently
+turn routine VIP deletion into a package purge, or purge a package we didn't install."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_agent_script_purge_is_optin_and_marker_guarded():
+    s = _read("utils/agent_scripts/linux_install.sh")
+    # The install marker is written only when WE install keepalived, and read by the purge
+    # guard — present in BOTH function copies (installer-mode + SKIP_TO_DAEMON).
+    assert s.count(".hom_installed") >= 4
+    # Multi-distro purge cascade exists, but ONLY inside the opt-in branch.
+    assert "apt-get purge -y -qq keepalived" in s
+    assert "apk del keepalived" in s
+    # Orphan self-heal (not_configured) must NEVER purge — graceful teardown only, both copies.
+    assert s.count('_kp_teardown "false"') >= 2
+    # Purge is honored only on an explicit teardown, parsed from the delivery response.
+    assert s.count(".purge // false") >= 2
+
+
+def test_agent_delivery_signals_purge_on_teardown():
+    s = _read("routers/agent.py")
+    assert "v.purge_on_teardown" in s
+    assert '"purge"' in s  # teardown response carries the opt-in flag
+
+
+def test_delete_endpoint_accepts_purge_package_default_off():
+    s = _read("routers/vip.py")
+    # Query param defaults to False (safe), and it sets the persisted teardown flag.
+    assert "purge_package: bool = False" in s
+    assert "purge_on_teardown=$2" in s
+
+
+def test_migration_adds_purge_column_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7

backend/tests/test_waf_stick_table_autoinject.py

@@ -0,0 +1,34 @@
+"""v1.7.4 — guard for the stick-table auto-inject in the HAProxy config renderer.
+
+A frontend that uses a stick counter (`track-sc<N>` or an `sc_*_rate(...)` fetch) but declares
+no `stick-table` makes HAProxy fatally reject the WHOLE cluster config with "table '<frontend>'
+used but not configured". This happens with rate-limit directives baked into a frontend's stored
+request_headers/options by an older version or a config import. The renderer now injects a default
+stick-table in that case. Pure source guard (the behavioural renderer test needs a DB and is
+disabled); this locks the wiring so it can't silently regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src() -> str:
+    with open(os.path.join(ROOT, "services", "haproxy_config.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_renderer_detects_stick_counter_usage():
+    s = _src()
+    # Detection: any track-sc<N> write OR an sc_*_rate(...) fetch flips _sc_counter_used.
+    assert "_sc_counter_used" in s
+    assert '"track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped)' in s
+
+
+def test_renderer_injects_stick_table_when_missing():
+    s = _src()
+    # Inject ONLY when a counter is used AND no stick-table was emitted (purely additive).
+    assert "if _sc_counter_used and not _stick_table_emitted:" in s
+    assert 'stick-table type ip size 100k expire 30s store http_req_rate(10s)' in s
+    # The inject must register the table so it is not added twice.
+    assert "_stick_table_emitted = True" in s

backend/utils/acme_backend_url.py

@@ -0,0 +1,266 @@
+"""Validation and resolution for the ACME HTTP-01 challenge backend URL.
+
+This URL tells HAProxy where to proxy ``/.well-known/acme-challenge/*``. It is
+rendered into ``backend _acme_challenge_backend`` as ``server _acme_mgmt host:port``
+and — this is the part that makes it unlike every other URL in the product —
+**resolved on the HAProxy node, not on the management host**. A value that works
+when pasted into the management server's own browser can be completely dead from
+the data plane.
+
+Two entry points, deliberately asymmetric:
+
+``validate_acme_backend_url``
+    Called at the WRITE BOUNDARY (cluster PUT, settings PUT). Rejects values that
+    cannot express a reachable target. Strict here is safe: it only ever affects a
+    value an operator is typing right now, and the error text can teach.
+
+``resolve_acme_backend_target``
+    Called at RENDER TIME. Never raises, never rejects. Strictness here would be a
+    catastrophe: the shipped defaults (``config.py`` ``http://localhost:8000``,
+    ``docker-compose.yml`` ``http://localhost:8080``) mean essentially every
+    existing install resolves to loopback today, and refusing to render would make
+    every ``acme_enabled`` cluster unappliable — including for urgent changes that
+    have nothing to do with ACME. It reports problems instead of enforcing them.
+
+Two conscious departures from ``utils/ssrf_guard.py``, whose policy is the exact
+opposite of what is needed here:
+
+* **RFC1918 is allowed, and is usually the correct answer.** The guard exists to
+  stop the server being tricked into dialling internal space. Here the operator is
+  deliberately naming their own management host, which on a split deployment is
+  private by definition.
+* **No DNS resolution.** Resolving from the management host would re-introduce the
+  very wrong-vantage-point mistake this work exists to remove: what this box can
+  resolve says nothing about what the HAProxy node can reach.
+"""
+import ipaddress
+import re
+from typing import List, NamedTuple, Optional
+from urllib.parse import urlparse
+
+# `haproxy_clusters.acme_backend_url` / `system_settings.value` are VARCHAR(500).
+# Without this check asyncpg raises 22001 and the operator gets an opaque 500.
+MAX_URL_LENGTH = 500
+
+ALLOWED_SCHEMES = ("http", "https")
+
+# Port assumed when the URL omits one. NOT the scheme's default: the bundled
+# docker-compose publishes nginx on 8080 (`nginx/nginx.conf` listens 8080,
+# `docker-compose.yml` maps 8080:8080), so an operator who wrote a bare
+# `http://10.0.0.5` has a WORKING path today that resolves to :8080. Changing this
+# to 80 would break those installs silently — the first symptom would be the
+# unattended renewal loop failing months later. The value is kept and the omission
+# is surfaced as a warning instead.
+DEFAULT_HTTP_PORT = 8080
+DEFAULT_HTTPS_PORT = 443
+
+# RFC 1123 host label set. Deliberately not a full IDN implementation: an operator
+# naming their management host in a config file pushed to HAProxy nodes should use
+# ASCII, and HAProxy itself would not accept anything else on a `server` line.
+_HOSTNAME_RE = re.compile(
+    r"^(?=.{1,253}$)[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?"
+    r"(\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*\.?$"
+)
+
+_CONTROL_CHARS = frozenset("\t\n\r\v\f\x00")
+
+
+class AcmeBackendUrlError(ValueError):
+    """A value that cannot express a usable challenge backend target.
+
+    ``code`` is stable and machine-readable so the UI can map it to help text;
+    ``args[0]`` is operator-facing prose.
+    """
+
+    def __init__(self, code: str, message: str):
+        super().__init__(message)
+        self.code = code
+
+
+class AcmeBackendTarget(NamedTuple):
+    """What the renderer should emit, plus everything worth telling the operator."""
+
+    host: str
+    port: int
+    ssl_flag: str
+    #: Non-fatal observations. Rendered anyway; surfaced in logs and the panel.
+    warnings: List[str]
+    #: Set when the value could not be parsed at all and the caller must not emit
+    #: a `server` line. None on success.
+    error_code: Optional[str]
+    error_message: Optional[str]
+
+
+def _classify_host(host: str) -> Optional[str]:
+    """Return a rejection code for hosts that cannot be a management address."""
+    try:
+        ip = ipaddress.ip_address(host)
+    except ValueError:
+        lowered = host.rstrip(".").lower()
+        # `localhost` is loopback by name and is the single most likely wrong value
+        # here — it is what both shipped defaults contain. Catching only the numeric
+        # form would let the exact failure this module exists to prevent straight
+        # through. RFC 6761 also reserves the whole `.localhost` tree.
+        if lowered == "localhost" or lowered.endswith(".localhost"):
+            return "loopback"
+        return None if _HOSTNAME_RE.match(host) else "invalid_host"
+
+    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    if ip.is_loopback:
+        return "loopback"
+    if ip.is_unspecified:
+        return "unspecified"
+    # Includes 169.254.169.254, the cloud metadata endpoint.
+    if ip.is_link_local:
+        return "link_local"
+    if ip.is_multicast:
+        return "multicast"
+    # NOTE: private (RFC1918) addresses fall through on purpose — see module docstring.
+    return None
+
+
+_REJECTION_PROSE = {
+    "too_long": f"URL must be at most {MAX_URL_LENGTH} characters.",
+    "whitespace": (
+        "URL must not contain spaces or line breaks. A trailing space survives parsing "
+        "and would be written into haproxy.cfg as part of the address."
+    ),
+    "no_scheme": (
+        "URL must start with http:// or https://. Without a scheme the value cannot be "
+        "parsed as an address and silently falls back to localhost, which on a HAProxy "
+        "node means the node itself."
+    ),
+    "bad_scheme": "URL scheme must be http or https.",
+    "userinfo": "URL must not contain credentials.",
+    "has_path": (
+        "Enter only the scheme, host and port — no path, query or fragment. The "
+        "challenge path is appended by HAProxy."
+    ),
+    "bad_port": "Port must be a number between 1 and 65535.",
+    "no_host": "URL must contain a host.",
+    "invalid_host": "Host is not a valid IP address or hostname.",
+    "loopback": (
+        "Loopback addresses cannot work here. HAProxy resolves this address on the "
+        "HAProxy node, so 127.0.0.1 means the node itself, not the management server. "
+        "Use the management server's routable address."
+    ),
+    "unspecified": (
+        "0.0.0.0 is a listen address, not a destination. Use the management server's "
+        "routable address."
+    ),
+    "link_local": "Link-local addresses cannot be used as a management address.",
+    "multicast": "Multicast addresses cannot be used as a management address.",
+}
+
+
+def validate_acme_backend_url(value: Optional[str]) -> Optional[str]:
+    """Validate an operator-supplied URL at the write boundary.
+
+    Returns the normalised value (stripped), or ``None`` for empty input, which
+    legitimately means "inherit from the next level of the resolution chain".
+    Raises :class:`AcmeBackendUrlError` otherwise.
+    """
+    if value is None:
+        return None
+    if not isinstance(value, str):
+        raise AcmeBackendUrlError("invalid_host", _REJECTION_PROSE["invalid_host"])
+
+    stripped = value.strip()
+    if not stripped:
+        return None
+
+    if len(stripped) > MAX_URL_LENGTH:
+        raise AcmeBackendUrlError("too_long", _REJECTION_PROSE["too_long"])
+    if any(c in _CONTROL_CHARS for c in stripped) or " " in stripped:
+        raise AcmeBackendUrlError("whitespace", _REJECTION_PROSE["whitespace"])
+
+    parsed = urlparse(stripped)
+
+    if not parsed.scheme:
+        raise AcmeBackendUrlError("no_scheme", _REJECTION_PROSE["no_scheme"])
+    if parsed.scheme.lower() not in ALLOWED_SCHEMES:
+        # `10.0.0.5:8080` parses as scheme='10.0.0.5' with no netloc, and
+        # `localhost:8080` as scheme='localhost'. Both are the same operator mistake,
+        # so point at the missing scheme rather than the nonsense one.
+        if not parsed.netloc:
+            raise AcmeBackendUrlError("no_scheme", _REJECTION_PROSE["no_scheme"])
+        raise AcmeBackendUrlError("bad_scheme", _REJECTION_PROSE["bad_scheme"])
+
+    if parsed.username is not None or parsed.password is not None:
+        raise AcmeBackendUrlError("userinfo", _REJECTION_PROSE["userinfo"])
+    if parsed.path not in ("", "/") or parsed.query or parsed.fragment:
+        raise AcmeBackendUrlError("has_path", _REJECTION_PROSE["has_path"])
+
+    try:
+        port = parsed.port
+    except ValueError:
+        # urlparse defers port parsing to attribute access; an out-of-range or
+        # non-numeric port raises here. Unguarded, this exception reaches the config
+        # generator's blanket `except` and collapses the cluster's whole config.
+        raise AcmeBackendUrlError("bad_port", _REJECTION_PROSE["bad_port"]) from None
+    if port is not None and not (1 <= port <= 65535):
+        raise AcmeBackendUrlError("bad_port", _REJECTION_PROSE["bad_port"])
+
+    host = parsed.hostname
+    if not host:
+        raise AcmeBackendUrlError("no_host", _REJECTION_PROSE["no_host"])
+
+    code = _classify_host(host)
+    if code is not None:
+        raise AcmeBackendUrlError(code, _REJECTION_PROSE[code])
+
+    return stripped
+
+
+def resolve_acme_backend_target(url: Optional[str]) -> AcmeBackendTarget:
+    """Resolve a stored URL into what the renderer emits. Never raises.
+
+    Values already in the database predate validation (and the shipped defaults are
+    themselves loopback), so anything unparseable or discouraged is reported through
+    ``warnings`` / ``error_code`` rather than refused.
+    """
+    warnings: List[str] = []
+    raw = (url or "").strip()
+
+    if not raw:
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "empty", "No challenge backend URL configured."
+        )
+
+    if any(c in _CONTROL_CHARS for c in raw) or " " in raw:
+        # Must never reach haproxy.cfg: a newline here writes attacker- or
+        # accident-chosen directives into a file pushed to every node.
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "whitespace", _REJECTION_PROSE["whitespace"]
+        )
+
+    parsed = urlparse(raw)
+    scheme = (parsed.scheme or "").lower()
+
+    try:
+        port = parsed.port
+    except ValueError:
+        return AcmeBackendTarget("", 0, "", warnings, "bad_port", _REJECTION_PROSE["bad_port"])
+
+    host = parsed.hostname
+    if not host or scheme not in ALLOWED_SCHEMES:
+        return AcmeBackendTarget(
+            "", 0, "", warnings, "no_scheme", _REJECTION_PROSE["no_scheme"]
+        )
+
+    if port is None:
+        port = DEFAULT_HTTPS_PORT if scheme == "https" else DEFAULT_HTTP_PORT
+        warnings.append(
+            f"No port given, assuming {port}. State the port explicitly — the assumed "
+            f"value is the bundled reverse proxy's port, not the scheme's default."
+        )
+
+    code = _classify_host(host)
+    if code == "invalid_host":
+        return AcmeBackendTarget("", 0, "", warnings, code, _REJECTION_PROSE[code])
+    if code is not None:
+        warnings.append(_REJECTION_PROSE[code])
+
+    ssl_flag = " ssl verify none" if scheme == "https" else ""
+    return AcmeBackendTarget(host, port, ssl_flag, warnings, None, None)

backend/utils/agent_scripts/linux_install.sh

@@ -153,7 +153,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -470,7 +470,12 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "haproxy-agent.service"; do
+# issue #31: match ONLY the installed agent (binary path + service), never the bare string
+# "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command line
+# or a sudo/PAM ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself ("Killed", install aborts). The systemd service is also stopped below; the
+# "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "haproxy-agent.service"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -635,7 +640,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -1055,7 +1060,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1289,7 +1299,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -1309,7 +1319,7 @@ get_keepalive_state() {
     # Method 1: journalctl (most reliable on RHEL/CentOS/Ubuntu with systemd)
     if command -v journalctl &>/dev/null; then
         state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-            | grep -oE "(MASTER|BACKUP)" | tail -1)
+            | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
     fi
 
     # Method 2: Fallback to log files (for non-systemd or restricted journalctl)
@@ -1317,22 +1327,27 @@ get_keepalive_state() {
         for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
             if [[ -r "$logfile" ]]; then
                 state=$(tail -200 "$logfile" 2>/dev/null \
-                    | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                    | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                 [[ -n "$state" ]] && break
             fi
         done
     fi
 
-    # Method 3: Check VIP presence on network interfaces (confirms MASTER)
+    # Method 3: VIP presence on local interfaces — the most portable signal, independent
+    # of logging/journald (works on any distro / init system / keepalived install type).
+    # MASTER iff ANY configured VIP is actually held locally; keepalived up but holding no
+    # VIP => BACKUP. Exact whole-line IP match (grep -Fxq) so 10.0.0.1 can't falsely match
+    # 10.0.0.10/100, and ALL configured VIPs are checked (not just the first).
     if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-        local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-            | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
-        if [[ -n "$conf_vip" ]]; then
-            if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
-                state="MASTER"
-            else
-                state="BACKUP"
-            fi
+        local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+            | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
+        if [[ -n "$conf_vips" ]]; then
+            local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
+            state="BACKUP"
+            local _cvip
+            for _cvip in $conf_vips; do
+                if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+            done
         fi
     fi
 
@@ -1352,7 +1367,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -1665,6 +1685,221 @@ check_ssl_updates() {
 }
 
 # Reload HAProxy service (Linux specific)
+# Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence. OPT-IN and inert by default:
+# a node with no APPLIED VIP gets status=not_configured and (lacking our ownership
+# marker) does NOTHING. Never clobbers a hand-managed keepalived. Uses $AGENT_TOKEN
+# (kept in sync with the live token in both daemon loops).
+fetch_and_deploy_keepalived_config() {
+    local marker="# Managed by HAProxy OpenManager"
+    local resp status http_code we_own="false" conf chk
+
+    # Timeouts so a hung management server can never stall the daemon loop.
+    resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+        "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    http_code="${resp##*$'\n'}"   # last line = HTTP status
+    resp="${resp%$'\n'*}"          # everything before = JSON body
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+    # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+    # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+    [[ "$http_code" == "404" ]] && status="teardown"
+    # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+    conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+    [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+    chk="$(dirname "$conf")/check_haproxy.sh"
+    if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+    # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
+    # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
+    # to the node. The heartbeat cannot carry this — it has the VIP address and a best-effort
+    # MASTER/BACKUP, while rendering a node's config needs eleven fields.
+    #
+    # Rate limited by content: the hash of the last report is cached next to the config, so the
+    # file (which may contain the VRRP password) is posted only when it actually changes, not on
+    # every cycle. Once we own the file there is nothing to adopt, so the record is cleared once.
+    _kp_discover() {
+        local cache="$(dirname "$conf")/.hom_discovery_hash" cur_hash="" body content_json
+        if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
+            # Nothing adoptable here. Clear a previous report exactly once.
+            [[ -f "$cache" ]] || return 0
+            curl -k -s --connect-timeout 10 --max-time 30 -X POST \
+                "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"config_path\":\"$conf\",\"exists\":false}" >/dev/null 2>&1 || return 0
+            rm -f "$cache"
+            return 0
+        fi
+        cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        [[ -z "$cur_hash" ]] && return 0
+        [[ -f "$cache" && "$(cat "$cache" 2>/dev/null)" == "$cur_hash" ]] && return 0
+        # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
+        # server hashes is byte-identical to what is on disk — the takeover authorisation is
+        # pinned to that hash.
+        content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
+        body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
+            '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
+        if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST \
+            "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            --data-binary "$body" 2>/dev/null; then
+            printf '%s' "$cur_hash" > "$cache" 2>/dev/null
+            log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+        fi
+    }
+    _kp_discover
+
+    _kp_report() {  # $1=state  $2=vip_id(or empty)  $3=hash  $4=message
+        local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+        curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+            >/dev/null 2>&1 || true
+    }
+    _kp_teardown() {
+        # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+        # keepalived and remove OUR config (the VIP is released) but KEEP the package — the safe
+        # enterprise default. Purge only when opted in AND we are the ones who installed it
+        # (the .hom_installed marker); never remove a package the admin pre-installed.
+        local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+        log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+        systemctl stop keepalived >/dev/null 2>&1
+        systemctl disable keepalived >/dev/null 2>&1
+        rm -f "$conf" "$chk"
+        if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+            log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+            fi
+            rm -f "$marker_inst"
+            if command -v keepalived >/dev/null 2>&1; then
+                _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+            else
+                _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+            fi
+        elif [[ "$purge" == "true" ]]; then
+            log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+            _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+        else
+            _kp_report "disabled" "" "" "torn down"
+        fi
+    }
+
+    case "$status" in
+        available) : ;;  # fall through to converge
+        teardown)
+            # Explicit server-side delete. Honor the operator's opt-in package purge (.purge);
+            # marker-guarded inside _kp_teardown, and a node we don't own (no marker conf) is a no-op.
+            local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+            [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"
+            return 0 ;;
+        not_configured)
+            [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only, never purge
+            return 0 ;;
+        *) return 0 ;;   # unknown / auth error → no-op
+    esac
+
+    # --- status == available: converge ---
+    local vip_id new_conf check_script install_if new_hash
+    vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+    new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+    new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+    check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+    install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+    [[ -z "$new_conf" ]] && return 0
+
+    # Ownership guard: never overwrite a keepalived.conf we don't own.
+    #
+    # v1.10.4 adoption is the ONE exception, and it does not weaken the guard: the server
+    # authorises a single takeover of a specific file by pinning the md5 the operator adopted
+    # from. We overwrite only when that hash still matches what is on disk, so a config edited
+    # between adoption and Apply is still refused — the operator's later edit wins over a stale
+    # adoption rather than being silently destroyed.
+    if [[ -f "$conf" && "$we_own" != "true" ]]; then
+        local allow_takeover expected_hash disk_hash
+        allow_takeover=$(echo "$resp" | jq -r '.keepalived.allow_takeover // false' 2>/dev/null)
+        expected_hash=$(echo "$resp" | jq -r '.keepalived.takeover_expected_hash // empty' 2>/dev/null)
+        disk_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        if [[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]; then
+            log "INFO" "KEEPALIVED: adopting $conf (one-shot takeover authorised; on-disk hash matches)"
+        elif [[ "$allow_takeover" == "true" ]]; then
+            log "WARN" "KEEPALIVED: adoption authorised but $conf changed since it was adopted — refusing"
+            _kp_report "externally_managed" "$vip_id" "$disk_hash" \
+                "config changed after adoption; re-adopt to pick up the current file"
+            return 0
+        else
+            log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+            _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+            return 0
+        fi
+    fi
+
+    # Hybrid install: install keepalived only if missing.
+    if ! command -v keepalived >/dev/null 2>&1; then
+        if [[ "$install_if" == "true" ]]; then
+            log "INFO" "KEEPALIVED: installing package..."
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+            fi
+        fi
+        if ! command -v keepalived >/dev/null 2>&1; then
+            log "ERROR" "KEEPALIVED: not installed/available"
+            _kp_report "error" "$vip_id" "" "keepalived not installed"
+            return 0
+        fi
+        # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+        # delete removes only OUR install, never an admin's pre-existing keepalived package.
+        mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+    fi
+
+    # Idempotency: skip write+reload when the on-disk content already matches.
+    if [[ -f "$conf" ]]; then
+        local cur_hash would_hash
+        cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+        if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            return 0
+        fi
+    fi
+
+    mkdir -p "$(dirname "$conf")"
+    if [[ -n "$check_script" ]]; then
+        printf '%s' "$check_script" > "$chk"
+        chown root:root "$chk" 2>/dev/null
+        chmod 0755 "$chk"   # root-owned, not world-writable — required by enable_script_security
+    fi
+    # Validate on a TEMP file and swap in only on success: a bad render must never land on
+    # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+    local tmp_conf="${conf}.hom.tmp"
+    printf '%s' "$new_conf" > "$tmp_conf"
+    chmod 0644 "$tmp_conf"
+    if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+        rm -f "$tmp_conf"
+        log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+        _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+        return 0
+    fi
+    mv -f "$tmp_conf" "$conf"
+    chown root:root "$conf" 2>/dev/null
+    chmod 0644 "$conf"
+    systemctl enable keepalived >/dev/null 2>&1
+    if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+        log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+        _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+    else
+        log "ERROR" "KEEPALIVED: reload/restart failed"
+        _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+    fi
+    return 0
+}
+
 reload_haproxy_service() {
     log "INFO" "Performing zero-downtime HAProxy configuration reload..."
     
@@ -2356,8 +2591,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence at the SSL cadence (inert for non-VIP nodes)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2789,28 +3028,31 @@ SYSTEM_INFO_EOF
 
         if command -v journalctl &>/dev/null; then
             state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-                | grep -oE "(MASTER|BACKUP)" | tail -1)
+                | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
         fi
 
         if [[ -z "$state" ]]; then
             for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
                 if [[ -r "$logfile" ]]; then
                     state=$(tail -200 "$logfile" 2>/dev/null \
-                        | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                        | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                     [[ -n "$state" ]] && break
                 fi
             done
         fi
 
+        # Method 3: VIP presence on local interfaces — portable, logging-independent.
+        # MASTER iff ANY configured VIP is held locally (exact whole-line match, all VIPs).
         if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-            local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-                | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
-            if [[ -n "$conf_vip" ]]; then
-                if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
-                    state="MASTER"
-                else
-                    state="BACKUP"
-                fi
+            local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+                | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
+            if [[ -n "$conf_vips" ]]; then
+                local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
+                state="BACKUP"
+                local _cvip
+                for _cvip in $conf_vips; do
+                    if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+                done
             fi
         fi
 
@@ -2849,7 +3091,15 @@ SYSTEM_INFO_EOF
         local keepalive_info=$(get_keepalive_state)
         local keepalive_state=$(echo "$keepalive_info" | cut -d'|' -f1)
         local keepalive_ip=$(echo "$keepalive_info" | cut -d'|' -f2)
-        
+
+        # Network interfaces (DAEMON) — reported so the HA/VIP create form can offer this
+        # node's real NICs. /sys/class/net is universal across distros (no iproute2 needed);
+        # exclude loopback here, and the UI further hides docker/veth/bridge interfaces. Empty
+        # -> "[]" (never "[\"\"]") so a node with no NICs doesn't surface a blank option.
+        local net_ifaces ni_json
+        net_ifaces=$(ls /sys/class/net 2>/dev/null | grep -vE '^lo$' | tr '\n' ',' | sed 's/,$//')
+        if [ -n "$net_ifaces" ]; then ni_json="[\"${net_ifaces//,/\",\"}\"]"; else ni_json="[]"; fi
+
         # Prepare heartbeat payload with all data
         local heartbeat_payload="{
             \"name\": \"$AGENT_NAME\",
@@ -2859,6 +3109,8 @@ SYSTEM_INFO_EOF
             \"platform\": \"$platform\",
             \"architecture\": \"$(uname -m)\",
             \"version\": \"{{AGENT_VERSION}}\",
+            \"capabilities\": [\"haproxy_management\", \"ssl_deployment\", \"config_reload\", \"systemd_service\", \"keepalived_management\"],
+            \"network_interfaces\": $ni_json,
             \"haproxy_status\": \"$haproxy_status\",
             \"haproxy_version\": \"$haproxy_version\",
             \"cluster_id\": $CLUSTER_ID,
@@ -3041,9 +3293,213 @@ CONFIG_RESPONSE_EOF
     HAPROXY_BIN="${HAPROXY_BIN_PATH}"
     HAPROXY_CONFIG="${HAPROXY_CONFIG_PATH}"
     log "INFO" "DAEMON: Initialized HAProxy paths - bin: $HAPROXY_BIN, config: $HAPROXY_CONFIG"
-    
+
+    # Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence (DAEMON copy). Inert by
+    # default: a node with no APPLIED VIP gets not_configured and (lacking our marker)
+    # does nothing. Never clobbers a hand-managed keepalived. $AGENT_TOKEN is synced to
+    # the live token at the top of each loop iteration below.
+    fetch_and_deploy_keepalived_config() {
+        local marker="# Managed by HAProxy OpenManager"
+        local resp status http_code we_own="false" conf chk
+
+        # Timeouts so a hung management server can never stall the daemon loop.
+        resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+            "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        http_code="${resp##*$'\n'}"   # last line = HTTP status
+        resp="${resp%$'\n'*}"          # everything before = JSON body
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+        # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+        # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+        [[ "$http_code" == "404" ]] && status="teardown"
+        # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+        conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+        [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+        chk="$(dirname "$conf")/check_haproxy.sh"
+        if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+        # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
+        # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
+        # to the node. The heartbeat cannot carry this — it has the VIP address and a best-effort
+        # MASTER/BACKUP, while rendering a node's config needs eleven fields.
+        #
+        # Rate limited by content: the hash of the last report is cached next to the config, so the
+        # file (which may contain the VRRP password) is posted only when it actually changes, not on
+        # every cycle. Once we own the file there is nothing to adopt, so the record is cleared once.
+        _kp_discover() {
+            local cache="$(dirname "$conf")/.hom_discovery_hash" cur_hash="" body content_json
+            if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
+                # Nothing adoptable here. Clear a previous report exactly once.
+                [[ -f "$cache" ]] || return 0
+                curl -k -s --connect-timeout 10 --max-time 30 -X POST \
+                    "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                    -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                    -d "{\"config_path\":\"$conf\",\"exists\":false}" >/dev/null 2>&1 || return 0
+                rm -f "$cache"
+                return 0
+            fi
+            cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            [[ -z "$cur_hash" ]] && return 0
+            [[ -f "$cache" && "$(cat "$cache" 2>/dev/null)" == "$cur_hash" ]] && return 0
+            # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
+            # server hashes is byte-identical to what is on disk — the takeover authorisation is
+            # pinned to that hash.
+            content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
+            body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
+                '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
+            if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST \
+                "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                --data-binary "$body" 2>/dev/null; then
+                printf '%s' "$cur_hash" > "$cache" 2>/dev/null
+                log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+            fi
+        }
+        _kp_discover
+
+        _kp_report() {
+            local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+            curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+                >/dev/null 2>&1 || true
+        }
+        _kp_teardown() {
+            # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+            # keepalived and remove OUR config (VIP released) but KEEP the package. Purge only when
+            # opted in AND we installed it (.hom_installed marker) — never an admin's package.
+            local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+            log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+            systemctl stop keepalived >/dev/null 2>&1
+            systemctl disable keepalived >/dev/null 2>&1
+            rm -f "$conf" "$chk"
+            if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+                log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+                fi
+                rm -f "$marker_inst"
+                if command -v keepalived >/dev/null 2>&1; then
+                    _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+                else
+                    _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+                fi
+            elif [[ "$purge" == "true" ]]; then
+                log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+                _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+            else
+                _kp_report "disabled" "" "" "torn down"
+            fi
+        }
+
+        case "$status" in
+            available) : ;;
+            teardown)
+                local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+                [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"   # explicit delete: honor opt-in purge
+                return 0 ;;
+            not_configured)
+                [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only
+                return 0 ;;
+            *) return 0 ;;
+        esac
+
+        local vip_id new_conf check_script install_if new_hash
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+        new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+        check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+        install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+        [[ -z "$new_conf" ]] && return 0
+
+        if [[ -f "$conf" && "$we_own" != "true" ]]; then
+            local allow_takeover expected_hash disk_hash
+            allow_takeover=$(echo "$resp" | jq -r '.keepalived.allow_takeover // false' 2>/dev/null)
+            expected_hash=$(echo "$resp" | jq -r '.keepalived.takeover_expected_hash // empty' 2>/dev/null)
+            disk_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            if [[ "$allow_takeover" == "true" && -n "$expected_hash" && "$disk_hash" == "$expected_hash" ]]; then
+                log "INFO" "KEEPALIVED: adopting $conf (one-shot takeover authorised; on-disk hash matches)"
+            elif [[ "$allow_takeover" == "true" ]]; then
+                log "WARN" "KEEPALIVED: adoption authorised but $conf changed since it was adopted — refusing"
+                _kp_report "externally_managed" "$vip_id" "$disk_hash" \
+                    "config changed after adoption; re-adopt to pick up the current file"
+                return 0
+            else
+                log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+                _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+                return 0
+            fi
+        fi
+
+        # Hybrid install: install keepalived only if missing.
+
+        if ! command -v keepalived >/dev/null 2>&1; then
+            if [[ "$install_if" == "true" ]]; then
+                log "INFO" "KEEPALIVED: installing package..."
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+                fi
+            fi
+            if ! command -v keepalived >/dev/null 2>&1; then
+                log "ERROR" "KEEPALIVED: not installed/available"
+                _kp_report "error" "$vip_id" "" "keepalived not installed"
+                return 0
+            fi
+            # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+            # delete removes only OUR install, never an admin's pre-existing keepalived package.
+            mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+        fi
+
+        if [[ -f "$conf" ]]; then
+            local cur_hash would_hash
+            cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+            if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                return 0
+            fi
+        fi
+
+        mkdir -p "$(dirname "$conf")"
+        if [[ -n "$check_script" ]]; then
+            printf '%s' "$check_script" > "$chk"
+            chown root:root "$chk" 2>/dev/null
+            chmod 0755 "$chk"
+        fi
+        # Validate on a TEMP file and swap in only on success: a bad render must never land on
+        # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+        local tmp_conf="${conf}.hom.tmp"
+        printf '%s' "$new_conf" > "$tmp_conf"
+        chmod 0644 "$tmp_conf"
+        if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+            rm -f "$tmp_conf"
+            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+            return 0
+        fi
+        mv -f "$tmp_conf" "$conf"
+        chown root:root "$conf" 2>/dev/null
+        chmod 0644 "$conf"
+        systemctl enable keepalived >/dev/null 2>&1
+        if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+            log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+            _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+        else
+            log "ERROR" "KEEPALIVED: reload/restart failed"
+            _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+        fi
+        return 0
+    }
+
     log "DEBUG" "DAEMON: Starting agent monitoring loop for $AGENT_NAME"
-    
+
     # Enhanced daemon loop with upgrade capability - EXACT MacOS COPY
     while true; do
         sleep 30
@@ -3208,6 +3664,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP (Keepalived) convergence at the SSL cadence (~2.5 min);
+        # inert for non-VIP nodes (status=not_configured + no ownership marker).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/agent_scripts/macos_install.sh

@@ -318,7 +318,11 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "com.haproxy.agent"; do
+# issue #31: match ONLY the installed agent (binary path + LaunchDaemon label), never the bare
+# string "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command
+# line or a sudo ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself. The "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "com.haproxy.agent"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -522,7 +526,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -867,7 +871,7 @@ SSL_SYNC_TIMESTAMP_FILE="/tmp/haproxy-agent-ssl-sync-${AGENT_NAME}"
 get_cluster_paths() {
     log "DEBUG" "Fetching cluster paths from management API"
     local cluster_response=$("$CURL_BIN" -k -s -X GET "$MANAGEMENT_URL/api/clusters" \
-        -H "Authorization: Bearer $AGENT_TOKEN")
+        -H "X-API-Key: $AGENT_TOKEN")
     
     if [[ $? -eq 0 ]]; then
         local _cfg _bin _sock
@@ -920,7 +924,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1164,13 +1173,39 @@ get_keepalive_state() {
     return 0
 }
 
+# Issue #27 — keepalived/VRRP is Linux-only. macOS agents cannot run keepalived, so they
+# can never be VIP members (the backend also withholds the keepalived_management capability,
+# so the UI warns at assign time). This is a safe no-op that, if a VIP is somehow mis-assigned
+# to a macOS node, reports a clear "unsupported on macOS" status instead of silently never
+# converging. It never installs/writes anything.
+fetch_and_deploy_keepalived_config() {
+    local resp status vip_id
+    resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    if [[ "$status" == "available" ]]; then
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+        curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+    fi
+    return 0
+}
+
 # Send heartbeat
 send_heartbeat() {
     local haproxy_status=$(get_haproxy_status)
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -2197,8 +2232,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence (no-op on macOS; reports unsupported if assigned)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2629,7 +2668,25 @@ SYSTEM_INFO_EOF
         echo "|"
         return 0
     }
-    
+
+    # Issue #27 — keepalived/VRRP is Linux-only (DAEMON version). Safe no-op that reports a
+    # clear "unsupported on macOS" status if a VIP is ever mis-assigned to a macOS node.
+    fetch_and_deploy_keepalived_config() {
+        local resp status vip_id
+        resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        if [[ "$status" == "available" ]]; then
+            vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+            log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+            curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+        fi
+        return 0
+    }
+
     # Send heartbeat (DAEMON version - includes all stats)
     send_heartbeat() {
         local haproxy_status=$(get_haproxy_status)
@@ -3021,6 +3078,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP convergence at the SSL cadence (no-op on macOS; reports
+        # unsupported if a VIP is mis-assigned to this node).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/csr_key_crypto.py

@@ -0,0 +1,120 @@
+"""Issue #53 — at-rest encryption for the pending CSR private key (v1.10.1).
+
+Mirrors the established Fernet + HKDF(SECRET_KEY) pattern already used for the VRRP secret
+(services/keepalived_config.py), TOTP secrets (services/mfa_service.py) and DNS provider
+credentials (utils/dns_credentials.py): prefer an explicit CSR_ENCRYPTION_KEY env var (enables
+key rotation), else derive a stable key from SECRET_KEY via HKDF with its own versioned info
+string, so a rotation of one secret class never affects another.
+
+WHY this key and not every key in the system: the CSR private key is the one key that sits IDLE.
+It is generated at CSR creation, waits for an external CA to sign the request (days to weeks),
+and is destroyed the moment the signed certificate is imported — it is never transmitted to an
+agent and never leaves the server. `ssl_certificates.private_key_content` and the ACME order keys
+are different: agents must receive them in plaintext on every poll, so encrypting them at rest
+buys nothing without an end-to-end redesign.
+
+STORAGE: the Fernet token replaces the PEM in the SAME `ssl_csrs.private_key_pem` TEXT column.
+No new column, no new table, and deliberately NO `SCHEMA_VERSION` bump — a bump would re-run the
+migration sequence and re-seed the four built-in roles to their defaults (see UPGRADE_GUIDE.md),
+which is a needless side effect for a storage-format change.
+
+BACKWARD COMPATIBILITY: rows written before this release hold a raw PEM. `decrypt_csr_private_key`
+detects those by their `-----BEGIN` header and returns them unchanged. The discriminator is exact,
+not a heuristic: a Fernet token is base64url text and can never contain "-----". Legacy rows drain
+naturally, since a CSR's key copy is NULLed on import.
+
+KEY ROTATION: if SECRET_KEY rotates while CSR_ENCRYPTION_KEY is unset, previously stored keys
+become undecryptable and `decrypt_csr_private_key` returns None. Callers MUST surface a clear
+"delete this CSR and create a new one" error — the CSR is unusable at that point, because the
+signed certificate can no longer be paired with its key.
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+from typing import Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# A PEM private key always carries this header; a Fernet token is base64url and never can.
+_PEM_MARKER = "-----BEGIN"
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit CSR_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF with a
+    versioned info string (so stored keys survive restarts)."""
+    explicit = os.getenv("CSR_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("CSR_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    logger.warning(
+        "CSR_ENCRYPTION_KEY not set; deriving the CSR private-key encryption key from SECRET_KEY. "
+        "Set CSR_ENCRYPTION_KEY to a Fernet key to enable key rotation."
+    )
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"csr-private-key-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def is_encrypted(stored: Optional[str]) -> bool:
+    """True when the stored value is a Fernet token rather than a legacy raw PEM.
+
+    Single source of the format discriminator: `decrypt_csr_private_key` branches on this, so
+    the "what does a stored value look like" rule is stated exactly once.
+    """
+    return bool(stored) and _PEM_MARKER not in stored
+
+
+def encrypt_csr_private_key(pem: str) -> str:
+    """Fernet-encrypt a PEM private key to a storable token string."""
+    return _get_fernet().encrypt(pem.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_csr_private_key(stored: Optional[str]) -> Optional[str]:
+    """Return the PEM private key for a stored value.
+
+    Accepts BOTH shapes so an upgrade needs no data migration:
+      - a raw PEM written before v1.10.1 -> returned unchanged
+      - a Fernet token -> decrypted
+
+    Returns None when the value is empty or cannot be decrypted (e.g. SECRET_KEY rotated without
+    CSR_ENCRYPTION_KEY). Callers MUST treat None as "this CSR's key is unrecoverable" and tell the
+    operator to delete it and create a new one; never fall through to a pairing attempt.
+    """
+    if not stored:
+        return None
+    if not is_encrypted(stored):
+        return stored  # legacy plaintext row, pre-v1.10.1
+    try:
+        return _get_fernet().decrypt(stored.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt a stored CSR private key (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting a stored CSR private key: %s", exc)
+        return None

backend/utils/dns_credentials.py

@@ -0,0 +1,88 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): at-rest encryption for per-account DNS provider credentials.
+
+Mirrors the established Fernet + HKDF(SECRET_KEY) pattern used for the VRRP secret
+(backend/services/keepalived_config.py) and TOTP secrets (backend/services/mfa_service.py):
+prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY env var (enables key rotation), else derive a
+stable key from SECRET_KEY via HKDF with a versioned info string.
+
+DNS provider credentials are a small dict (e.g. {"api_token": "..."}). They are JSON-serialized,
+encrypted to a Fernet token string for storage, and only ever decrypted in-process when a DNS-01
+order needs to talk to the provider. Plaintext credentials are NEVER logged or returned by the API.
+
+NOTE on key rotation: if SECRET_KEY rotates and DNS_PROVIDER_ENCRYPTION_KEY is not set, previously
+stored credentials become undecryptable (decrypt returns None). Callers MUST treat a None result as
+"credentials unavailable — re-enter in Settings" and surface a clear error, never a silent hang.
+"""
+from __future__ import annotations
+
+import base64
+import json
+import logging
+import os
+from typing import Dict, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so credentials survive restarts)."""
+    explicit = os.getenv("DNS_PROVIDER_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("DNS_PROVIDER_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    logger.warning(
+        "DNS_PROVIDER_ENCRYPTION_KEY not set; deriving the DNS-credentials encryption key from "
+        "SECRET_KEY. Set DNS_PROVIDER_ENCRYPTION_KEY to a Fernet key to enable key rotation."
+    )
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"dns-provider-creds-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_dns_credentials(credentials: Dict[str, str]) -> str:
+    """JSON-serialize and Fernet-encrypt a credentials dict to a storable token string."""
+    payload = json.dumps(credentials, separators=(",", ":")).encode("utf-8")
+    return _get_fernet().encrypt(payload).decode("utf-8")
+
+
+def decrypt_dns_credentials(token: str) -> Optional[Dict[str, str]]:
+    """Decrypt a stored token back to the credentials dict. Returns None if the token can't be
+    decrypted (e.g. key rotated) — callers must surface a clear 're-enter credentials' error."""
+    try:
+        plain = _get_fernet().decrypt(token.encode("utf-8")).decode("utf-8")
+        data = json.loads(plain)
+        if not isinstance(data, dict):
+            logger.error("Decrypted DNS credentials are not a JSON object")
+            return None
+        return data
+    except InvalidToken:
+        logger.warning("Failed to decrypt DNS provider credentials (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting DNS provider credentials: %s", exc)
+        return None

backend/utils/entity_snapshot.py

@@ -329,9 +329,10 @@ async def _rollback_update(
                     tcp_request_rules = $26, timeout_client = $27, timeout_http_request = $28,
                     rate_limit = $29, compression = $30, log_separate = $31,
                     monitor_uri = $32, maxconn = $33,
-                    cluster_id = $34, is_active = $35, last_config_status = $36, 
+                    cluster_id = $34, is_active = $35, last_config_status = $36,
+                    log_format = $37, filters = $38,
                     updated_at = CURRENT_TIMESTAMP
-                WHERE id = $37
+                WHERE id = $39
             """,
                 old_values.get('name'),
                 old_values.get('bind_address'),
@@ -369,6 +370,8 @@ async def _rollback_update(
                 old_values.get('cluster_id'),
                 old_values.get('is_active'),
                 old_values.get('last_config_status'),
+                old_values.get('log_format'),  # Issue #38
+                old_values.get('filters'),     # Issue #38
                 entity_id
             )
             

backend/utils/haproxy_config_parser.py

@@ -105,6 +105,11 @@ class ParsedFrontend:
     response_headers: Optional[str] = None
     options: Optional[str] = None  # HAProxy frontend options (option httplog, option forwardfor, etc.)
     tcp_request_rules: Optional[str] = None  # TCP request directives (for TCP mode)
+    # Issue #38: SPOE (and other) filter directives + frontend log-format.
+    # Stored as full directive lines; `filters` is newline-joined to preserve
+    # ordering when multiple `filter ...` lines exist.
+    log_format: Optional[str] = None  # `log-format` / `log-format-sd` line(s)
+    filters: Optional[str] = None  # `filter ...` line(s), e.g. `filter spoe engine coraza config ...`
 
 
 @dataclass
@@ -256,8 +261,23 @@ class HAProxyConfigParser:
             acl_rules_list = []
             use_backend_rules_list = []
             tcp_request_rules_list = []
+            filters_list = []
+            log_format_list = []
 
             for line in lines:
+                # Issue #38: capture `filter ...` (SPOE/Coraza etc.) and
+                # `log-format`/`log-format-sd` directives. Pre-fix these matched
+                # no branch below and were silently dropped, so an imported SPOE
+                # config lost `filter spoe engine coraza ...` (→ HAProxy fatal
+                # "unable to find SPOE engine") and the frontend log-format.
+                # `continue` isolates them from the header/option handling below.
+                if line.startswith('filter '):
+                    filters_list.append(line.strip())
+                    continue
+                if re.match(r'^log-format(-sd)?\s', line, re.IGNORECASE):
+                    log_format_list.append(line.strip())
+                    continue
+
                 # Parse bind directive
                 # IMPORTANT: Handle multiple bind lines correctly
                 # Example: bind *:1002 (HTTP) and bind *:443 ssl (HTTPS)
@@ -540,6 +560,13 @@ class HAProxyConfigParser:
             if tcp_request_rules_list:
                 frontend.tcp_request_rules = '\n'.join(tcp_request_rules_list)
 
+            # Issue #38: assign captured SPOE filters + log-format
+            if filters_list:
+                frontend.filters = '\n'.join(filters_list)
+
+            if log_format_list:
+                frontend.log_format = '\n'.join(log_format_list)
+
             self.frontends.append(frontend)
             logger.info(f"Parsed frontend: {name} -> {frontend.default_backend}")
 
@@ -666,9 +693,14 @@ class HAProxyConfigParser:
                             'transparent', 'abortonclose', 'allbackups', 'checkcache', 'clitcpka',
                             'srvtcpka', 'http-no-delay', 'socket-stats', 'tcp-smart-accept',
                             'tcp-smart-connect', 'independant-streams', 'log-separate-errors',
-                            'log-health-checks', 'accept-invalid-http-request', 'accept-invalid-http-response'
+                            'log-health-checks', 'accept-invalid-http-request', 'accept-invalid-http-response',
+                            # Issue #38: SPOP health check for SPOE agent backends
+                            # (e.g. coraza-spoa). Already collected below regardless, but
+                            # listing it suppresses the spurious "unknown option" warning
+                            # for the exact SPOE use-case.
+                            'spop-check'
                         ]
-                        
+
                         if option_name not in valid_options:
                             # Unknown/invalid option - add warning but still collect it
                             self.warnings.append(