perf(api): opt-in uvicorn workers + heartbeat query consolidation (v1.8.6, Issue #35)

A user running the API on a 2-core/4GB host reported slow-feeling API
responses (Issue #35 follow-up). Review of the hot paths found no
pathological defect; the dominant factors are the single uvicorn worker
(one core serves all requests) and the constant agent-poll baseline
(4 requests per agent every 30s). Two zero-risk improvements:

- backend/Dockerfile: CMD now honors UVICORN_WORKERS, falling back to
  WEB_CONCURRENCY and then 1. Flagless uvicorn natively honors
  WEB_CONCURRENCY, so the fallback keeps any deployment that relied on it
  byte-for-byte compatible; with the final default of 1 worker uvicorn runs
  in-process exactly as before. >1 enables the multiprocess supervisor so
  multi-core hosts can use all cores. Background tasks are already
  multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as exercised
  by the k8s HPA deployment (2-10 replicas). `exec` keeps uvicorn as PID 1
  (clean SIGTERM, verified ~1s docker stop with 2 workers).
  docker-compose.yml passes UVICORN_WORKERS through as empty-when-unset so a
  user-set WEB_CONCURRENCY is never overridden; .env.template documents it.

- routers/agent.py heartbeat (by-name endpoint): the agent's
  status/version/upgrade_status were read with three separate single-column
  SELECTs against the same row; now one SELECT. Identical values and None
  semantics (single consistent snapshot instead of three reads); saves two
  round-trips per heartbeat per agent every 30s. The legacy by-id heartbeat
  endpoint is untouched; the heartbeat API contract is unchanged for agents
  of every version.

- README: new "Performance Tuning" section (worker/replica scaling, and how
  to use the X-Response-Time header plus "Slow request detected" logs to
  pinpoint slow endpoints).

Verification: full backend suite in docker green (1063 passed, 151 skipped;
also re-run by the runtime image build); worker-count expansion matrix
(unset->1, UVICORN_WORKERS=2->2, WEB_CONCURRENCY=3->3, both->UVICORN_WORKERS,
empty->fallback) all correct; default run confirmed single-process with
uvicorn as PID 1 and healthy API; UVICORN_WORKERS=2 confirmed parent + 2
workers, healthy API, clean shutdown; live heartbeats verified for
register + existing-agent paths AND degraded agents (no stats socket /
haproxy stopped / garbage stats CSV / unknown backend in server_statuses):
all return 200, agent row updates correctly, zero backend errors.
No schema, API, or agent changes.

.env.template

@@ -17,6 +17,11 @@ REDIS_URL=redis://redis:6379
 # Change this to a strong random string in production
 SECRET_KEY=your-secret-key-change-this-in-production
 
+# Optional: dedicated Fernet key for encrypting VRRP secrets of HA/VIP (Issue #27).
+# If unset, it is derived from SECRET_KEY (HKDF), exactly like MFA. Set an explicit
+# key (urlsafe-base64, 32 bytes) in production if you want independent key rotation.
+# VIP_ENCRYPTION_KEY=
+
 # ============================================================================
 # PUBLIC URL CONFIGURATION
 # ============================================================================
@@ -54,6 +59,15 @@ AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
 # Config sync interval in seconds
 AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 
+# ============================================================================
+# BACKEND PERFORMANCE
+# ============================================================================
+# Number of uvicorn worker processes for the backend API (default: 1).
+# On multi-core hosts, setting this to the core count (e.g. 2) lets the API
+# use all cores. Safe to increase: background tasks are multi-replica safe
+# (the k8s deployment already runs 2+ replicas via HPA).
+UVICORN_WORKERS=1
+
 # ============================================================================
 # CORS CONFIGURATION
 # ============================================================================

.github/workflows/docker-build.yml

@@ -7,6 +7,8 @@ on:
 jobs:
   build_and_push:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
 
     steps:
       - name: checkout
@@ -26,6 +28,20 @@ jobs:
           fi
           echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
 
+      # The backend image is built with `context: ./backend`, so the
+      # repo-root version.json is OUTSIDE the build context and never
+      # reaches the container. Backend `main.py` falls back to a
+      # compile-time constant when /app/version.json is missing, which
+      # caused a real production drift: a redeploy of the v1.5.2 tree
+      # silently still reported "v1.5.0" in `/api/version` because the
+      # constant in main.py had been bumped but the file was not
+      # available to read. Stage version.json into the backend
+      # context here so the canonical file IS shipped and the
+      # constant only serves as a defensive fallback. The staged file
+      # is gitignored to keep `git status` clean for developers.
+      - name: stage version.json into backend build context
+        run: cp version.json backend/version.json
+
       - name: set up qemu
         uses: docker/setup-qemu-action@v3
 
@@ -62,3 +78,30 @@ jobs:
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.version.outputs.TAG }}
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.prodversion.outputs.VERSION }}
 
+      # Keep the GitHub Releases/Tags in sync with version.json. The docker
+      # images above are tagged with the product version, but nothing here
+      # created the matching git tag, so the repo's Tags/Releases drifted
+      # behind (stuck at the last manually-created tag). After the images are
+      # pushed, cut a Release (which also creates the tag) for the current
+      # version.json, but only if one does not already exist, so re-runs
+      # without a version bump are a no-op.
+      - name: create github release from version.json
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          VERSION="${{ steps.prodversion.outputs.VERSION }}"
+          TAG="v${VERSION}"
+          RELEASE_NAME=$(jq -r '.releaseName // empty' version.json)
+          if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "Release $TAG already exists, skipping."
+          else
+            TITLE="$TAG"
+            [ -n "$RELEASE_NAME" ] && TITLE="$TAG — $RELEASE_NAME"
+            gh release create "$TAG" \
+              --repo "$GITHUB_REPOSITORY" \
+              --target "$GITHUB_SHA" \
+              --title "$TITLE" \
+              --notes "Automated release for $TAG (from version.json)."
+            echo "Created release $TAG"
+          fi
+

.gitignore

@@ -39,8 +39,10 @@ venv.bak/
 *.sqlite
 *.sqlite3
 
-# Docker
+# Build-time staged version.json (CI `cp version.json backend/`).
-.dockerignore
+# The canonical file lives at repo root; this path is a transient
+# copy for the backend Docker build context.
+backend/version.json
 
 # IDE
 .vscode/

LICENSE

@@ -1,22 +1,679 @@
-MIT License
+HAProxy OpenManager
+Copyright (C) 2025-2026  Taylan Bakırcıoğlu and HAProxy OpenManager Contributors
 
-Copyright (c) 2025 HAProxy OpenManager Contributors
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
+This program is distributed in the hope that it will be useful,
-of this software and associated documentation files (the "Software"), to deal
+but WITHOUT ANY WARRANTY; without even the implied warranty of
-in the Software without restriction, including without limitation the rights
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+GNU Affero General Public License for more details.
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
+You should have received a copy of the GNU Affero General Public License
-copies or substantial portions of the Software.
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+---
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
 
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.

UPGRADE_GUIDE.md

@@ -1,3 +1,31 @@
+# Upgrade Notes — v1.7.0 (HA / VIP Keepalived management, Issue #27)
+
+**Backward compatible & opt-in.** Upgrading to v1.7.0 changes nothing for existing
+clusters/agents until you create a VIP:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `3`, so on first start the (idempotent)
+  migration sequence re-runs once and adds two **new** tables (`vip_instances`,
+  `vip_members`) plus an additive `vip_instances.applied_snapshot` column (enables
+  rejecting a pending VIP change and restoring the previous applied state). No existing
+  table is altered. Existing rows and the **admin password
+  are not reset** (default users are create-if-missing). The only data effect is that
+  the **four built-in system roles** (`super_admin`/`operator`/`security_admin`/`viewer`)
+  are re-seeded to their canonical permission sets **plus** the new `vip.*` permissions —
+  this is the long-standing behavior of the role seeder; **custom roles are untouched**.
+- **Agents:** the agent script gains an opt-in keepalived deploy that is a **no-op** on
+  any node without an applied VIP, and it **never overwrites a hand-managed
+  `/etc/keepalived/keepalived.conf`** (it reports "externally managed" instead).
+- **Scope:** VRRP VIPs target bare-metal / VMware / on-prem L2 networks. On AWS/Azure/GCP
+  the cloud fabric doesn't honor VRRP/gratuitous-ARP; the UI surfaces this. Ensure host
+  firewalls permit VRRP (IP protocol 112).
+- **Optional env:** `VIP_ENCRYPTION_KEY` (see `.env.template`) — if unset, the VRRP secret
+  encryption key is derived from `SECRET_KEY` (like MFA).
+
+No rollback steps are required to *disable* the feature: simply don't create VIPs (or
+delete them — agents tear down their managed keepalived on the next poll).
+
+---
+
 # Agent Upgrade Guide - Dashboard Stats Fix
 
 ## Problem

backend/Dockerfile

@@ -37,5 +37,13 @@ USER appuser
 # Expose port
 EXPOSE 8000
 
-# Run the application in production mode (without --reload)
+# Run the application in production mode (without --reload).
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+# UVICORN_WORKERS (default 1) opts into multiple worker processes on multi-core
+# hosts; with 1 worker uvicorn runs in-process, identical to the flagless CMD
+# this replaces. Falls back to WEB_CONCURRENCY when UVICORN_WORKERS is unset
+# because flagless uvicorn honored WEB_CONCURRENCY (uvicorn config.py) — this
+# keeps any deployment that relied on it byte-for-byte compatible. Background
+# tasks are multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as
+# already exercised by the k8s HPA deployment. `exec` keeps uvicorn as PID 1
+# so signal handling is unchanged.
+CMD ["sh", "-c", "exec uvicorn main:app --host 0.0.0.0 --port 8000 --workers ${UVICORN_WORKERS:-${WEB_CONCURRENCY:-1}}"]

backend/database/migrations.py

@@ -50,9 +50,62 @@ async def ensure_agents_table():
         await conn.execute("ALTER TYPE config_status ADD VALUE IF NOT EXISTS 'DELETION';")
         logger.info("Ensured REJECTED and DELETION values exist in config_status enum.")
 
-        # First, create essential tables if they don't exist
+        # First, create essential tables if they don't exist.
-        await create_essential_tables(conn)
+        #
-        
+        # Rolling-restart resilience: create_essential_tables() runs idempotent
+        # CREATE ... IF NOT EXISTS statements on every startup. Its CREATE INDEX
+        # block needs a SHARE lock that conflicts with concurrent writes (e.g.
+        # agent heartbeats updating backend_servers/agents). During a redeploy a
+        # writer can hold that lock, so the DDL blocked for the full 60s
+        # command_timeout -> TimeoutError -> startup crash -> crash-loop.
+        #
+        # Fix: fail fast on locks (lock_timeout), retry briefly, and on
+        # persistent contention SKIP the idempotent bootstrap and continue — on
+        # an established DB the objects already exist; a fresh DB has no writers
+        # so the first attempt always succeeds. lock_timeout is scoped to this
+        # call and RESET afterwards, so every other migration below keeps its
+        # original (wait-indefinitely) behavior. Non-lock errors still propagate
+        # (genuine schema problems must NOT be masked).
+        import asyncio as _asyncio
+        _lock_excs = (_asyncio.TimeoutError,)
+        try:
+            import asyncpg as _asyncpg
+            _lock_excs = _lock_excs + (
+                _asyncpg.exceptions.LockNotAvailableError,
+                _asyncpg.exceptions.QueryCanceledError,
+            )
+        except Exception:
+            pass
+        try:
+            await conn.execute("SET lock_timeout = '10s'")
+        except Exception:
+            pass
+        try:
+            for _attempt in range(1, 4):
+                try:
+                    await create_essential_tables(conn)
+                    break
+                except _lock_excs as _lock_err:
+                    if _attempt < 3:
+                        logger.warning(
+                            f"create_essential_tables: lock contention "
+                            f"(attempt {_attempt}/3), retrying in 3s "
+                            f"({type(_lock_err).__name__})"
+                        )
+                        await _asyncio.sleep(3)
+                    else:
+                        logger.warning(
+                            "create_essential_tables: persistent lock contention; "
+                            "skipping idempotent schema bootstrap and continuing "
+                            "startup (objects already exist on an established DB) "
+                            f"({type(_lock_err).__name__})"
+                        )
+        finally:
+            try:
+                await conn.execute("RESET lock_timeout")
+            except Exception:
+                pass
+
         # Ensure status column exists in config_versions table
         status_column_exists = await conn.fetchval("""
             SELECT 1 FROM information_schema.columns 
@@ -431,6 +484,7 @@ async def ensure_agents_table():
                 connection_type VARCHAR(50) DEFAULT 'agent',
                 stats_socket_path VARCHAR(500) DEFAULT '/run/haproxy/admin.sock',
                 haproxy_config_path VARCHAR(500) DEFAULT '/etc/haproxy/haproxy.cfg',
+                keepalived_config_path VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf',
                 pool_id INTEGER REFERENCES haproxy_cluster_pools(id) ON DELETE SET NULL,
                 haproxy_user VARCHAR(255) DEFAULT 'haproxy',
                 haproxy_group VARCHAR(255) DEFAULT 'haproxy',
@@ -1298,6 +1352,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk', 'apply.emergency',
                     'agents.read', 'agents.create', 'agents.update', 'agents.delete', 'agents.script', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.create', 'clusters.update', 'clusters.delete', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.upload', 'config.backup', 'config.restore', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'users.read', 'users.create', 'users.update', 'users.delete', 'users.password', 'users.roles',
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
@@ -1319,6 +1374,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk',
                     'agents.read', 'agents.update', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read'
@@ -1336,6 +1392,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history',
                     'agents.read', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
@@ -1354,6 +1411,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.history',
                     'agents.read',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read',
@@ -1639,13 +1697,126 @@ async def ensure_agent_activity_logs_table():
             await close_database_connection(conn)
         # Don't raise - this is not critical for system operation
 
+# Schema-version gate for the migration runner.
+#
+# >>> BUMP THIS whenever you add/modify ANY step in _run_all_migrations_inner()
+# >>> that changes the schema (table/column/index/constraint) OR seeded/role data
+# >>> (e.g. update_system_roles_to_enterprise_rbac). Otherwise the new step will
+# >>> NOT run on databases already marked at the current version.
+#
+# When the DB already records >= this version, run_all_migrations() skips the
+# whole (lock-heavy) idempotent sequence, so redeploys/scale-ups issue NO DDL and
+# a concurrently-serving replica's traffic cannot block ALTER / CREATE INDEX (the
+# rolling-deploy startup crash that motivated this gate).
+#
+# Backward compatibility (the product runs at many versions across companies):
+#  - First start on this code: no marker -> applied_version is NULL -> the FULL
+#    sequence runs (upgrades any prior version), THEN the marker is written. So
+#    upgrading from any older version is unaffected.
+#  - The marker is written ONLY after _run_all_migrations_inner() completes with
+#    no exception, so an interrupted/failed migration never marks an incomplete
+#    schema as done — the next start retries.
+#  - Behavior change vs the historical "re-run every idempotent ensure_* on every
+#    start": once marked, same-version restarts no longer re-run (and therefore no
+#    longer auto-repair manual drift). To force a re-run, bump SCHEMA_VERSION or
+#    delete the schema_migrations row.
+#
+# v1.7.0 (Issue #27 — HA/VIP Keepalived management): bumped 1 -> 2 so the new
+# additive ensure_vip_tables() step (two brand-new tables) actually runs on
+# databases already marked at version 1. The whole re-run is idempotent.
+# v1.7.0 self-review: bumped 2 -> 3 so the additive `applied_snapshot` column on
+# vip_instances (enables VIP reject/restore-to-previous) lands on DBs marked at 2.
+# v1.7.0 self-review: bumped 3 -> 4 for the additive `keepalived_config_path` column on
+# haproxy_clusters (cluster-driven keepalived.conf path, like haproxy_config_path).
+# v1.7.0 self-review: bumped 4 -> 5 to drop the table-level UNIQUE on vip_instances.name
+# and replace it with a partial unique index (active rows only), so a soft-deleted VIP's
+# name is reusable — consistent with the address/VRID partial indexes. Idempotent re-run.
+# v1.7.2: bumped 5 -> 6 for the additive `purge_on_teardown` column on vip_instances
+# (opt-in "also uninstall the keepalived package on delete"; default FALSE keeps the safe
+# graceful-teardown behaviour). Additive + idempotent.
+# v1.7.2: bumped 6 -> 7 for the additive `pending_delete` column on vip_instances
+# (approval-gated VIP deletion: a delete is staged for Apply Management and the VIP keeps
+# running until APPROVED, so an agent never tears down without explicit human approval).
+# v1.8.0 (Issue #35 — ACME DNS-01 challenge support): bumped 7 -> 8 for additive DNS-01
+# columns on letsencrypt_accounts/letsencrypt_orders/acme_challenges and the brand-new
+# letsencrypt_account_dns_credentials table (ensure_letsencrypt_dns_credentials step).
+# All additive + idempotent; default challenge_type 'http-01' keeps existing flows byte-identical.
+SCHEMA_VERSION = 8
+
+
 async def run_all_migrations():
-    """Run all database migrations"""
+    """Run all database migrations.
+
+    Hardened for multiple backend replicas / rolling deploys:
+    - A session-level advisory lock serializes the run so only one pod migrates
+      at a time (others wait, then hit the version gate and skip). It is
+      session-scoped, so it auto-releases if a pod dies mid-migration.
+    - A schema-version marker (schema_migrations) gates the run: when the DB is
+      already at SCHEMA_VERSION the whole idempotent sequence is skipped, so no
+      DDL is issued and a serving replica's traffic can't block it.
+    If the advisory lock or marker can't be used, we fall back to running the
+    (idempotent) migrations rather than crashing startup.
+    """
     logger.info("Starting database migrations...")
-    
+    MIGRATION_ADVISORY_LOCK_KEY = 1836016242  # single-key advisory space ("migr"); distinct from the (ns,id) locks used elsewhere
+    lock_conn = None
+    lock_acquired = False
+    try:
+        lock_conn = await get_database_connection()
+        try:
+            await lock_conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            lock_acquired = True
+            logger.info("Acquired migration advisory lock (migrations serialized across pods)")
+        except Exception as _lock_e:
+            logger.warning(f"Could not acquire migration advisory lock; proceeding (migrations are idempotent): {_lock_e}")
+
+        # Schema-version gate: skip the lock-heavy sequence if the DB is current.
+        applied_version = None
+        try:
+            await lock_conn.execute("""
+                CREATE TABLE IF NOT EXISTS schema_migrations (
+                    id INTEGER PRIMARY KEY DEFAULT 1,
+                    version INTEGER NOT NULL,
+                    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                    CONSTRAINT schema_migrations_singleton CHECK (id = 1)
+                )
+            """)
+            applied_version = await lock_conn.fetchval("SELECT version FROM schema_migrations WHERE id = 1")
+        except Exception as _mk_e:
+            logger.warning(f"schema_migrations marker unavailable; running full migrations: {_mk_e}")
+            applied_version = None
+
+        if applied_version is not None and applied_version >= SCHEMA_VERSION:
+            logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
+            return
+
+        await _run_all_migrations_inner()
+
+        try:
+            await lock_conn.execute("""
+                INSERT INTO schema_migrations (id, version, applied_at)
+                VALUES (1, $1, CURRENT_TIMESTAMP)
+                ON CONFLICT (id) DO UPDATE SET version = EXCLUDED.version, applied_at = EXCLUDED.applied_at
+            """, SCHEMA_VERSION)
+            logger.info(f"Recorded schema version {SCHEMA_VERSION} in schema_migrations.")
+        except Exception as _wr_e:
+            logger.warning(f"Could not record schema version marker (migrations still applied): {_wr_e}")
+    finally:
+        if lock_acquired and lock_conn is not None:
+            try:
+                await lock_conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            except Exception:
+                pass
+        if lock_conn is not None:
+            await close_database_connection(lock_conn)
+
+
+async def _run_all_migrations_inner():
+    """The full idempotent migration sequence. Runs under the migration advisory
+    lock and is gated by the schema-version marker in run_all_migrations()."""
     # First, ensure basic database schema exists
     await run_init_sql()
-    
+
     # Then run additional migrations
     await ensure_agents_table()
     await ensure_config_versions_metadata_column()
@@ -1684,6 +1855,9 @@ async def run_all_migrations():
     await ensure_system_settings_table()
     await ensure_acme_tables()
     await ensure_acme_columns_on_existing_tables()
+    # Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+    # MUST run after ensure_acme_tables() (FK references letsencrypt_accounts).
+    await ensure_letsencrypt_dns_credentials()
     # Issue #11 cleanup: must run AFTER acme_tables/columns to ensure FK refs exist
     await cleanup_orphan_acme_challenge_backend()
     # v1.5.0 Feature A (ACME diagnostics) + Feature B (site wizard)
@@ -1700,8 +1874,212 @@ async def run_all_migrations():
     # (cluster_id, bind_address, bind_port) WHERE is_active.
     await ensure_frontends_bind_unique_constraint()
 
+    # Issue #18 — TOTP MFA (v1.6.0): additive columns + 3 new tables
+    await ensure_mfa_columns()
+
+    # Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
+    # MUST stay last: FK-references haproxy_cluster_pools/agents/users, all created above.
+    await ensure_vip_tables()
+
     logger.info("Database migrations completed successfully.")
 
+
+async def ensure_mfa_columns():
+    """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
+
+    All operations are idempotent (ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS).
+    Default behavior preserved: every existing user gets mfa_enabled=FALSE, so login
+    flow is byte-identical for accounts that don't opt in.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            ALTER TABLE users
+                ADD COLUMN IF NOT EXISTS mfa_enabled BOOLEAN DEFAULT FALSE NOT NULL,
+                ADD COLUMN IF NOT EXISTS mfa_method VARCHAR(20),
+                ADD COLUMN IF NOT EXISTS mfa_secret_encrypted TEXT,
+                ADD COLUMN IF NOT EXISTS mfa_enrolled_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_totp_step BIGINT;
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_backup_codes (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                code_hash VARCHAR(255) NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute("""
+            CREATE INDEX IF NOT EXISTS idx_mfa_backup_codes_user
+              ON mfa_backup_codes(user_id);
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_logins (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                challenge_token VARCHAR(64) UNIQUE NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                ip_address INET
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_token ON mfa_pending_logins(challenge_token);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_expires ON mfa_pending_logins(expires_at);"
+        )
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_enrollments (
+                user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+                secret_encrypted TEXT NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_enroll_expires ON mfa_pending_enrollments(expires_at);"
+        )
+
+        logger.info("✅ MFA migration completed (Issue #18 — Phase 1)")
+    except Exception as e:
+        logger.error(f"Failed to ensure MFA columns: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_user_activity_logs_table
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+async def ensure_vip_tables():
+    """Issue #27 — HA/VIP (Keepalived) management (v1.7.0). Additive only:
+    two brand-new tables (vip_instances, vip_members) + indexes. No ALTER of any
+    existing table, so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (haproxy_cluster_pools, agents,
+    users) are created earlier in the sequence — this function is registered LAST.
+
+    Backward-compat: a cluster/agent with no VIP row is unaffected; the agent
+    delivery endpoint returns 'not_configured' for every node without a membership.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Cluster-driven keepalived.conf path (mirrors haproxy_config_path): additive +
+        # idempotent, with a universal default so operators need set nothing. The agent
+        # pulls this from its cluster, exactly like the HAProxy paths.
+        await conn.execute(
+            "ALTER TABLE haproxy_clusters ADD COLUMN IF NOT EXISTS keepalived_config_path "
+            "VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf';")
+
+        # VIP instance: one row per virtual IP (one VRRP group), anchored to a pool.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_instances (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(255) NOT NULL,
+                description TEXT,
+                pool_id INTEGER NOT NULL REFERENCES haproxy_cluster_pools(id) ON DELETE CASCADE,
+                virtual_ip VARCHAR(45) NOT NULL,
+                prefix_length INTEGER NOT NULL DEFAULT 24,
+                virtual_router_id INTEGER NOT NULL,
+                advert_int INTEGER NOT NULL DEFAULT 1,
+                auth_pass_encrypted TEXT,
+                use_unicast BOOLEAN NOT NULL DEFAULT TRUE,
+                track_haproxy BOOLEAN NOT NULL DEFAULT TRUE,
+                is_active BOOLEAN NOT NULL DEFAULT TRUE,
+                last_config_status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
+                applied_snapshot JSONB,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_vrid_range CHECK (virtual_router_id BETWEEN 1 AND 255)
+            );
+        """)
+        # Additive (idempotent) for DBs that created vip_instances before applied_snapshot
+        # existed (v1.7.0 self-review): holds the field-level state as of the last Apply so
+        # a pending edit can be rejected and fully reverted to the previous applied state.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS applied_snapshot JSONB;")
+        # Opt-in package removal (v1.7.2): when an operator deletes a VIP and explicitly ticks
+        # "also uninstall keepalived from the node(s)", we set this flag so the teardown
+        # delivery tells the agent to purge the OS package. Default FALSE = the safe enterprise
+        # default (stop+disable+remove our config, but KEEP the package). Additive + idempotent;
+        # a node we never managed stays untouched regardless.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Approval-gated deletion (v1.7.2): deleting a RUNNING VIP from the UI does NOT take
+        # effect immediately — it sets pending_delete=TRUE and stages a vip-*-delete version
+        # for Apply Management. The VIP stays is_active=TRUE (agents keep serving it, NOTHING
+        # is torn down) until the operator APPROVES; only then does apply flip is_active=FALSE
+        # and the agents tear down. Reject clears the flag and the VIP keeps running untouched.
+        # This guarantees an agent never tears a VIP down without an explicit human approval —
+        # protecting production. Additive + idempotent.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS pending_delete BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Uniqueness as PARTIAL indexes on active rows so a soft-deleted VIP frees its
+        # name/address/VRID for immediate reuse (a table-level UNIQUE would keep blocking it).
+        # NAME (v1.7.0 self-review): the original CREATE used a table-level UNIQUE on name,
+        # which left a soft-deleted VIP's name blocked (you couldn't re-create a VIP with the
+        # same name) — inconsistent with addr/VRID. Drop that constraint and use a partial
+        # index instead. Idempotent: no-op on a fresh table (no inline UNIQUE) and on re-run.
+        await conn.execute("ALTER TABLE vip_instances DROP CONSTRAINT IF EXISTS vip_instances_name_key;")
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_name_active ON vip_instances(name) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_addr_active ON vip_instances(virtual_ip) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_vrid_active ON vip_instances(pool_id, virtual_router_id) WHERE is_active=TRUE;"
+        )
+
+        # Per-node membership: which agents participate + their VRRP role/priority,
+        # the applied (delivered) config snapshot, and the agent's deploy ack.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_members (
+                id SERIAL PRIMARY KEY,
+                vip_id INTEGER NOT NULL REFERENCES vip_instances(id) ON DELETE CASCADE,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                network_interface VARCHAR(64) NOT NULL,
+                role VARCHAR(10) NOT NULL DEFAULT 'BACKUP',
+                priority INTEGER NOT NULL DEFAULT 100,
+                applied_config_content TEXT,
+                applied_config_hash VARCHAR(64),
+                last_deploy_state VARCHAR(24),
+                last_deploy_message TEXT,
+                last_deploy_hash VARCHAR(64),
+                last_deploy_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_member_role CHECK (role IN ('MASTER','BACKUP')),
+                CONSTRAINT vip_member_priority_range CHECK (priority BETWEEN 1 AND 254),
+                CONSTRAINT vip_member_unique UNIQUE (vip_id, agent_id)
+            );
+        """)
+        # Last line of defense against split-brain: at most one MASTER per VIP.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_one_master ON vip_members(vip_id) WHERE role='MASTER';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_vip ON vip_members(vip_id);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_agent ON vip_members(agent_id);"
+        )
+
+        logger.info("✅ VIP tables ensured (Issue #27 — HA/VIP Keepalived management)")
+    except Exception as e:
+        logger.error(f"Failed to ensure VIP tables: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_mfa_columns
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 async def add_ssl_certificate_id_to_backend_servers():
     """Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
     conn = None
@@ -3278,6 +3656,23 @@ async def ensure_acme_columns_on_existing_tables():
             ('last_attempt_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS last_attempt_at TIMESTAMPTZ"),
             # Commit 3a: track auto-completion task lock/poll timestamps for atomic claim across replicas
             ('orders_updated_at_idx', "CREATE INDEX IF NOT EXISTS idx_letsencrypt_orders_status_updated ON letsencrypt_orders(status, updated_at) WHERE status = 'valid' AND ssl_certificate_id IS NULL"),
+            # Issue #35 (v1.8.0 — ACME DNS-01): per-account challenge method + DNS provider selection
+            ('acct_challenge_type', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('acct_dns_provider', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS dns_provider VARCHAR(50)"),
+            # per-order challenge method + bounded DNS-01 retry chain (dns01_parent_order_id is a PLAIN INTEGER, not a FK,
+            # to avoid a self-referential cascade interacting with account/order bulk DELETEs)
+            ('order_challenge_type', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('order_dns01_attempts', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_attempts INTEGER DEFAULT 0"),
+            ('order_dns01_last_attempt_at', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_last_attempt_at TIMESTAMPTZ"),
+            ('order_dns01_parent_order_id', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_parent_order_id INTEGER"),
+            ('order_dns01_retry_claimed', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_retry_claimed BOOLEAN DEFAULT FALSE"),
+            # per-challenge DNS-01 lifecycle state
+            ('chal_challenge_type', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('chal_dns_txt_value', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_txt_value TEXT"),
+            ('chal_dns_record_published', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_published BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_record_cleaned', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_cleaned BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_published_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_published_at TIMESTAMPTZ"),
+            ('chal_manual_confirm_deadline', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS manual_confirm_deadline TIMESTAMPTZ"),
         ]:
             try:
                 await conn.execute(sql)
@@ -3293,6 +3688,34 @@ async def ensure_acme_columns_on_existing_tables():
         logger.error(f"Error adding ACME columns: {e}")
 
 
+async def ensure_letsencrypt_dns_credentials():
+    """Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+
+    Idempotent (CREATE TABLE IF NOT EXISTS). FK to letsencrypt_accounts (created earlier by
+    ensure_acme_tables). Credentials are Fernet-encrypted at rest (backend/utils/dns_credentials.py);
+    only the provider name + timestamps are ever surfaced to the API.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS letsencrypt_account_dns_credentials (
+                id SERIAL PRIMARY KEY,
+                account_id INTEGER NOT NULL UNIQUE REFERENCES letsencrypt_accounts(id) ON DELETE CASCADE,
+                dns_provider VARCHAR(50) NOT NULL,
+                credentials_encrypted TEXT NOT NULL,
+                created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
+            )
+        """)
+        logger.info("Ensured letsencrypt_account_dns_credentials table")
+        await close_database_connection(conn)
+    except Exception as e:
+        if conn:
+            await close_database_connection(conn)
+        logger.error(f"Error ensuring letsencrypt_account_dns_credentials: {e}")
+
+
 async def cleanup_orphan_acme_challenge_backend():
     """
     Issue #11: One-time cleanup of orphan `_acme_challenge_backend` rows that may

backend/main.py

@@ -8,7 +8,8 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.5.0", "releaseName": "ACME Diagnostics & Site Wizard", "releaseDate": "2026-05-08"}
+# Build/deploy marker for the v1.8.x (Issue #35, DNS-01) rollout — ensures the pipeline ships this commit's image.
+_version_info = {"version": "1.8.4", "releaseName": "Agent installer self-kill fix", "releaseDate": "2026-06-27"}
 for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
     try:
         with open(_vpath) as _vf:
@@ -40,6 +41,8 @@ from routers.settings import router as settings_router
 from routers.letsencrypt import router as letsencrypt_router
 from routers.acme_diagnostics import router as acme_diagnostics_router
 from routers.site_wizard import router as site_wizard_router
+from routers.mfa import router as mfa_router
+from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -302,6 +305,21 @@ async def complete_pending_acme_orders():
                         WHERE (
                             status IN ('pending', 'processing', 'ready')
                             OR (status = 'valid' AND ssl_certificate_id IS NULL)
+                            -- Issue #35: bounded DNS-01 retry. ONLY dns-01 invalids with remaining
+                            -- budget + elapsed backoff are claimed; http-01 invalids are NEVER matched
+                            -- (their existing skip-and-log is preserved).
+                            OR (
+                                status = 'invalid' AND challenge_type = 'dns-01'
+                                AND ssl_certificate_id IS NULL
+                                AND COALESCE(dns01_retry_claimed, FALSE) = FALSE
+                                AND COALESCE(dns01_attempts, 0) < 3
+                                AND (
+                                    dns01_last_attempt_at IS NULL
+                                    OR dns01_last_attempt_at < NOW() - (
+                                        (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
+                                        || ' minutes')::INTERVAL
+                                )
+                            )
                         )
                         AND created_at > NOW() - INTERVAL '7 days'
                         AND (updated_at IS NULL OR updated_at < NOW() - INTERVAL '30 seconds')
@@ -335,9 +353,17 @@ async def complete_pending_acme_orders():
                 continue
 
             logger.info(f"[ACME-COMPLETE] Claimed {len(claimed_ids)} order(s) for completion: {claimed_ids}")
-            
+
+            from services.dns01_orchestrator import (
+                advance_dns01_order, retry_invalid_dns01, reconcile_dns01_cleanup,
+            )
+
             for oid in claimed_ids:
                 try:
+                    # Issue #35: advance the DNS-01 publish->confirm->respond state machine for
+                    # pending dns-01 orders (no-op for http-01 or non-pending orders).
+                    await advance_dns01_order(oid)
+
                     status_info = await acme_svc.check_order_status(oid)
                     current_status = status_info.get('status')
 
@@ -350,12 +376,21 @@ async def complete_pending_acme_orders():
                         result = await _complete_certificate(oid)
                         logger.info(f"[ACME-COMPLETE] Order {oid} completed - {result.get('message', '')}")
                     elif current_status == 'invalid':
-                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid, skipping")
+                        # Issue #35: bounded DNS-01 fresh-order retry (no-op for http-01).
+                        await retry_invalid_dns01(oid)
+                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid")
                     elif current_status in ('pending', 'processing'):
                         logger.info(f"[ACME-COMPLETE] Order {oid} still {current_status}, will retry next cycle")
                 except Exception as poll_err:
                     logger.error(f"[ACME-COMPLETE] Failed to complete order {oid}: {poll_err}")
 
+            # Issue #35: best-effort cleanup of TXT records left published on terminal orders
+            # (covers a failed cleanup or the kill-switch being flipped off). NOT gated by the switch.
+            try:
+                await reconcile_dns01_cleanup()
+            except Exception as rec_err:
+                logger.debug(f"[ACME-COMPLETE] DNS-01 reconcile skipped: {rec_err}")
+
             # NOTE: v1.5.0 wizard-staged processing now runs BEFORE the
             # claimed_ids early-continue above (Bulgu #2 fix), so it executes
             # every cycle regardless of pending/processing volume.
@@ -671,7 +706,9 @@ async def check_letsencrypt_renewals():
                     skip = False
                     try:
                         order = await conn2.fetchrow(
-                            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1",
+                            "SELECT o.account_id, o.domains, o.cluster_ids, o.challenge_type, a.dns_provider "
+                            "FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id "
+                            "WHERE o.id = $1",
                             order_id
                         )
                         if order:
@@ -687,15 +724,37 @@ async def check_letsencrypt_renewals():
                             if existing:
                                 logger.info(f"[ACME-RENEWAL] Skipping cert {cert['id']} - order {existing['id']} already in progress")
                                 skip = True
+                            elif (order['challenge_type'] == 'dns-01'):
+                                # Issue #35: manual DNS-01 cannot auto-renew unattended; and for an
+                                # automated provider, don't re-mint hourly if a recent retry chain already
+                                # exhausted its budget (avoids tripping the CA new-order rate limit).
+                                if (order['dns_provider'] or 'manual') == 'manual':
+                                    logger.warning(f"[ACME-RENEWAL] cert {cert['id']} uses manual DNS-01; cannot auto-renew unattended (publish the TXT and renew manually)")
+                                    skip = True
+                                else:
+                                    exhausted = await conn2.fetchrow("""
+                                        SELECT id FROM letsencrypt_orders
+                                        WHERE domains::text = $1::text AND challenge_type = 'dns-01'
+                                        AND status = 'invalid' AND COALESCE(dns01_attempts, 0) >= 3
+                                        AND created_at > NOW() - INTERVAL '24 hours'
+                                        LIMIT 1
+                                    """, json.dumps(domains))
+                                    if exhausted:
+                                        logger.warning(f"[ACME-RENEWAL] cert {cert['id']} DNS-01 renewal recently failed (check DNS); skipping re-mint for 24h")
+                                        skip = True
                     finally:
                         await close_database_connection(conn2)
 
                     if not order or skip:
                         continue
 
-                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids)
+                    challenge_type = order['challenge_type'] or 'http-01'
-                    await acme_svc.respond_to_challenges(new_order['order_id'])
+                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids, challenge_type=challenge_type)
-                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} for cert {cert['id']} ({cert['name']})")
+                    # http-01 responds immediately (token served continuously); dns-01 is driven by the
+                    # orchestrator AFTER the TXT is published (never respond before publish).
+                    if challenge_type != 'dns-01':
+                        await acme_svc.respond_to_challenges(new_order['order_id'])
+                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} ({challenge_type}) for cert {cert['id']} ({cert['name']})")
                 except Exception as cert_err:
                     logger.error(f"[ACME-RENEWAL] Failed to initiate renewal for cert {cert['id']}: {cert_err}")
 
@@ -820,6 +879,7 @@ app.include_router(config_router)  # Configuration management
 app.include_router(maintenance_router, prefix="/api", tags=["maintenance"])  # Database cleanup & maintenance
 app.include_router(auth_router)
 app.include_router(user_router)
+app.include_router(mfa_router)
 app.include_router(frontend_router)
 app.include_router(backend_router)
 app.include_router(cluster_router)
@@ -834,6 +894,7 @@ app.include_router(settings_router)
 app.include_router(letsencrypt_router)
 app.include_router(acme_diagnostics_router)  # v1.5.0 Issue #13: ACME Diagnostic Panel
 app.include_router(site_wizard_router)  # v1.5.0 Issue #14: New Site Setup Wizard
+app.include_router(vip_router)  # v1.7.0 Issue #27: HA/VIP (Keepalived) management
 
 
 # Legacy URL alias: /api/proxied-hosts/* → 308 redirect to /api/sites/*.
@@ -1051,7 +1112,7 @@ async def serve_acme_challenge(token: str):
     try:
         conn = await get_database_connection()
         row = await conn.fetchrow(
-            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) LIMIT 1",
+            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) AND (challenge_type = 'http-01' OR challenge_type IS NULL) LIMIT 1",
             token
         )
         if row:

backend/middleware/error_handler.py

@@ -126,10 +126,26 @@ class GlobalExceptionHandler:
             except Exception as body_error:
                 logger.debug(f"Could not extract raw body for debugging: {body_error}")
         
-        # Enhanced log message for agent heartbeats
+        # Build a sanitized log summary. The raw `str(exc)` from Pydantic
-        log_message = f"Validation error: {str(exc)}"
+        # contains the user-supplied `input` value for each failed field —
+        # which leaks secrets like TOTP codes, backup codes, mfa_token, and
+        # passwords to plaintext logs. Use only field NAMES + types here;
+        # `validation_details` (already sanitized to {field, message, type})
+        # is attached separately for downstream structured logging.
+        _field_names = [
+            err.get("field", "unknown")
+            for err in error_details.get("validation_errors", [])
+        ]
+        _err_count = len(error_details.get("validation_errors", []))
+        log_message = (
+            f"Validation error: {_err_count} field(s) failed validation: "
+            f"[{', '.join(_field_names)}]"
+        )
         if agent_name != "unknown":
-            log_message = f"Agent '{agent_name}' heartbeat validation error: {str(exc)}"
+            log_message = (
+                f"Agent '{agent_name}' heartbeat validation error: "
+                f"{_err_count} field(s) failed: [{', '.join(_field_names)}]"
+            )
         
         # Log validation error with enhanced details
         log_with_correlation(

backend/middleware/mfa_rate_limit_key.py

@@ -0,0 +1,147 @@
+"""MFA rate-limit key extraction (user-aware + ingress-aware).
+
+Problem with the default ``slowapi.util.get_remote_address``:
+
+  * Behind an ingress / reverse proxy / load balancer, every request looks
+    like it originates from the same upstream IP (the proxy itself). A
+    5/minute IP-bucket therefore becomes a 5/minute *whole-organization*
+    bucket. In a 500-user enterprise rolling out MFA, this stalls the
+    rollout to a trickle.
+
+Strategy (combined B + C from the design review):
+
+  1. **User-aware key (preferred):** if the request carries a valid Bearer
+     JWT, derive the bucket from ``user:<id>``. Each authenticated user
+     gets an isolated bucket, regardless of source IP. A malicious user
+     burning their own quota cannot starve the other 499.
+
+  2. **Trusted-proxy XFF fallback:** if the request is unauthenticated
+     (e.g. ``/login`` flow, future endpoints) and the TCP peer is in
+     ``MFA_TRUSTED_PROXY_CIDRS``, peel the first hop off ``X-Forwarded-For``.
+     This preserves real-IP buckets behind a known ingress without
+     accepting spoofed headers from the public internet.
+
+  3. **Default fallback:** plain ``request.client.host`` (slowapi default).
+
+JWT decode is intentionally signature-verified (replay/spoof protection)
+and *sync* — slowapi's decorator hook is sync, and our JWT library
+(python-jose) is sync as well. Failed verification silently downgrades
+to IP-based bucketing — never crashes the decorator.
+
+Env vars:
+  * ``MFA_TRUSTED_PROXY_CIDRS`` — comma-separated CIDR list of trusted
+    upstream proxies. Empty (default) disables XFF parsing entirely,
+    which is the safe choice when the deployment topology is unknown.
+    Examples:
+        ``MFA_TRUSTED_PROXY_CIDRS=10.0.0.0/8,172.16.0.0/12``
+        ``MFA_TRUSTED_PROXY_CIDRS=192.168.0.0/16``
+"""
+from __future__ import annotations
+
+import logging
+import os
+from ipaddress import ip_address, ip_network
+from typing import List
+
+from fastapi import Request
+from jose import jwt
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_trusted_cidrs() -> List:
+    """Parse the trusted-proxy CIDR list at import time.
+
+    A malformed entry is logged and skipped — we never crash the process
+    over a typo in operational config.
+    """
+    raw = os.getenv("MFA_TRUSTED_PROXY_CIDRS", "").strip()
+    if not raw:
+        return []
+    nets = []
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        try:
+            nets.append(ip_network(entry, strict=False))
+        except ValueError:
+            logger.warning(
+                "MFA_TRUSTED_PROXY_CIDRS: ignoring invalid CIDR %r", entry
+            )
+    return nets
+
+
+_TRUSTED_NETS = _parse_trusted_cidrs()
+
+
+def _peer_ip(request: Request) -> str:
+    """The TCP peer IP — never raises; falls back to ``0.0.0.0``."""
+    return request.client.host if request.client else "0.0.0.0"
+
+
+def _is_trusted_peer(peer: str) -> bool:
+    if not _TRUSTED_NETS:
+        return False
+    try:
+        peer_ip = ip_address(peer)
+    except ValueError:
+        return False
+    return any(peer_ip in net for net in _TRUSTED_NETS)
+
+
+def _real_ip(request: Request) -> str:
+    """If TCP peer is in a trusted proxy CIDR, peel off the first
+    ``X-Forwarded-For`` IP; otherwise return the peer.
+
+    ``X-Forwarded-For`` from an *untrusted* peer is intentionally ignored —
+    accepting it would let any client spoof their bucket.
+    """
+    peer = _peer_ip(request)
+    if not _is_trusted_peer(peer):
+        return peer
+    xff = request.headers.get("X-Forwarded-For")
+    if not xff:
+        return peer
+    first = xff.split(",")[0].strip()
+    return first or peer
+
+
+def _user_id_from_jwt(request: Request) -> str | None:
+    """Sync JWT decode → ``user_id`` (or ``sub``) claim. None on failure.
+
+    Uses the same secret + algorithm as ``auth_middleware`` so a token that
+    is valid for the API surface is also valid for the rate-limit key.
+    Bad / missing / expired tokens silently return None — slowapi falls
+    back to IP bucketing.
+    """
+    auth = request.headers.get("Authorization", "")
+    if not auth.startswith("Bearer "):
+        return None
+    token = auth[7:].strip()
+    if not token or token in {"null", "undefined"}:
+        return None
+    # Late import to avoid pulling jose into module-load when not needed.
+    try:
+        from config import JWT_ALGORITHM, JWT_SECRET_KEY
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None
+    uid = payload.get("user_id") or payload.get("sub")
+    if uid is None:
+        return None
+    return str(uid)
+
+
+def mfa_rate_limit_key(request: Request) -> str:
+    """slowapi ``key_func`` for MFA endpoints.
+
+    Order:
+      1. Authenticated → ``user:<id>``
+      2. Trusted-proxy XFF → ``ip:<first hop>``
+      3. TCP peer → ``ip:<peer>``
+    """
+    uid = _user_id_from_jwt(request)
+    if uid is not None:
+        return f"user:{uid}"
+    return f"ip:{_real_ip(request)}"

backend/middleware/mfa_rate_limits.py

@@ -0,0 +1,79 @@
+"""MFA rate-limit configuration (env-overridable).
+
+Best-practice pattern:
+  - Secure-by-default values live in code (kept in sync with the threat model).
+  - Operations can override per-environment via env vars (ConfigMap on K8s)
+    WITHOUT a code change / re-release.
+  - All limits funnel through a single named constant so the decorator stays
+    declarative (``@limiter.limit(MFA_LIMITS.enroll_start)``).
+
+Env-var precedence::
+
+    MFA_RATE_LIMIT_<NAME>  >  default in code
+
+slowapi limit string syntax: ``<count>/<period>`` where period is
+``second|minute|hour|day``. Example: ``"5/minute"``.
+
+NOTE: slowapi binds limits at import time. A change to an env var requires a
+backend restart (rolling restart on K8s, ``docker compose restart backend``
+locally). This is consistent with how ``SECRET_KEY`` / ``MFA_ENCRYPTION_KEY``
+behave.
+"""
+from __future__ import annotations
+
+import logging
+import os
+import re
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+# slowapi limit-string format guard. Keeps a typo from silently disabling
+# rate-limiting at process start.
+_LIMIT_RE = re.compile(r"^\d+/(second|minute|hour|day)$")
+
+
+def _env(name: str, default: str) -> str:
+    """Read ``MFA_RATE_LIMIT_<NAME>``; fall back to ``default``.
+
+    Validates the limit string. On bad input, logs a warning and returns the
+    secure default instead of crashing the process.
+    """
+    value = os.getenv(f"MFA_RATE_LIMIT_{name}", default).strip()
+    if not _LIMIT_RE.match(value):
+        logger.warning(
+            "MFA_RATE_LIMIT_%s='%s' is not a valid slowapi limit string "
+            "(expected '<n>/<second|minute|hour|day>'); using default '%s'.",
+            name, value, default,
+        )
+        return default
+    return value
+
+
+@dataclass(frozen=True)
+class MfaRateLimits:
+    """Aggregate of MFA endpoint rate-limit strings (slowapi format).
+
+    Defaults assume the rate-limit ``key_func`` is ``mfa_rate_limit_key``
+    (user-aware + ingress-aware), NOT raw IP. Per-user buckets are safe to
+    keep generous because a misbehaving user only burns their own quota and
+    cannot starve the rest of the org. If you re-key on raw IP, retighten
+    these values (see README + ``MFA_RATE_LIMIT_<NAME>`` env overrides).
+    """
+
+    # Enrollment lifecycle — per-user buckets, large enough for org-wide rollout
+    enroll_start: str = _env("ENROLL_START", "10/minute")
+    enroll_confirm: str = _env("ENROLL_CONFIRM", "10/minute")
+
+    # Self-service maintenance
+    disable: str = _env("DISABLE", "10/minute")
+    regenerate_backup_codes: str = _env("REGENERATE_BACKUP_CODES", "5/hour")
+
+    # Admin operations (per-admin bucket; bulk reset stays tight because
+    # it is an emergency-only flow).
+    admin_reset: str = _env("ADMIN_RESET", "60/hour")
+    admin_reset_all: str = _env("ADMIN_RESET_ALL", "1/day")
+
+
+# Module-level singleton — import this from routers/mfa.py.
+MFA_LIMITS = MfaRateLimits()

backend/middleware/rate_limiter.py

@@ -2,6 +2,7 @@
 Rate Limiting Middleware for Production Security
 Protects API endpoints from abuse and DDoS attacks
 """
+import os
 import time
 import logging
 from typing import Callable
@@ -16,10 +17,15 @@ from database.connection import redis_client
 
 logger = logging.getLogger(__name__)
 
-# Rate limiter instance using Redis backend  
+# Resolve the Redis storage URI from REDIS_URL (set by docker-compose / k8s
+# ConfigMap). Falls back to the compose service hostname for backward
+# compatibility when REDIS_URL is unset.
+_REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379").rstrip("/")
+_LIMITER_STORAGE_URI = f"{_REDIS_URL}/0" if "/" not in _REDIS_URL.split("//", 1)[-1] else _REDIS_URL
+
 limiter = Limiter(
     key_func=get_remote_address,
-    storage_uri="redis://redis:6379/0",
+    storage_uri=_LIMITER_STORAGE_URI,
     default_limits=["1000/hour"],  # Default global limit
     retry_after=lambda name, t: int(t) + 10
 )

backend/models/cluster.py

@@ -8,6 +8,7 @@ class HAProxyClusterCreate(BaseModel):
     stats_socket_path: str = "/run/haproxy/admin.sock"
     haproxy_config_path: str = "/etc/haproxy/haproxy.cfg"
     haproxy_bin_path: str = "/usr/sbin/haproxy"  # HAProxy binary path
+    keepalived_config_path: str = "/etc/keepalived/keepalived.conf"  # HA/VIP: keepalived.conf path (Issue #27)
     pool_id: Optional[int] = None  # Which pool this cluster belongs to
 
 class HAProxyClusterUpdate(BaseModel):
@@ -17,6 +18,7 @@ class HAProxyClusterUpdate(BaseModel):
     stats_socket_path: Optional[str] = None
     haproxy_config_path: Optional[str] = None
     haproxy_bin_path: Optional[str] = None
+    keepalived_config_path: Optional[str] = None
     pool_id: Optional[int] = None
     is_active: Optional[bool] = None
     acme_enabled: Optional[bool] = None

backend/models/mfa.py

@@ -0,0 +1,73 @@
+"""MFA-specific Pydantic models — Issue #18, v1.6.0.
+
+Kept in a separate module so the existing User / UserUpdate contracts in
+``backend/models/user.py`` stay byte-identical for backwards compatibility.
+"""
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+class MfaVerifyRequest(BaseModel):
+    """Body of POST /api/auth/login/mfa-verify (pre-auth — no JWT)."""
+
+    mfa_token: str = Field(..., min_length=64, max_length=64)
+    # 6 digits for TOTP or 8 alphanumerics (with optional dash) for backup codes
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaEnrollStartResponse(BaseModel):
+    """Returned by POST /api/mfa/enroll/start."""
+
+    secret: str
+    otpauth_uri: str
+    expires_in: int  # pending enrollment TTL in seconds
+
+
+class MfaEnrollConfirmRequest(BaseModel):
+    """Body of POST /api/mfa/enroll/confirm (TOTP only — backup codes not yet issued)."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaEnrollConfirmResponse(BaseModel):
+    enabled: bool
+    backup_codes: List[str]
+    method: Literal["totp"] = "totp"
+
+
+class MfaDisableRequest(BaseModel):
+    """Body of POST /api/mfa/disable — TOTP or backup."""
+
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaRegenerateBackupRequest(BaseModel):
+    """Body of POST /api/mfa/backup-codes/regenerate — TOTP only."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaRegenerateBackupResponse(BaseModel):
+    backup_codes: List[str]
+
+
+class MfaAdminResetRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset/{user_id}."""
+
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaAdminResetAllRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset-all (emergency)."""
+
+    confirm: Literal["RESET ALL MFA"]
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaStatusResponse(BaseModel):
+    enabled: bool
+    method: Optional[str] = None
+    enrolled_at: Optional[str] = None
+    last_used_at: Optional[str] = None
+    backup_codes_remaining: int = 0

backend/models/site_wizard.py

@@ -488,6 +488,32 @@ class ServerStep(BaseModel):
                     "splits at the first space, so a space inside the "
                     "SNI value would corrupt the rendered config."
                 )
+        # Bulgu #90 (round-24 audit) — `cookie_value` is interpolated
+        # into the rendered `server <name> <addr>:<port> ... cookie
+        # <value> ...` line. HAProxy tokenises the line by whitespace
+        # and treats `;` as an INLINE COMMENT — and RFC 6265 cookie-
+        # value grammar separately disallows whitespace, `;`, `,`,
+        # `\\` and `"`. Pre-fix the validator only rejected newlines,
+        # so an operator could set `cookie_value="srv1; secure"` and
+        # see HAProxy silently truncate the server line at `;`
+        # (everything after becomes a comment). The Set-Cookie header
+        # rendered to clients would also fail the RFC's cookie-value
+        # grammar and most browsers drop the cookie, breaking session
+        # affinity without any error surface. Restrict to the
+        # conservative intersection — alphanumeric plus `_.-` —
+        # identical to backend.cookie_name. Legitimate session
+        # identifiers all fit in this set.
+        if info and info.field_name == "cookie_value":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.\-]+", v):
+                raise ValueError(
+                    "server.cookie_value must contain only alphanumerics, "
+                    "'.', '_' or '-' (HAProxy tokenises the server line "
+                    "by whitespace and treats ';' as an inline comment; "
+                    "RFC 6265 separately disallows whitespace, ';', ',' "
+                    "and backslash in cookie values). Got "
+                    f"{v!r}."
+                )
         return v
     # R17 (label corrected R18): CA bundle used by HAProxy to VERIFY the
     # upstream server's TLS certificate. Maps to the `ca-file` directive
@@ -658,6 +684,32 @@ class BackendStep(BaseModel):
         fields like `request_headers` / `response_headers` /
         `tcp_request_rules` are intentionally line-oriented and ARE
         NOT touched here.
+
+        Bulgu #90 (round-24 audit) — extend the validator beyond
+        newlines. Pre-fix `cookie_name` accepted strings like
+        `SESS'; DROP TABLE backends; --` (the SQL substring is
+        harmless thanks to parameterised queries, but the `;` is
+        HAProxy's INLINE COMMENT character: the renderer emits
+        `cookie SESS'; DROP TABLE backends; -- insert indirect
+        nocache`, which HAProxy parses as `cookie SESS'` followed by
+        an inline comment that swallows the persistence options the
+        operator typed). Result: session affinity silently broken
+        and the operator has zero diagnostic signal — the wizard
+        accepted the input, the apply succeeded, but cookies never
+        get re-emitted with the expected name. The same trap exists
+        for any HAProxy-section-keyword or whitespace token because
+        HAProxy tokenises by space.
+
+        `cookie_name` MUST be a single token. We use the conservative
+        intersection of RFC 6265 cookie-token chars and HAProxy
+        directive-name chars: alphanumeric plus `_.-`. Operators
+        with legitimate session cookies all live inside this set.
+
+        `cookie_options` is a space-separated keyword list
+        (`insert indirect nocache` etc., optionally `domain example.
+        com`, `attr SameSite=Lax`). Allow letters/digits/space/dot/
+        hyphen/underscore/equals. Reject `;` (comment), backslash,
+        quotes, and other shell metacharacters.
         """
         if v is None:
             return v
@@ -669,6 +721,28 @@ class BackendStep(BaseModel):
                 "newlines would smuggle additional directives into "
                 "the rendered config)"
             )
+        field_name = info.field_name if info else "cookie field"
+        if field_name == "cookie_name":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.\-]+", v):
+                raise ValueError(
+                    "backend.cookie_name must contain only alphanumerics, "
+                    "'.', '_' or '-' (HAProxy tokenises the `cookie` "
+                    "directive by whitespace and treats ';' as an inline "
+                    "comment — anything else silently truncates the "
+                    "rendered persistence options). Got "
+                    f"{v!r}."
+                )
+        elif field_name == "cookie_options":
+            import re as _re
+            if not _re.fullmatch(r"[A-Za-z0-9_.=\- ]*", v):
+                raise ValueError(
+                    "backend.cookie_options must contain only "
+                    "alphanumerics, spaces, '=', '.', '_' or '-' "
+                    "(HAProxy parses `;` as an inline comment, and "
+                    "quoting / backslash metacharacters are not "
+                    f"part of the `cookie` keyword grammar). Got {v!r}."
+                )
         return v
 
     @field_validator("health_check_uri")

backend/models/vip.py

@@ -0,0 +1,245 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Pydantic request/response models for the VIP management API. Field validation
+is strict because several values flow into a generated keepalived.conf and into
+root-run agent commands — we reuse the same FORBIDDEN-metacharacter discipline as
+models/agent.py (Bulgu #81) and validate the VIP as a real IPv4 address.
+
+Pydantic idiom: the project runs pydantic>=2.5; this module uses the v2-native
+@field_validator/@model_validator style (matching models/ssl.py).
+"""
+import ipaddress
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+# Shell/keepalived.conf metacharacters that must never appear in a value that
+# reaches the generated config or a root-run agent command (mirrors
+# models/agent.py:202, the Bulgu #81 convention).
+_FORBIDDEN = set('$`;&|<>"\'\\\n\r\x00*?')
+
+
+def _validate_iface(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('network_interface must be a non-empty string')
+    s = v.strip()
+    if len(s) > 64:
+        raise ValueError('network_interface too long (max 64 chars)')
+    if any(c in _FORBIDDEN for c in s):
+        raise ValueError('network_interface contains a forbidden character')
+    # Linux iface names: letters, digits, and . _ - : @ (vlans/aliases/altnames)
+    import re as _re
+    if not _re.match(r'^[A-Za-z0-9][A-Za-z0-9._:@-]{0,63}$', s):
+        raise ValueError(
+            'network_interface must start alphanumeric and contain only '
+            'letters, digits, and . _ - : @'
+        )
+    return s
+
+
+def _validate_ipv4(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('virtual_ip must be a non-empty string')
+    s = v.strip()
+    try:
+        addr = ipaddress.ip_address(s)
+    except ValueError:
+        raise ValueError(f'virtual_ip={v!r} is not a valid IP address')
+    if addr.version != 4:
+        raise ValueError('virtual_ip must be IPv4 — IPv6 VIPs are not supported yet')
+    return s
+
+
+class VIPMemberIn(BaseModel):
+    agent_id: int
+    network_interface: str
+    role: str = 'BACKUP'
+    priority: int = 100
+
+    @field_validator('network_interface')
+    @classmethod
+    def _iface(cls, v):
+        return _validate_iface(v)
+
+    @field_validator('role')
+    @classmethod
+    def _role(cls, v):
+        u = (v or '').strip().upper()
+        if u not in ('MASTER', 'BACKUP'):
+            raise ValueError("role must be 'MASTER' or 'BACKUP'")
+        return u
+
+    @field_validator('priority')
+    @classmethod
+    def _priority(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 254):
+            raise ValueError('priority must be an integer between 1 and 254')
+        return v
+
+
+def _validate_members(members: List['VIPMemberIn']) -> List['VIPMemberIn']:
+    # >=1 node: a single-node VIP is a keepalived-managed floating IP without failover
+    # (valid, e.g. a one-box HAProxy that wants a stable VIP, or before a 2nd node is added).
+    # Two or more nodes give actual VRRP failover. The UI flags the single-node case.
+    if not members:
+        raise ValueError('a VIP needs at least 1 member node')
+    agent_ids = [m.agent_id for m in members]
+    if len(set(agent_ids)) != len(agent_ids):
+        raise ValueError('each node may appear at most once in a VIP')
+    masters = [m for m in members if m.role == 'MASTER']
+    if len(masters) != 1:
+        raise ValueError('exactly one member must be MASTER')
+    master_prio = masters[0].priority
+    if any(m.role == 'BACKUP' and m.priority >= master_prio for m in members):
+        raise ValueError('the MASTER must have a strictly higher priority than every BACKUP')
+    return members
+
+
+def _validate_auth_pass(v: Optional[str]) -> Optional[str]:
+    if v is None or v == '':
+        return None
+    # keepalived PASS auth_pass is silently truncated to 8 chars (B-3) — reject longer
+    # so MASTER/BACKUP never silently disagree.
+    if not (1 <= len(v) <= 8):
+        raise ValueError('auth_pass must be 1-8 characters (keepalived PASS limit)')
+    if any(c in _FORBIDDEN for c in v):
+        raise ValueError('auth_pass contains a forbidden character')
+    # No whitespace: keepalived PASS auth_pass is a single token, and a whitespace-containing
+    # secret would only partially redact in the masked config diff (review HIGH-2).
+    if any(c.isspace() for c in v):
+        raise ValueError('auth_pass must not contain whitespace')
+    return v
+
+
+class VIPCreate(BaseModel):
+    name: str
+    description: Optional[str] = None
+    pool_id: int
+    virtual_ip: str
+    prefix_length: int = 24
+    virtual_router_id: Optional[int] = None   # auto-allocated within the pool when omitted
+    advert_int: int = 1
+    auth_pass: Optional[str] = None           # plaintext on the wire; stored Fernet-encrypted
+    use_unicast: bool = True
+    track_haproxy: bool = True
+    members: List[VIPMemberIn]
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if not isinstance(v, str) or not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255:
+            raise ValueError('name too long (max 255 chars)')
+        if any(c in _FORBIDDEN for c in s):
+            raise ValueError('name contains a forbidden character')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v)
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 32):
+            raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be an integer between 1 and 255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('advert_int must be an integer between 1 and 255 (seconds)')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        _validate_members(self.members)
+        return self
+
+
+class VIPUpdate(BaseModel):
+    """All fields optional — only provided fields are changed. Any change sets the
+    VIP back to last_config_status='PENDING' (router-side)."""
+    name: Optional[str] = None
+    description: Optional[str] = None
+    virtual_ip: Optional[str] = None
+    prefix_length: Optional[int] = None
+    virtual_router_id: Optional[int] = None
+    advert_int: Optional[int] = None
+    auth_pass: Optional[str] = None           # provide only to rotate; omit to keep existing
+    use_unicast: Optional[bool] = None
+    track_haproxy: Optional[bool] = None
+    members: Optional[List[VIPMemberIn]] = None
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if v is None:
+            return v
+        if not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255 or any(c in _FORBIDDEN for c in s):
+            raise ValueError('name invalid (too long or forbidden character)')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v) if v is not None else v
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 32):
+            raise ValueError('prefix_length must be 1-32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be 1-255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('advert_int must be 1-255 seconds')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        if self.members is not None:
+            _validate_members(self.members)
+        return self

backend/requirements.txt

@@ -15,4 +15,5 @@ bcrypt>=4.0.1
 slowapi>=0.1.9
 psutil>=5.9.8
 pytz>=2023.3
-josepy>=1.14.0
+josepy>=1.14.0
+pyotp>=2.9.0

backend/routers/acme_diagnostics.py

@@ -16,9 +16,11 @@ Per-user 5/min rate-limit via user_activity_logs SQL count (M18 / R50).
 
 import json
 import logging
+import uuid
 from datetime import datetime
 from typing import List, Optional
 
+import asyncpg
 from fastapi import APIRouter, Header, HTTPException
 
 from auth_middleware import check_user_permission, get_current_user_from_token
@@ -58,16 +60,39 @@ async def _enforce_rate_limit(conn, user_id: int, action: str) -> None:
 
 
 async def _load_order(conn, order_id: int) -> dict:
-    row = await conn.fetchrow(
+    """Fetch the order row, or raise a clean 404.
-        """
+
-        SELECT id, account_id, status, domains, cluster_ids, error_detail,
+    Bulgu #96 (prod-canary audit): `letsencrypt_orders.id` is a Postgres
-               post_completion_actions, pending_apply_version_name,
+    int4 column. A path-param `order_id` outside the int4 range
-               wizard_staged_until, created_by
+    (e.g. > 2_147_483_647) used to bubble up as
-        FROM letsencrypt_orders
+    `asyncpg.exceptions.DataError: invalid input for query argument $1:
-        WHERE id = $1
+    ... (value out of int32 range)` — which the diagnostics endpoint
-        """,
+    then surfaced in a `diagnostics_unavailable` envelope, leaking the
-        order_id,
+    raw Postgres / asyncpg error string ("query argument $1",
-    )
+    "int32 range") into the operator-visible response body.
+    Semantically an out-of-range ID can never reference a real order,
+    so we treat it identically to "row not found" and return a clean
+    404 — same shape as the not-found path, no SQL detail leakage.
+    """
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT id, account_id, status, domains, cluster_ids, error_detail,
+                   post_completion_actions, pending_apply_version_name,
+                   wizard_staged_until, created_by, challenge_type
+            FROM letsencrypt_orders
+            WHERE id = $1
+            """,
+            order_id,
+        )
+    except asyncpg.exceptions.DataError as exc:
+        logger.info(
+            "ACME order lookup rejected by Postgres (out-of-range / "
+            "uncastable id): order_id=%s exc=%s",
+            order_id,
+            exc,
+        )
+        raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     if not row:
         raise HTTPException(status_code=404, detail=f"Order {order_id} not found")
     return dict(row)
@@ -86,35 +111,154 @@ def _parse_jsonb_list(raw, default):
     return default
 
 
+def _diagnostic_failure_envelope(
+    order_id: int,
+    correlation_id: str,
+    exc: Exception,
+    *,
+    stage: str,
+) -> dict:
+    """Build a structured response when the diagnostic suite itself
+    cannot run. Bulgu #94 (Round-25): we return HTTP 200 with this
+    envelope rather than 500 so the UI can still SHOW the operator
+    what happened — the panel's whole purpose is to surface failure
+    causes, and the panel itself silently 500-ing is the worst-case
+    UX. The server-side log carries the full traceback keyed by
+    correlation_id for operator follow-up.
+    """
+    return {
+        "order_id": order_id,
+        "status": "diagnostics_unavailable",
+        "checks": [
+            {
+                "id": "diagnostics_runner",
+                "label": "Diagnostic runner",
+                "status": "fail",
+                "severity": "error",
+                "message": (
+                    f"Diagnostics could not run ({stage}): "
+                    f"{exc.__class__.__name__}: {exc}"
+                ),
+                "details": {
+                    "stage": stage,
+                    "exception_type": exc.__class__.__name__,
+                    "exception_message": str(exc),
+                    "correlation_id": correlation_id,
+                    "hint": (
+                        "Check the backend log for correlation_id "
+                        f"{correlation_id} for the full traceback."
+                    ),
+                },
+                "duration_ms": None,
+            }
+        ],
+        "humanized_error": {
+            "title": "Diagnostic panel could not run",
+            "message": (
+                "The diagnostic runner itself crashed before any check "
+                "could complete. This is independent of whether the ACME "
+                "provider is reachable from this cluster."
+            ),
+            "hint": (
+                "Share the correlation_id below with the platform team; "
+                "they can grep the API log for the full traceback."
+            ),
+            "correlation_id": correlation_id,
+        },
+        "meta": {
+            "correlation_id": correlation_id,
+            "error_stage": stage,
+            "error_type": exc.__class__.__name__,
+            "error_message": str(exc),
+        },
+        "generated_at": datetime.utcnow().isoformat() + "Z",
+    }
+
+
 @router.post("/orders/{order_id}/diagnostics")
 async def run_diagnostics(order_id: int, authorization: str = Header(None)):
-    """Run the full pre-flight + post-failure diagnostic suite."""
+    """Run the full pre-flight + post-failure diagnostic suite.
+
+    Bulgu #94 (Round-25 audit): this endpoint must NEVER return HTTP 500
+    for an in-suite failure. The diagnostic panel exists precisely to
+    explain what is broken; producing an opaque 500 defeats the entire
+    feature. Authentication / authorisation / rate-limit / not-found
+    errors still raise the appropriate 4xx, but any unexpected
+    exception from check execution is converted to a 200 response with
+    a structured failure envelope so the UI can display the cause.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
         await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostics_run")
         order = await _load_order(conn, order_id)
+    except HTTPException:
+        await close_database_connection(conn)
+        raise
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        logger.exception(
+            "ACME diagnostics setup failed for order=%s correlation_id=%s",
+            order_id,
+            correlation_id,
+        )
+        try:
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="load_order")
+        finally:
+            await close_database_connection(conn)
 
+    try:
         domains = _parse_jsonb_list(order["domains"], [])
         cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
 
-        results = await run_checks(
+        try:
-            conn,
+            results = await run_checks(
-            domains=domains,
+                conn,
-            cluster_ids=cluster_ids,
+                domains=domains,
-            account_id=order["account_id"],
+                cluster_ids=cluster_ids,
-        )
+                account_id=order["account_id"],
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            # run_checks now wraps individual checks, but a top-level
+            # crash (e.g. lost DB connection) still needs to be visible.
+            logger.exception(
+                "ACME diagnostics top-level failure for order=%s correlation_id=%s",
+                order_id,
+                correlation_id,
+            )
+            return _diagnostic_failure_envelope(order_id, correlation_id, exc, stage="run_checks")
 
-        humanized_error = humanize_error_detail(order["error_detail"])
+        try:
+            humanized_error = humanize_error_detail(order["error_detail"])
+        except Exception as exc:  # noqa: BLE001 — defensive
+            logger.warning(
+                "humanize_error_detail failed for order=%s correlation_id=%s: %s",
+                order_id,
+                correlation_id,
+                exc,
+            )
+            humanized_error = {
+                "title": "ACME error (raw)",
+                "message": str(order["error_detail"]) if order["error_detail"] else "",
+                "hint": "",
+                "parse_error": exc.__class__.__name__,
+            }
 
         return {
             "order_id": order_id,
             "status": order["status"],
             "checks": results,
             "humanized_error": humanized_error,
+            "meta": {
+                "correlation_id": correlation_id,
+                "checks_total": len(results),
+                "checks_failed": sum(1 for r in results if r.get("status") == "fail"),
+                "checks_warn": sum(1 for r in results if r.get("status") == "warn"),
+            },
             "generated_at": datetime.utcnow().isoformat() + "Z",
         }
     finally:
@@ -127,7 +271,15 @@ async def rerun_diagnostic_check(
     check_id: str,
     authorization: str = Header(None),
 ):
-    """Re-run a single check (DNS / port80 / routing / account / agents)."""
+    """Re-run a single check (DNS / port80 / routing / account / agents).
+
+    Bulgu #94 follow-up (Round-25 audit): the rerun path is just as
+    sensitive to opaque 500s as the full-suite POST. If
+    `_enforce_rate_limit` / `_load_order` / `run_checks` raises an
+    unexpected exception, we surface a structured `fail` row in the
+    same shape the table already renders — so the operator clicking
+    "Re-run" never sees an opaque toast and the row updates in place.
+    """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
@@ -138,30 +290,120 @@ async def rerun_diagnostic_check(
             detail=f"Unknown check_id '{check_id}'. Valid: {', '.join(CHECK_IDS)}",
         )
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
     try:
-        await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
+        try:
-        order = await _load_order(conn, order_id)
+            await _enforce_rate_limit(conn, current_user["id"], "acme_diagnostic_check_rerun")
+            order = await _load_order(conn, order_id)
+        except HTTPException:
+            raise
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun setup failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run setup failed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "setup",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "setup"},
+            }
 
-        domains = _parse_jsonb_list(order["domains"], [])
+        try:
-        cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
+            domains = _parse_jsonb_list(order["domains"], [])
-
+            cluster_ids = _parse_jsonb_list(order["cluster_ids"], [])
-        results = await run_checks(
+            results = await run_checks(
-            conn,
+                conn,
-            domains=domains,
+                domains=domains,
-            cluster_ids=cluster_ids,
+                cluster_ids=cluster_ids,
-            account_id=order["account_id"],
+                account_id=order["account_id"],
-            only=[check_id],
+                only=[check_id],
-        )
+                challenge_type=(order.get("challenge_type") or "http-01"),
+            )
+        except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+            logger.exception(
+                "ACME rerun run_checks failed for order=%s check=%s correlation_id=%s",
+                order_id,
+                check_id,
+                correlation_id,
+            )
+            return {
+                "order_id": order_id,
+                "check": {
+                    "id": check_id,
+                    "label": check_id,
+                    "status": "fail",
+                    "severity": "error",
+                    "message": (
+                        f"Re-run crashed: "
+                        f"{exc.__class__.__name__}: {exc}"
+                    ),
+                    "details": {
+                        "stage": "run_checks",
+                        "exception_type": exc.__class__.__name__,
+                        "exception_message": str(exc),
+                        "correlation_id": correlation_id,
+                    },
+                    "duration_ms": None,
+                },
+                "meta": {"correlation_id": correlation_id, "error_stage": "run_checks"},
+            }
 
         return {
             "order_id": order_id,
             "check": results[0] if results else None,
+            "meta": {"correlation_id": correlation_id},
         }
     finally:
         await close_database_connection(conn)
 
 
+async def _user_activity_columns(conn) -> set:
+    """Return the set of column names actually present on user_activity_logs.
+
+    Bulgu #95 (Round-25 audit) — the canonical migration for
+    `user_activity_logs` defines `id, user_id, action, resource_type,
+    resource_id, details, ip_address, user_agent, created_at, timestamp`.
+    There is NO `status` column. The original `/events` SELECT pulled
+    `status` directly, so every diagnostic-panel open against an order
+    that had any user-activity-log correlation raised
+    `UndefinedColumnError: column "status" does not exist` and the API
+    returned HTTP 500. We now introspect the schema and only project
+    the columns that exist, so deployments at any migration level keep
+    rendering the diagnostic panel.
+    """
+    try:
+        rows = await conn.fetch(
+            """
+            SELECT column_name
+            FROM information_schema.columns
+            WHERE table_name = 'user_activity_logs'
+            """
+        )
+        return {r["column_name"] for r in rows}
+    except Exception as exc:  # noqa: BLE001 — schema introspection is best-effort
+        logger.warning("user_activity_logs schema introspection failed: %s", exc)
+        return set()
+
+
 @router.get("/orders/{order_id}/events")
 async def get_order_events(
     order_id: int,
@@ -174,6 +416,12 @@ async def get_order_events(
       resource_id=order_id) for context.
 
     Sorted by created_at ASC (oldest first) so the timeline reads naturally.
+
+    Bulgu #94/#95 (Round-25 audit): every sub-query is wrapped so that a
+    partial failure (missing column, missing table, malformed JSONB) is
+    surfaced via `meta.errors[]` rather than collapsing the whole panel
+    to HTTP 500. The diagnostic UI is a debugging surface — it must not
+    itself become opaque when one of its data sources is degraded.
     """
     current_user = await get_current_user_from_token(authorization)
     if not await check_user_permission(current_user["id"], "ssl", "read"):
@@ -182,95 +430,168 @@ async def get_order_events(
     if limit <= 0 or limit > 500:
         limit = 100
 
+    correlation_id = uuid.uuid4().hex[:12]
     conn = await get_database_connection()
+    section_errors: List[dict] = []
     try:
-        # Existence check
+        try:
-        await _load_order(conn, order_id)
+            await _load_order(conn, order_id)
-
+        except HTTPException:
-        # Detect whether acme_order_events exists (zero-impact for envs that
+            raise
-        # have not yet run the v1.5.0 migration). Returns empty event_log when
+        except Exception as exc:  # noqa: BLE001 — surface, don't 500
-        # not yet present rather than 500-ing.
+            logger.exception(
-        events_table_exists = await conn.fetchval(
+                "ACME events load_order failed order=%s correlation_id=%s",
-            """
+                order_id,
-            SELECT EXISTS (
+                correlation_id,
-                SELECT 1 FROM information_schema.tables WHERE table_name = 'acme_order_events'
             )
-            """
+            return {
-        )
+                "order_id": order_id,
+                "events": [],
+                "count": 0,
+                "meta": {
+                    "correlation_id": correlation_id,
+                    "errors": [
+                        {
+                            "section": "load_order",
+                            "exception_type": exc.__class__.__name__,
+                            "message": str(exc),
+                        }
+                    ],
+                },
+            }
 
         events: List[dict] = []
-        if events_table_exists:
+
-            event_rows = await conn.fetch(
+        # --- Section 1: acme_order_events ---
+        try:
+            events_table_exists = await conn.fetchval(
                 """
-                SELECT id, event_type, severity, message, details, correlation_id, created_at
+                SELECT EXISTS (
-                FROM acme_order_events
+                    SELECT 1 FROM information_schema.tables
-                WHERE order_id = $1
+                    WHERE table_name = 'acme_order_events'
-                ORDER BY created_at ASC, id ASC
+                )
-                LIMIT $2
+                """
-                """,
+            )
+            if events_table_exists:
+                event_rows = await conn.fetch(
+                    """
+                    SELECT id, event_type, severity, message, details, correlation_id, created_at
+                    FROM acme_order_events
+                    WHERE order_id = $1
+                    ORDER BY created_at ASC, id ASC
+                    LIMIT $2
+                    """,
+                    order_id,
+                    limit,
+                )
+                for r in event_rows:
+                    # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
+                    # raw JSON strings (no codec on the pool). For the FE
+                    # contract the `details` field MUST be either a dict or
+                    # null — otherwise the React renderer ends up trying to
+                    # access `details.foo` on a plain string and silently
+                    # gets undefined.
+                    _det = r["details"]
+                    if isinstance(_det, str):
+                        try:
+                            _det = json.loads(_det)
+                        except Exception:
+                            _det = {}
+                    if not isinstance(_det, (dict, list)):
+                        _det = {} if _det is None else {"raw": str(_det)}
+                    events.append({
+                        "source": "acme_order_event",
+                        "id": r["id"],
+                        "event_type": r["event_type"],
+                        "severity": r["severity"],
+                        "message": r["message"],
+                        "details": _det,
+                        "correlation_id": r["correlation_id"],
+                        "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
+                            if r["created_at"] else None,
+                    })
+        except Exception as exc:  # noqa: BLE001 — surface and continue
+            logger.exception(
+                "ACME events acme_order_events query failed order=%s correlation_id=%s",
                 order_id,
-                limit,
+                correlation_id,
             )
-            for r in event_rows:
+            section_errors.append({
-                # R18c round 8 (Bulgu A): asyncpg returns JSONB columns as
+                "section": "acme_order_events",
-                # raw JSON strings (no codec on the pool). For the FE
+                "exception_type": exc.__class__.__name__,
-                # contract the `details` field MUST be either a dict or
+                "message": str(exc),
-                # null — otherwise the React renderer ends up trying to
+            })
-                # access `details.foo` on a plain string and silently
+
-                # gets undefined.
+        # --- Section 2: user_activity_logs (best-effort, schema-aware) ---
-                _det = r["details"]
+        try:
-                if isinstance(_det, str):
+            ua_columns = await _user_activity_columns(conn)
-                    try:
+            if "resource_id" in ua_columns and "resource_type" in ua_columns:
-                        _det = json.loads(_det)
+                # Project only columns we know exist. `status` is NOT in
-                    except Exception:
+                # the canonical schema and was the original 500 cause.
-                        _det = {}
+                projection_candidates = [
-                if not isinstance(_det, (dict, list)):
+                    "id", "action", "resource_type", "resource_id",
-                    _det = {} if _det is None else {"raw": str(_det)}
+                    "details", "created_at", "user_id", "status",
-                events.append({
+                ]
-                    "source": "acme_order_event",
+                projection = [c for c in projection_candidates if c in ua_columns]
-                    "id": r["id"],
+                if "id" not in projection or "created_at" not in projection:
-                    "event_type": r["event_type"],
+                    raise RuntimeError(
-                    "severity": r["severity"],
+                        "user_activity_logs is missing required columns "
-                    "message": r["message"],
+                        "(id / created_at) — skipping correlation"
-                    "details": _det,
+                    )
-                    "correlation_id": r["correlation_id"],
+                sql = (
-                    "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
+                    f"SELECT {', '.join(projection)} "
-                        if r["created_at"] else None,
+                    "FROM user_activity_logs "
+                    "WHERE resource_type = 'letsencrypt_order' AND resource_id = $1 "
+                    "ORDER BY created_at ASC, id ASC LIMIT $2"
+                )
+                ua_rows = await conn.fetch(sql, str(order_id), limit)
+                for r in ua_rows:
+                    rd = dict(r)
+                    raw_details = rd.get("details")
+                    if isinstance(raw_details, str):
+                        msg = raw_details[:500]
+                        details_obj = {}
+                        try:
+                            parsed = json.loads(raw_details)
+                            if isinstance(parsed, (dict, list)):
+                                details_obj = parsed
+                        except Exception:
+                            details_obj = {}
+                    elif isinstance(raw_details, (dict, list)):
+                        msg = ""
+                        details_obj = raw_details
+                    else:
+                        msg = ""
+                        details_obj = {}
+                    raw_status = rd.get("status") or ""
+                    severity = "info" if str(raw_status).lower() in ("success", "ok", "") else "warn"
+                    events.append({
+                        "source": "user_activity_log",
+                        "id": rd.get("id"),
+                        "event_type": rd.get("action"),
+                        "severity": severity,
+                        "message": msg,
+                        "details": details_obj,
+                        "correlation_id": None,
+                        "created_at": rd["created_at"].isoformat().replace("+00:00", "Z")
+                            if rd.get("created_at") else None,
+                    })
+            else:
+                section_errors.append({
+                    "section": "user_activity_logs",
+                    "exception_type": "SchemaMissing",
+                    "message": "user_activity_logs lacks resource_type/resource_id columns",
                 })
-
+        except Exception as exc:  # noqa: BLE001 — surface and continue
-        # User activity rows correlated by resource — schema is permissive
+            logger.exception(
-        # (`resource_type`/`resource_id` may not always be populated for older
+                "ACME events user_activity_logs query failed order=%s correlation_id=%s",
-        # rows) so this query stays best-effort.
+                order_id,
-        ua_rows = await conn.fetch(
+                correlation_id,
-            """
-            SELECT id, action, resource_type, resource_id, status, details, created_at, user_id
-            FROM user_activity_logs
-            WHERE resource_type = 'letsencrypt_order' AND resource_id = $1
-            ORDER BY created_at ASC, id ASC
-            LIMIT $2
-            """,
-            str(order_id),
-            limit,
-        ) if await conn.fetchval(
-            """
-            SELECT EXISTS (
-                SELECT 1 FROM information_schema.columns
-                WHERE table_name = 'user_activity_logs' AND column_name = 'resource_id'
             )
-            """
+            section_errors.append({
-        ) else []
+                "section": "user_activity_logs",
-
+                "exception_type": exc.__class__.__name__,
-        for r in ua_rows:
+                "message": str(exc),
-            events.append({
-                "source": "user_activity_log",
-                "id": r["id"],
-                "event_type": r["action"],
-                "severity": "info" if (r["status"] or "").lower() in ("success", "ok", "") else "warn",
-                "message": (r["details"] or "")[:500] if isinstance(r["details"], str) else "",
-                "details": r["details"] if not isinstance(r["details"], (str, type(None))) else {},
-                "correlation_id": None,
-                "created_at": r["created_at"].isoformat().replace("+00:00", "Z")
-                    if r["created_at"] else None,
             })
 
         events.sort(key=lambda e: (e["created_at"] or "", e.get("id") or 0))
@@ -279,6 +600,10 @@ async def get_order_events(
             "order_id": order_id,
             "events": events,
             "count": len(events),
+            "meta": {
+                "correlation_id": correlation_id,
+                "errors": section_errors,
+            },
         }
     finally:
         await close_database_connection(conn)

backend/routers/agent.py

@@ -29,6 +29,40 @@ AGENT_VERSIONS = {
     "linux": "2.0.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -730,14 +764,15 @@ async def generate_uninstall_script(platform: str, authorization: str = Header(N
     ```
     """
     try:
-        # Validate platform
+        # Normalize platform to a canonical key (always 'linux' or 'macos').
-        platform_lower = platform.lower()
+        # macOS agents register with platform 'darwin' (from `uname -s`), so the
-        if platform_lower not in ['linux', 'macos']:
+        # strict ['linux','macos'] check used to 400 on the UI's uninstall flow
-            raise HTTPException(
+        # (GET /generate-uninstall-script/darwin). Reuse the same get_platform_key()
-                status_code=400, 
+        # helper the install-script generator uses, so darwin/osx/mac and the
-                detail=f"Invalid platform: {platform}. Must be 'linux' or 'macos'"
+        # linux distro variants all resolve correctly. Backward-compatible:
-            )
+        # 'linux'/'macos' still map to themselves.
-        
+        platform_lower = get_platform_key(platform)
+
         # Read uninstall script from agent_scripts directory (same as install scripts)
         import os
         script_filename = f"uninstall-agent-{platform_lower}.sh"
@@ -861,7 +896,25 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
         # Validate cluster access if agent belongs to a cluster
         if agent['cluster_id']:
             await validate_user_cluster_access(current_user['id'], agent['cluster_id'], conn)
-        
+
+        # HA/VIP (Issue #27): block deleting a node that's still a member of an active VIP.
+        # Otherwise the CASCADE would silently drop it from the VIP (breaking the one-MASTER
+        # topology with no signal) and the still-running node would keep advertising the VIP
+        # with no way to be told to tear down (review MED-3). Make the operator remove it
+        # from the VIP first — that stages a clean PENDING change they can apply.
+        try:
+            vip_member = await conn.fetchrow(
+                "SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+                "WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
+        except Exception:  # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
+            vip_member = None
+        if vip_member:
+            await close_database_connection(conn)
+            raise HTTPException(
+                status_code=409,
+                detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
+                        f"VIP on the HA / VIP page (and apply) before deleting the agent."))
+
         await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
         
         await close_database_connection(conn)
@@ -1436,47 +1489,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
+
-        # Sanitize common malformed JSON patterns from agents
+        try:
-        original_body = body_str
+            heartbeat_dict = json.loads(body_str)
-        sanitized = False
+        except json.JSONDecodeError:
-        
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
-        if re.search(r':\s*,', body_str):
+            repaired, changed = _sanitize_agent_json(body_str)
-            body_str = re.sub(r':\s*,', ': null,', body_str)
+            if changed:
-            sanitized = True
+                agent_name = "unknown"
-        
+                try:
-        # Fix 2: Empty values before closing brace
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
-        if re.search(r':\s*}', body_str):
+                    if name_match:
-            body_str = re.sub(r':\s*}', ': null}', body_str)
+                        agent_name = name_match.group(1)
-            sanitized = True
+                except Exception:
-        
+                    pass
-        # Fix 3: Trailing commas
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
-        if re.search(r',(\s*[}\]])', body_str):
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
-            sanitized = True
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
-        
+
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1656,9 +1695,13 @@ async def agent_heartbeat_by_name(
             """, x_api_key)
 
         # Check if agent was in upgrading status and version has changed
-        current_agent_status = await conn.fetchval("SELECT status FROM agents WHERE id = $1", agent_id)
+        # (v1.8.6: one round-trip instead of three; row is None exactly when the
-        current_agent_version = await conn.fetchval("SELECT version FROM agents WHERE id = $1", agent_id)
+        # per-column fetchvals would each have returned None)
-        current_upgrade_status = await conn.fetchval("SELECT upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_row = await conn.fetchrow(
+            "SELECT status, version, upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_status = current_agent_row['status'] if current_agent_row else None
+        current_agent_version = current_agent_row['version'] if current_agent_row else None
+        current_upgrade_status = current_agent_row['upgrade_status'] if current_agent_row else None
         
         # Determine new status - preserve upgrading status unless version actually changed
         new_status = current_agent_status or 'online'
@@ -1749,9 +1792,14 @@ async def agent_heartbeat_by_name(
             heartbeat_data.operating_system, heartbeat_data.kernel_version,
             heartbeat_data.uptime, heartbeat_data.cpu_count, heartbeat_data.memory_total,
             heartbeat_data.disk_space, 
-            # Convert lists to JSON for JSONB columns
+            # Convert lists to JSON for JSONB columns. Don't WIPE network_interfaces/capabilities
-            heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str) else json.dumps(heartbeat_data.network_interfaces or []),
+            # when a heartbeat omits them: send NULL so COALESCE keeps the existing value (a bare
-            heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str) else json.dumps(heartbeat_data.capabilities or []),
+            # `or []` would store "[]" and erase the reported NICs / keepalived_management on every
+            # daemon heartbeat that doesn't include them — issue #27 corporate test).
+            (heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str)
+             else (json.dumps(heartbeat_data.network_interfaces) if heartbeat_data.network_interfaces else None)),
+            (heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str)
+             else (json.dumps(heartbeat_data.capabilities) if heartbeat_data.capabilities else None)),
             agent_ip, heartbeat_data.haproxy_status, heartbeat_data.haproxy_version,
             heartbeat_data.applied_config_version, new_status, update_applied_version,
             heartbeat_data.keepalive_state, heartbeat_data.keepalive_ip)
@@ -2180,13 +2228,171 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             logger.info(f"SSL INCREMENTAL: No certificates updated since {since}")
             
         return response_data
-        
+
     except HTTPException:
         raise
     except Exception as e:
         logger.error(f"SSL certificates retrieval failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
+
+@router.get("/{agent_name}/keepalived-config")
+async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
+    teardown / no-op signal.
+
+    Snapshot-based (T-1): keys off vip_instances.is_active + the member's
+    applied_config_content, NOT the live last_config_status — so a PENDING edit never
+    flips a running member to not_configured (no mid-edit teardown). Auth is MANDATORY (a
+    valid agent key is required), but — like the /config and /ssl-certificates endpoints —
+    the agent API key is a SHARED/global install token, so the config is resolved by the
+    requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
+    (a hard 403 would break every VIP member whose name isn't the one row the shared token
+    resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
+    the agent stays inert; a node with no membership row always gets not_configured.
+    """
+    conn = None
+    try:
+        # Auth FIRST and MANDATORY: a valid agent key is REQUIRED (the response carries the
+        # VRRP secret). The token is shared/global, so resolve by agent_name and only LOG a
+        # name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
+        # unauthenticated caller can't probe which agent names exist.
+        from auth_middleware import validate_agent_api_key
+        if not x_api_key:
+            raise HTTPException(status_code=401, detail="Agent API key required")
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        # The agent API key is a SHARED/global install token (many agent rows per token),
+        # so validate_agent_api_key resolves it to one arbitrary agent for that token. Resolve
+        # the keepalived config strictly by the requested agent_name and treat a token/name
+        # mismatch as an advisory audit log — exactly like the /config and /ssl-certificates
+        # endpoints. (A hard 403 here would reject every VIP member whose name isn't the one
+        # row the shared token happens to return, so the VIP could never converge — review HIGH-1.)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        # Resolve the agent + its cluster's keepalived.conf path (cluster-driven, like the
+        # HAProxy paths). config_path is returned in EVERY response so the agent knows where
+        # to write/own-marker-check even on not_configured/teardown.
+        agent = await conn.fetchrow("""
+            SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
+                   hc.keepalived_config_path
+            FROM agents a
+            LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
+            WHERE a.name = $1
+        """, agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+        config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
+        if not agent['enabled']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        row = await conn.fetchrow("""
+            SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
+                   v.purge_on_teardown,
+                   m.applied_config_content, m.applied_config_hash
+            FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
+            WHERE m.agent_id = $1
+            -- Active VIP first (an agent has at most one). With NO active VIP, pick the most
+            -- RECENTLY updated inactive membership so a teardown reflects the latest delete
+            -- (incl. its purge flag) — not a stale older VIP the node was once part of.
+            ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
+            LIMIT 1
+        """, agent['id'])
+
+        if not row:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+        if not row['is_active']:
+            # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
+            # the agent still only purges on nodes where IT installed keepalived (install marker).
+            return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
+                    "config_path": config_path, "keepalived": None,
+                    "purge": bool(row['purge_on_teardown'])}
+        if not row['applied_config_content']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        from services.keepalived_config import build_haproxy_check_script
+        check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
+        return {
+            "agent_name": agent_name,
+            "status": "available",
+            "config_path": config_path,
+            "keepalived": {
+                "desired_state": "enabled",
+                "install_if_missing": True,
+                "vip_id": row['vip_id'],
+                "vip_name": row['vip_name'],
+                "config_content": row['applied_config_content'],
+                "config_hash": row['applied_config_hash'],
+                "check_script": check_script,
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
+        # Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
+        return {"agent_name": agent_name, "status": "not_configured",
+                "config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-status")
+async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — agent reports the outcome of a keepalived deploy/teardown.
+
+    Auth mirrors config-applied's post-Bulgu-#75 guard (reject a MISSING key — never the
+    `and` short-circuit that accepted no-key requests). The token is a shared/global install
+    token, so the status is recorded strictly for the requested agent_name and a token/name
+    mismatch is an advisory audit log (like /config-applied), not a 403 — review HIGH-1.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived status using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        vip_id = status_data.get("vip_id")
+        state = (status_data.get("state") or "").strip()[:24]
+        config_hash = (status_data.get("config_hash") or "")[:64]
+        message = status_data.get("message")
+        if vip_id is None:
+            # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
+            await conn.execute("""
+                UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
+                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                WHERE agent_id=$1
+            """, agent['id'], state, message, config_hash)
+        else:
+            await conn.execute("""
+                UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
+                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                WHERE agent_id=$1 AND vip_id=$2
+            """, agent['id'], int(vip_id), state, message, config_hash)
+        return {"status": "ok"}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-status update failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-status update failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 @router.get("/script-version")
 async def get_latest_script_version(platform: str = "macos"):
     """Get the latest available agent script version for specified platform"""

backend/routers/auth.py

@@ -9,14 +9,48 @@ from datetime import datetime, timedelta
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
 from models.user import LoginRequest, User, UserCreate, UserUpdate, UserPasswordUpdate
+from models.mfa import MfaVerifyRequest
 from utils.activity_log import log_user_activity
 from auth_middleware import get_current_user_from_token
+from services import mfa_service
 
 # Rate limiting temporarily disabled
 
 router = APIRouter(prefix="/api/auth", tags=["Authentication"])
 logger = logging.getLogger(__name__)
 
+MFA_PENDING_TTL_SECONDS = 300  # 5 minutes — pre-verification challenge lifetime
+MFA_PENDING_MAX_ATTEMPTS = 5  # invalidate token after this many wrong codes
+
+
+async def _fetch_mfa_state(conn, user_id: int):
+    """Return (mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step) or None
+    when the MFA columns aren't yet present (pre-migration deploys).
+    """
+    try:
+        return await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+    except Exception as exc:
+        logger.warning(f"MFA columns not available (assuming disabled): {exc}")
+        return None
+
+
+async def _cleanup_expired_pending_logins(conn, user_id: int) -> None:
+    """Lazy cleanup of expired pending MFA challenges for this user."""
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_logins WHERE user_id = $1 AND expires_at < NOW()",
+            user_id,
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-login cleanup skipped: {exc}")
+
 # Security scheme
 security = HTTPBearer()
 
@@ -96,7 +130,7 @@ async def login(login_request: LoginRequest, request: Request):
                 SELECT id, username, email, password_hash, is_active, role, 
                        created_at, updated_at, last_login_at
                 FROM users 
-                WHERE username = $1
+                WHERE username = $1 AND is_active = TRUE
             """, login_request.username)
         except Exception as schema_error:
             logger.warning(f"Schema error, trying fallback query: {schema_error}")
@@ -106,7 +140,7 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login_at
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
             except Exception as column_error:
                 logger.warning(f"last_login_at column error, trying last_login: {column_error}")
@@ -115,26 +149,67 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
         
         if not user:
+            # Covers both "no such user" and "soft-deleted (is_active=FALSE)".
+            # We deliberately return the same generic 401 in either case to
+            # avoid leaking whether an account exists (account enumeration
+            # prevention). Soft-deleted rows are filtered out by the
+            # `AND is_active = TRUE` predicate above.
             await close_database_connection(conn)
             logger.warning(f"Failed login attempt for username: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
+
-        if not user['is_active']:
-            await close_database_connection(conn)
-            logger.warning(f"Login attempt for inactive account: {login_request.username}")
-            raise HTTPException(status_code=401, detail="Account is deactivated")
-        
         # Verify password
         import bcrypt
         if not bcrypt.checkpw(login_request.password.encode('utf-8'), user['password_hash'].encode('utf-8')):
             await close_database_connection(conn)
             logger.warning(f"Wrong password for user: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
+
+        # Issue #18 — MFA branch (v1.6.0): if the user opted in, defer JWT mint and
+        # last_login update until /api/auth/login/mfa-verify completes.
+        mfa_state = await _fetch_mfa_state(conn, user['id'])
+        if mfa_state and mfa_state.get('mfa_enabled'):
+            await _cleanup_expired_pending_logins(conn, user['id'])
+            challenge_token = mfa_service.generate_challenge_token()
+            try:
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_logins (user_id, challenge_token, expires_at, ip_address)
+                    VALUES ($1, $2, NOW() + ($3 || ' seconds')::interval, $4)
+                    """,
+                    user['id'],
+                    challenge_token,
+                    str(MFA_PENDING_TTL_SECONDS),
+                    str(request.client.host) if request.client else None,
+                )
+            except Exception as exc:
+                await close_database_connection(conn)
+                logger.error(f"Failed to create MFA pending login: {exc}")
+                raise HTTPException(status_code=500, detail="MFA challenge creation failed")
+
+            await close_database_connection(conn)
+
+            await log_user_activity(
+                user_id=user['id'],
+                action='mfa.login.challenge_issued',
+                resource_type='mfa',
+                resource_id=str(user['id']),
+                details={'login_method': 'username_password'},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get('user-agent'),
+            )
+
+            return {
+                "mfa_required": True,
+                "mfa_token": challenge_token,
+                "methods": ["totp", "backup"],
+                "expires_in": MFA_PENDING_TTL_SECONDS,
+            }
+
         # Update last login (try different column names)
         try:
             await conn.execute("""
@@ -242,6 +317,348 @@ async def login(login_request: LoginRequest, request: Request):
         logger.error(f"Login error: {e}")
         raise HTTPException(status_code=500, detail="Login failed")
 
+@router.post(
+    "/login/mfa-verify",
+    summary="MFA Verification (Step 2 of Login)",
+    response_description="JWT access token after successful TOTP/backup verification",
+)
+async def login_mfa_verify(payload: MfaVerifyRequest, request: Request):
+    """
+    # MFA Verification — Step 2 of the two-step login flow
+
+    Submit a 6-digit TOTP code OR an 8-character backup code (with optional dash)
+    together with the ``mfa_token`` returned by ``POST /api/auth/login`` for an
+    MFA-enabled account. On success, returns the same response shape as a
+    non-MFA login (Branch A).
+
+    ## Request Body
+    - **mfa_token**: 64-char challenge token from /login response
+    - **code**: 6 digits (TOTP) or `XXXX-YYYY` (backup)
+
+    ## Error Responses
+    - **401**: Invalid code (attempts counter increments)
+    - **410**: Challenge expired or invalidated (too many wrong attempts)
+    """
+    ip_address = str(request.client.host) if request.client else None
+    user_agent = request.headers.get('user-agent')
+
+    # Outcome captured from the transactional block so we can do JWT mint /
+    # activity logging AFTER commit (no side effects on rollback).
+    success_payload: Optional[dict] = None
+    failure: Optional[dict] = None  # { user_id, attempts, invalidated, reason, http_status, detail }
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Round 1 audit fix — wrap the whole verify+update in a single
+        # transaction with row-level locks (FOR UPDATE) so two concurrent
+        # /mfa-verify calls cannot both consume the same TOTP step or the
+        # same pending challenge. We never raise inside the transaction once
+        # we've started mutating the pending row (would rollback the mark);
+        # instead we capture `failure` and raise after commit.
+        async with conn.transaction():
+            pending = await conn.fetchrow(
+                """
+                SELECT id, user_id, attempts, expires_at, used_at
+                  FROM mfa_pending_logins
+                 WHERE challenge_token = $1
+                 FOR UPDATE
+                """,
+                payload.mfa_token,
+            )
+
+            if not pending:
+                failure = {
+                    'user_id': None,
+                    'attempts': 0,
+                    'invalidated': True,
+                    'reason': 'challenge_not_found',
+                    'http_status': 410,
+                    'detail': 'MFA challenge not found or expired',
+                }
+            elif pending['used_at'] is not None:
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_already_used',
+                    'http_status': 410,
+                    'detail': 'MFA challenge already used',
+                }
+            elif pending['expires_at'] and pending['expires_at'] < datetime.utcnow():
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_expired',
+                    'http_status': 410,
+                    'detail': 'MFA challenge expired',
+                }
+            elif pending['attempts'] >= MFA_PENDING_MAX_ATTEMPTS:
+                await conn.execute(
+                    "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                    pending['id'],
+                )
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'too_many_attempts_pre_check',
+                    'http_status': 410,
+                    'detail': 'MFA challenge invalidated (too many attempts)',
+                }
+
+            if failure is None:
+                # Lock the user row so the atomic TOTP-step bump cannot race a
+                # parallel verify on a different pending challenge for the
+                # same account.
+                user_row = await conn.fetchrow(
+                    """
+                    SELECT id, username, email, role, is_active,
+                           created_at, updated_at, last_login_at,
+                           mfa_secret_encrypted, mfa_last_used_totp_step
+                      FROM users
+                     WHERE id = $1
+                     FOR UPDATE
+                    """,
+                    pending['user_id'],
+                )
+                if not user_row or not user_row['is_active']:
+                    failure = {
+                        'user_id': pending['user_id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': False,
+                        'reason': 'user_unavailable',
+                        'http_status': 401,
+                        'detail': 'User not available',
+                    }
+                elif not user_row['mfa_secret_encrypted']:
+                    failure = {
+                        'user_id': user_row['id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': True,
+                        'reason': 'mfa_not_configured',
+                        'http_status': 410,
+                        'detail': 'MFA not configured for this user',
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(user_row['mfa_secret_encrypted'])
+                    verified_method: Optional[str] = None
+                    codes_remaining: Optional[int] = None
+
+                    if secret_plain:
+                        ok, step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, user_row['mfa_last_used_totp_step']
+                        )
+                        if ok:
+                            # Atomic step bump — refuse if another request already
+                            # consumed this (or a newer) TOTP step.
+                            bumped = await conn.fetchval(
+                                """
+                                UPDATE users
+                                   SET mfa_last_used_totp_step = $1,
+                                       mfa_last_used_at = NOW()
+                                 WHERE id = $2
+                                   AND (mfa_last_used_totp_step IS NULL
+                                        OR mfa_last_used_totp_step < $1)
+                                 RETURNING id
+                                """,
+                                step,
+                                user_row['id'],
+                            )
+                            if bumped:
+                                verified_method = 'totp'
+
+                    if verified_method is None:
+                        # Backup codes — atomic single-use consumption.
+                        rows = await conn.fetch(
+                            """
+                            SELECT id, code_hash FROM mfa_backup_codes
+                             WHERE user_id = $1 AND used_at IS NULL
+                            """,
+                            user_row['id'],
+                        )
+                        for row in rows:
+                            if await mfa_service.check_backup_code(payload.code, row['code_hash']):
+                                consumed_id = await conn.fetchval(
+                                    """
+                                    UPDATE mfa_backup_codes
+                                       SET used_at = NOW()
+                                     WHERE id = $1 AND used_at IS NULL
+                                     RETURNING id
+                                    """,
+                                    row['id'],
+                                )
+                                if consumed_id:
+                                    verified_method = 'backup'
+                                    codes_remaining = await conn.fetchval(
+                                        "SELECT COUNT(*) FROM mfa_backup_codes WHERE user_id = $1 AND used_at IS NULL",
+                                        user_row['id'],
+                                    )
+                                break
+
+                    if verified_method is None:
+                        new_attempts = pending['attempts'] + 1
+                        invalidated = new_attempts >= MFA_PENDING_MAX_ATTEMPTS
+                        await conn.execute(
+                            """
+                            UPDATE mfa_pending_logins
+                               SET attempts = $1,
+                                   used_at = CASE WHEN $2 THEN NOW() ELSE used_at END
+                             WHERE id = $3
+                            """,
+                            new_attempts,
+                            invalidated,
+                            pending['id'],
+                        )
+                        failure = {
+                            'user_id': user_row['id'],
+                            'attempts': new_attempts,
+                            'invalidated': invalidated,
+                            'reason': 'invalid_code',
+                            'http_status': 410 if invalidated else 401,
+                            'detail': 'MFA challenge invalidated (too many attempts)'
+                            if invalidated else 'Invalid MFA code',
+                        }
+                    else:
+                        # Verified — finalize state inside the transaction so a
+                        # concurrent verify sees used_at on retry.
+                        await conn.execute(
+                            "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                            pending['id'],
+                        )
+                        if verified_method != 'totp':
+                            await conn.execute(
+                                "UPDATE users SET mfa_last_used_at = NOW() WHERE id = $1",
+                                user_row['id'],
+                            )
+                        try:
+                            await conn.execute(
+                                "UPDATE users SET last_login_at = CURRENT_TIMESTAMP WHERE id = $1",
+                                user_row['id'],
+                            )
+                        except Exception as exc:
+                            logger.warning(f"last_login_at update failed (continuing): {exc}")
+
+                        user_roles = await conn.fetch(
+                            """
+                            SELECT r.id, r.name, r.display_name, r.permissions
+                              FROM user_roles ur
+                              JOIN roles r ON ur.role_id = r.id
+                             WHERE ur.user_id = $1 AND ur.is_active = TRUE AND r.is_active = TRUE
+                            """,
+                            user_row['id'],
+                        )
+
+                        permissions: dict = {}
+                        roles_list: list = []
+                        for role_row in user_roles:
+                            roles_list.append({
+                                'id': role_row['id'],
+                                'name': role_row['name'],
+                                'display_name': role_row['display_name'],
+                            })
+                            role_permissions = role_row['permissions']
+                            if isinstance(role_permissions, str):
+                                import json
+                                role_permissions = json.loads(role_permissions)
+                            if role_permissions:
+                                for perm in role_permissions:
+                                    if '.' in perm:
+                                        resource, action = perm.split('.', 1)
+                                        if resource not in permissions:
+                                            permissions[resource] = {}
+                                        permissions[resource][action] = True
+
+                        success_payload = {
+                            'user': dict(user_row),
+                            'roles_list': roles_list,
+                            'permissions': permissions,
+                            'method': verified_method,
+                            'codes_remaining': codes_remaining,
+                        }
+
+        # ------------------------------------------------------------------
+        # Transaction has committed. Side-effects (JWT mint, audit log) below.
+        # ------------------------------------------------------------------
+        await close_database_connection(conn)
+        conn = None
+
+        if failure is not None:
+            if failure['user_id'] is not None:
+                await log_user_activity(
+                    user_id=failure['user_id'],
+                    action='mfa.login.failed',
+                    resource_type='mfa',
+                    resource_id=str(failure['user_id']),
+                    details={
+                        'reason': failure['reason'],
+                        'attempts': failure['attempts'],
+                        'invalidated': failure['invalidated'],
+                    },
+                    ip_address=ip_address,
+                    user_agent=user_agent,
+                )
+            raise HTTPException(status_code=failure['http_status'], detail=failure['detail'])
+
+        # Success path
+        assert success_payload is not None  # for type checkers; transaction guarantees this
+        user_row = success_payload['user']
+
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        token_payload = {
+            "user_id": user_row['id'],
+            "username": user_row['username'],
+            "email": user_row['email'],
+            "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+            "exp": datetime.utcnow() + timedelta(hours=24),
+        }
+        token = jwt.encode(token_payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+        await log_user_activity(
+            user_id=user_row['id'],
+            action='mfa.login.success',
+            resource_type='mfa',
+            resource_id=str(user_row['id']),
+            details={
+                'method': success_payload['method'],
+                'codes_remaining': success_payload['codes_remaining']
+                if success_payload['method'] == 'backup' else None,
+            },
+            ip_address=ip_address,
+            user_agent=user_agent,
+        )
+
+        return {
+            "access_token": token,
+            "token_type": "bearer",
+            "expires_in": 86400,
+            "user": {
+                "id": user_row['id'],
+                "username": user_row['username'],
+                "email": user_row['email'],
+                "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+                "is_active": user_row['is_active'],
+                "created_at": user_row['created_at'].isoformat() if user_row.get('created_at') else None,
+                "last_login_at": datetime.utcnow().isoformat(),
+            },
+            "roles": success_payload['roles_list'],
+            "permissions": success_payload['permissions'],
+        }
+
+    except HTTPException:
+        raise
+    except Exception as exc:
+        logger.error(f"MFA verify error: {exc}")
+        raise HTTPException(status_code=500, detail="MFA verification failed")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
 @router.post("/logout", summary="User Logout", response_description="Logout confirmation")
 async def logout(request: Request, authorization: str = Header(None)):
     """

backend/routers/backend.py

@@ -333,31 +333,38 @@ async def get_backends(
                 """)
         
         result = []
+        # Issue #24: servers must honor include_inactive exactly like the backend
+        # queries above. Pre-fix these sub-queries hardcoded `is_active = TRUE`
+        # (added in f34a6ee to hide soft-deleted entities), so a server toggled
+        # OFF (is_active=false) vanished from the UI with no way to reactivate it.
+        # Default callers (include_inactive=false) keep the is_active filter →
+        # byte-identical behavior; include_inactive=true now also returns inactive
+        # (disabled / soft-deleted) servers. last_config_status is selected so the
+        # frontend can tell DISABLED (re-enableable) from DELETION (pending delete).
+        server_active_filter = "" if include_inactive else "AND is_active = TRUE"
         for backend in backends:
-            # Get servers for this backend with cluster_id (ONLY show active servers)
-            # CRITICAL FIX: Add is_active = TRUE filter to prevent soft-deleted servers from appearing
             if cluster_id:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
+                    FROM backend_servers
-                    WHERE backend_name = $1 AND cluster_id = $2 AND is_active = TRUE ORDER BY server_name
+                    WHERE backend_name = $1 AND cluster_id = $2 {server_active_filter} ORDER BY server_name
                 """, backend["name"], cluster_id)
             else:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
+                    FROM backend_servers
-                    WHERE backend_name = $1 AND is_active = TRUE ORDER BY server_name
+                    WHERE backend_name = $1 {server_active_filter} ORDER BY server_name
-                """, backend["name"]) 
+                """, backend["name"])
             
             # Prepare server list with real-time HAProxy status from agents
             server_list = []
@@ -413,6 +420,7 @@ async def get_backends(
                     "fall": s.get("fall"),
                     "rise": s.get("rise"),
                     "is_active": s["is_active"],
+                    "last_config_status": s.get("last_config_status") or "APPLIED",  # Issue #24: lets UI distinguish DISABLED (re-enableable) from DELETION
                     "status": server_status,
                     "status_age_minutes": status_age_minutes,
                     "last_status_update": s.get("haproxy_status_updated_at").isoformat().replace('+00:00', 'Z') if s.get("haproxy_status_updated_at") else None,
@@ -1909,12 +1917,12 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
         conn = await get_database_connection()
         
-        # Get server info
+        # Get server info. Issue #24: fetch the FULL row (not just 5 columns) so we
+        # can snapshot the pre-toggle state for reject-rollback (see config version below).
         server = await conn.fetchrow("""
-            SELECT id, server_name, backend_name, is_active, cluster_id
+            SELECT * FROM backend_servers WHERE id = $1
-            FROM backend_servers WHERE id = $1
         """, server_id)
-        
+
         if not server:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail="Server not found")
@@ -1975,22 +1983,45 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
                 
                 # Generate new HAProxy config (after database commit)
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
-                
+
                 # Create new config version
                 config_hash = hashlib.sha256(config_content.encode()).hexdigest()
                 version_name = f"server-{server_id}-toggle-{int(time.time())}"
-                
+
                 # Get system admin user ID for created_by
                 conn2 = await get_database_connection()
                 admin_user_id = await conn2.fetchval("SELECT id FROM users WHERE username = 'admin' LIMIT 1") or 1
-                
+
+                # Issue #24: persist an entity snapshot so a Reject of this toggle
+                # rolls back is_active to its pre-toggle value. Pre-fix the toggle's
+                # config version carried NO metadata, so reject only reset
+                # last_config_status and the server stayed disabled (out of sync with
+                # the still-active live config). Mirrors the server-edit snapshot path;
+                # reject's rollback_entity_from_snapshot('server') restores is_active.
+                import json
+                from utils.entity_snapshot import save_entity_snapshot
+                entity_snapshot_metadata = await save_entity_snapshot(
+                    conn=conn2,
+                    entity_type="server",
+                    entity_id=server_id,
+                    old_values=dict(server),            # full pre-toggle row
+                    new_values={"is_active": new_status},
+                    operation="UPDATE",
+                )
+                old_config = await conn2.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                metadata = {"pre_apply_snapshot": old_config or "", **entity_snapshot_metadata}
+
                 # Create PENDING config version
                 config_version_id = await conn2.fetchval("""
-                    INSERT INTO config_versions 
+                    INSERT INTO config_versions
-                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
-                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING', $6)
                     RETURNING id
-                """, cluster_id, version_name, config_content, config_hash, admin_user_id)
+                """, cluster_id, version_name, config_content, config_hash, admin_user_id, json.dumps(metadata))
                 
                 logger.error(f"SERVER TOGGLE DEBUG: Created PENDING config version {version_name} for cluster {cluster_id}")
                 

backend/routers/cluster.py

@@ -292,12 +292,14 @@ async def create_cluster(cluster: HAProxyClusterCreate, authorization: str = Hea
         
         # Create cluster
         cluster_id = await conn.fetchval("""
-            INSERT INTO haproxy_clusters (name, description, connection_type, is_active, 
+            INSERT INTO haproxy_clusters (name, description, connection_type, is_active,
-                                        stats_socket_path, haproxy_config_path, haproxy_bin_path, pool_id)
+                                        stats_socket_path, haproxy_config_path, haproxy_bin_path,
-            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7)
+                                        keepalived_config_path, pool_id)
+            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8)
             RETURNING id
         """, cluster.name, cluster.description, cluster.connection_type,
-            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path, cluster.pool_id)
+            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path,
+            cluster.keepalived_config_path, cluster.pool_id)
         
         await close_database_connection(conn)
         
@@ -436,7 +438,12 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"haproxy_bin_path = ${param_counter}")
             update_values.append(cluster.haproxy_bin_path)
             param_counter += 1
-            
+
+        if cluster.keepalived_config_path is not None:
+            update_fields.append(f"keepalived_config_path = ${param_counter}")
+            update_values.append(cluster.keepalived_config_path)
+            param_counter += 1
+
         if cluster.pool_id is not None:
             update_fields.append(f"pool_id = ${param_counter}")
             update_values.append(cluster.pool_id)
@@ -517,7 +524,7 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
 
 
 @router.get("/{cluster_id}", summary="Get Cluster by ID", response_description="Cluster details")
-async def get_cluster(cluster_id: int, authorization: str = Header(None)):
+async def get_cluster(cluster_id: int, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get Specific HAProxy Cluster
     
@@ -528,8 +535,12 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters/1" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters/1" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -554,20 +565,30 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     - **404**: Cluster not found
     - **500**: Server error
     """
-    try:
+    # R18c audit fix (round 6 final convergence): authenticate the caller
-        # R18c audit fix (round 6 final convergence): authenticate
+    # before fetching cluster topology by ID. Pre-fix this sibling of
-        # the caller before fetching cluster topology by ID. Pre-fix
+    # GET /api/clusters was anonymous, so an attacker could iterate cluster
-        # this sibling of GET /api/clusters was anonymous, so an
+    # IDs to enumerate the same info (stats socket, paths, ACME flags, pool
-        # attacker could iterate cluster IDs to enumerate the same
+    # identity) the list endpoint just locked down.
-        # info (stats socket, paths, ACME flags, pool identity) the
+    # Issue #22: agents send their token in the X-API-Key header (not a user
-        # list endpoint just locked down. Closes the asymmetry.
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         cluster = await conn.fetchrow("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name
             FROM haproxy_clusters c
@@ -591,6 +612,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
             "stats_socket_path": cluster["stats_socket_path"],
             "haproxy_config_path": cluster["haproxy_config_path"],
             "haproxy_bin_path": cluster["haproxy_bin_path"],
+            "keepalived_config_path": cluster.get("keepalived_config_path"),
             "pool_id": cluster["pool_id"],
             "pool_name": cluster["pool_name"],
             "acme_enabled": cluster.get("acme_enabled", False),
@@ -602,7 +624,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
         raise HTTPException(status_code=500, detail=str(e))
 
 @router.get("", summary="Get All Clusters", response_description="List of all clusters")
-async def get_clusters(authorization: str = Header(None)):
+async def get_clusters(authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All HAProxy Clusters
     
@@ -610,8 +632,12 @@ async def get_clusters(authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -662,23 +688,34 @@ async def get_clusters(authorization: str = Header(None)):
     ## Error Responses
     - **500**: Server error
     """
-    try:
+    # R18c audit fix (round 6 #3 — KRITIK info leak): require an authenticated
-        # R18c audit fix (round 6 #3 — KRITIK info leak): require an
+    # caller. Pre-fix the endpoint accepted anonymous GETs and returned cluster
-        # authenticated caller. Pre-fix the endpoint accepted
+    # topology including internal HAProxy paths (stats socket, config path, bin
-        # anonymous GETs and returned cluster topology including
+    # path), pool ids, ACME flags, and agent counts. This is both reconnaissance
-        # internal HAProxy paths (stats socket, config path, bin
+    # for an attacker and the spine of the cluster-scoped RBAC the rest of the
-        # path), pool ids, ACME flags, and agent counts. This is
+    # platform builds on, so guarding it at the read layer is essential.
-        # both reconnaissance for an attacker and the spine of the
+    # Issue #22: agents send their token in the X-API-Key header (not a user
-        # cluster-scoped RBAC the rest of the platform builds on,
+    # JWT), so accept either credential — mirrors the dual-auth on
-        # so guarding it at the read layer is essential after R18c
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
-        # round 5's roster + role guards.
+    # (Guard kept OUTSIDE the try below: get_clusters' broad `except Exception`
+    # re-wraps raised HTTPExceptions into 500, which produced the "500 - 401"
+    # in the issue log; raising here yields a clean 401.)
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         clusters = await conn.fetch("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name,
                    COALESCE(agent_counts.total_agents, 0) as total_agents,
@@ -737,6 +774,7 @@ async def get_clusters(authorization: str = Header(None)):
                 "stats_socket_path": cluster["stats_socket_path"],
                 "haproxy_config_path": cluster["haproxy_config_path"],
                 "haproxy_bin_path": cluster["haproxy_bin_path"],
+                "keepalived_config_path": cluster.get("keepalived_config_path"),
                 "pool_id": cluster["pool_id"],
                 "pool_name": cluster["pool_name"],
                 "acme_enabled": cluster.get("acme_enabled", False),
@@ -1313,6 +1351,8 @@ async def list_cluster_config_versions(cluster_id: int, authorization: str = Hea
                 version_type = "WAF Rule"
             elif "ssl-" in version['version_name']:
                 version_type = "SSL Certificate"
+            elif "vip-" in version['version_name']:
+                version_type = "HA / VIP"
             
             # Parse validation error if present
             validation_error = version.get("validation_error")
@@ -1417,10 +1457,14 @@ async def apply_pending_changes(
         # Users will handle configuration completeness through the centralized Apply Management page
         
         # Get all pending config versions for this cluster
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP apply/reject endpoints
+        # (keepalived is not part of haproxy.cfg). Exclude them so a generic cluster apply
+        # from any entity page never marks a VIP version APPLIED without enacting it. This
+        # is a no-op for every non-VIP cluster (no vip-* rows exist).
         pending_versions = await conn.fetch("""
             SELECT id, version_name, created_at, config_content, checksum, metadata
-            FROM config_versions 
+            FROM config_versions
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
             ORDER BY created_at ASC
         """, cluster_id)
         
@@ -2539,6 +2583,92 @@ async def get_config_version_diff(cluster_id: int, version_id: int, authorizatio
                     }
                 }
         
+        # HA/VIP (Issue #27): vip-{id}-{action} versions show the generated keepalived.conf
+        # each member node will deploy as the change content (VRRP secret masked). Mirrors
+        # the ssl-* special case above so VIP uses the STANDARD View Change diff modal.
+        vip_match = re.search(r'vip-(\d+)-(create|update|delete)', current_version['version_name'])
+        if vip_match:
+            vip_id = int(vip_match.group(1))
+            vip_action = vip_match.group(2)
+            rendered, vip_meta = None, None
+            old_content = ""
+            try:
+                from routers.vip import render_vip_config_masked
+                rendered, vip_meta = await render_vip_config_masked(conn, vip_id)
+                # For a create/update diff, fetch the PREVIOUS applied vip-* config for this VIP
+                # (the last-deployed keepalived.conf) so an EDIT shows ONLY the changed lines
+                # instead of the whole config as "added". Both sides are already secret-masked.
+                if vip_action != "delete":
+                    prev_applied = await conn.fetchrow(
+                        "SELECT config_content FROM config_versions WHERE version_name LIKE $1 "
+                        "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1",
+                        f"vip-{vip_id}-%", cluster_id, current_version['id'])
+                    if prev_applied and prev_applied['config_content']:
+                        old_content = prev_applied['config_content']
+            except Exception as vip_err:
+                logger.warning(f"VIP DIFF: render failed for vip {vip_id}: {vip_err}")
+            await close_database_connection(conn)
+
+            changes = []
+            line_number = 1
+            if vip_action == "delete" or rendered is None:
+                title = (vip_meta or {}).get("name") or f"VIP {vip_id}"
+                for line in [f"# HA/VIP change: {title}",
+                             "# keepalived will be stopped and the virtual IP released on each member node."
+                             if vip_action == "delete" else
+                             "# (configuration is not available to render yet)"]:
+                    changes.append({"type": "context", "line": line, "line_number": line_number})
+                    line_number += 1
+                summary = {"added": 0, "removed": 1 if vip_action == "delete" else 0,
+                           "total_changes": 1 if vip_action == "delete" else 0}
+            else:
+                # Real line diff. Match the STANDARD haproxy diff format: the line is stored
+                # WITHOUT a +/- prefix (the UI adds it from `type` — the old `+ {line}` here
+                # caused the doubled "+ +"). A CREATE (no previous applied config) shows
+                # everything as added; an UPDATE shows ONLY the lines that actually changed.
+                import difflib
+                new_content = current_version['config_content'] or rendered or ""
+                added_count = 0
+                removed_count = 0
+                line_number = 0
+                if not old_content:
+                    for i, l in enumerate(new_content.split('\n')):
+                        changes.append({"type": "added", "line": l, "line_number": i + 1})
+                        added_count += 1
+                else:
+                    for dl in difflib.unified_diff(old_content.split('\n'), new_content.split('\n'),
+                                                   lineterm='', n=3):
+                        if dl.startswith('@@'):
+                            mm = re.search(r'@@ -(\d+),?\d* \+(\d+),?\d* @@', dl)
+                            if mm:
+                                line_number = int(mm.group(2))
+                            continue
+                        if dl.startswith('---') or dl.startswith('+++'):
+                            continue
+                        if dl.startswith('+'):
+                            changes.append({"type": "added", "line": dl[1:], "line_number": line_number})
+                            added_count += 1
+                            line_number += 1
+                        elif dl.startswith('-'):
+                            changes.append({"type": "removed", "line": dl[1:], "line_number": line_number})
+                            removed_count += 1
+                        elif dl.startswith(' '):
+                            changes.append({"type": "context", "line": dl[1:], "line_number": line_number})
+                            line_number += 1
+                summary = {"added": added_count, "removed": removed_count,
+                           "total_changes": added_count + removed_count}
+
+            return {
+                "current_version": {
+                    "id": current_version['id'],
+                    "version_name": current_version['version_name'],
+                    "created_at": current_version['created_at'].isoformat().replace('+00:00', 'Z')
+                },
+                "previous_version": None,
+                "changes": changes,
+                "summary": summary,
+            }
+
         # Check if current version has config content
         if not current_version['config_content']:
             # Special handling for restore versions - they might not have content yet
@@ -4226,6 +4356,20 @@ async def undo_reject_config_version(
                 ),
             )
 
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP entity, so undo is handled
+        # by the VIP router — it re-stages the rejected change as PENDING from the version's
+        # captured pending_state (reactivating the VIP if a rejected create soft-deleted it,
+        # or re-applying a rejected edit). Returns an error string if it can't (e.g. the
+        # name/address/VRID was reused since reject) -> surface a clean 409.
+        if version_name.startswith('vip-'):
+            from routers.vip import restore_vip_from_rejected_version
+            err = await restore_vip_from_rejected_version(conn, version_id)
+            await close_database_connection(conn)
+            if err:
+                raise HTTPException(status_code=409, detail=f"Cannot undo this VIP change — {err}.")
+            return {"message": "VIP change restored to PENDING — review and Apply it from Apply Management",
+                    "version_name": version_name}
+
         async with conn.transaction():
             # Mark the version as PENDING again
             await conn.execute("""
@@ -4902,9 +5046,17 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         await validate_user_cluster_access(current_user['id'], cluster_id, conn)
 
         # Get all pending config versions for this cluster (CRITICAL: Include metadata for rollback!)
+        # HA/VIP (Issue #27): exclude vip-* versions — they are rejected/reverted by the
+        # VIP reject endpoint (which restores keepalived state), not the generic rollback.
+        # ORDER BY created_at ASC: the rollback loop dedups per entity and keeps the FIRST-processed
+        # snapshot, so the OLDEST snapshot must win — its old_values hold the true pre-change state.
+        # Critical when one entity has multiple pending versions (e.g. cluster ACME enable->disable->enable):
+        # rolling back to the oldest restores the original acme_enabled. (Matches the apply SELECT, which
+        # already orders created_at ASC.)
         pending_versions = await conn.fetch("""
             SELECT id, version_name, metadata FROM config_versions
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
+            ORDER BY created_at ASC
         """, cluster_id)
         
         # CRITICAL FIX: Detect and clean orphan config versions
@@ -5116,11 +5268,12 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         )
         
         # Mark all pending config versions as REJECTED (don't delete them)
+        # HA/VIP (Issue #27): leave vip-* versions to the VIP reject endpoint.
         rejected_count = len(pending_versions)
         await conn.execute("""
-            UPDATE config_versions 
+            UPDATE config_versions
             SET status = 'REJECTED'
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
         """, cluster_id)
 
         # Update WAF rules status to APPLIED (rolled back)

backend/routers/frontend.py

@@ -198,12 +198,28 @@ def _enforce_routing_rule_contradictions(
     for label, rule in conflicts:
         sig = _rule_to_signature(rule)
         if sig and sig in grandfathered_signatures:
+            # Bulgu #83 (round-23 audit) — re-word the operator-
+            # facing warning. The pre-fix message led with
+            # "Grandfathered <label> entry contains a self-
+            # contradictory X !X condition that pre-dated this
+            # validation", which (a) is internal jargon the
+            # operator does not parse, and (b) implies the rule
+            # is OLD when in fact the only thing this branch
+            # knows is that the rule was NOT changed by the
+            # current edit. The operator may well have authored
+            # the rule one minute earlier. State that explicitly
+            # and include the verbatim rule body so the operator
+            # does not have to hunt through the ACL Builder
+            # cards to find the offender.
+            rule_text = rule if isinstance(rule, str) else sig[:160]
             warnings.append(
-                f"Grandfathered {label} entry contains a "
+                f"{label}: rule was not modified by this edit but "
-                f"self-contradictory `X !X` condition that pre-dated "
+                f"contains a self-contradictory `X !X` condition "
-                f"this validation. The rule never fires; fix it at "
+                f"(`X AND NOT X` is always false, so the rule never "
-                f"your convenience. (rule: "
+                f"fires and traffic falls through to "
-                f"{rule if isinstance(rule, str) else sig[:160]})"
+                f"`default_backend`). Your current edit was saved; "
+                f"fix the rule at your convenience. "
+                f"(rule: {rule_text})"
             )
         else:
             blocking.append((label, rule))

backend/routers/letsencrypt.py

@@ -1,6 +1,7 @@
 from fastapi import APIRouter, HTTPException, Header
-from pydantic import BaseModel, Field, field_validator
+from pydantic import BaseModel, Field, field_validator, model_validator
-from typing import Optional, List
+from typing import Optional, List, Dict
+import base64
 import json
 import logging
 import re
@@ -10,6 +11,40 @@ from datetime import datetime
 from database.connection import get_database_connection, close_database_connection
 from services.acme_service import acme_service
 from services.haproxy_config import generate_haproxy_config_for_cluster
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import encrypt_dns_credentials, decrypt_dns_credentials
+
+# Issue #35: DNS-01 challenge methods.
+_CHALLENGE_TYPES = ("http-01", "dns-01")
+
+
+async def _dns01_enabled() -> bool:
+    """Global kill-switch (system_settings acme.dns01_enabled, default False). Read via the ACME
+    settings dict so non-admins never need the admin-only /api/settings/acme endpoint."""
+    try:
+        settings = await acme_service._get_settings()
+        val = settings.get('dns01_enabled')
+        if isinstance(val, str):
+            return val.strip().lower() in ('1', 'true', 'yes', 'on')
+        return bool(val)
+    except Exception:
+        return False
+
+
+# Per-user sliding-window rate limit for the manual dns-confirm action (soft anti-abuse so a user
+# can't spam the CA via the confirm button). Per-process; sufficient for a manual UI action.
+_DNS_CONFIRM_RL: Dict[int, list] = {}
+_DNS_CONFIRM_LIMIT = 5
+_DNS_CONFIRM_WINDOW = 60.0
+
+
+async def _enforce_dns_confirm_rate_limit(user_id: int) -> None:
+    now = time.time()
+    bucket = [t for t in _DNS_CONFIRM_RL.get(user_id, []) if now - t < _DNS_CONFIRM_WINDOW]
+    if len(bucket) >= _DNS_CONFIRM_LIMIT:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded: dns-confirm allowed 5 requests per minute")
+    bucket.append(now)
+    _DNS_CONFIRM_RL[user_id] = bucket
 
 logger = logging.getLogger(__name__)
 
@@ -25,8 +60,55 @@ class AccountCreate(BaseModel):
     email: str
     directory_url: Optional[str] = None
     tos_agreed: bool = True
-    eab_kid: Optional[str] = None
+    # EAB (External Account Binding) for CAs that require it (ZeroSSL, Google). The KID is opaque
-    eab_hmac_key: Optional[str] = None
+    # (bound only); the HMAC key must be base64url so newAccount's _b64url_decode won't raise a
+    # cryptic binascii error (a common copy mistake is standard-base64 '+'/'/' vs urlsafe '-'/'_').
+    eab_kid: Optional[str] = Field(default=None, max_length=256)
+    eab_hmac_key: Optional[str] = Field(default=None, max_length=512)
+    # Issue #35: per-account default challenge method + DNS provider (for dns-01).
+    challenge_type: str = "http-01"
+    dns_provider: Optional[str] = None
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @field_validator('eab_hmac_key')
+    @classmethod
+    def _validate_eab_hmac_key(cls, v):
+        if not v:
+            return v
+        try:
+            base64.urlsafe_b64decode(v + '=' * (-len(v) % 4))
+        except Exception:
+            raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
+        return v
+
+    @model_validator(mode='after')
+    def _require_provider_for_dns01(self):
+        if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
+            raise ValueError("dns_provider is required when challenge_type is 'dns-01'")
+        return self
+
+
+class DnsCredentialsUpsert(BaseModel):
+    dns_provider: str = Field(..., min_length=1, max_length=50)
+    credentials: Dict[str, str] = Field(default_factory=dict)
+
+    @field_validator('credentials')
+    @classmethod
+    def _validate_credentials(cls, v):
+        if len(v) > 20:
+            raise ValueError("Too many credential fields")
+        for key, val in v.items():
+            if not isinstance(key, str) or not re.match(r'^[a-zA-Z0-9_]{1,50}$', key):
+                raise ValueError(f"Invalid credential field name: {key!r}")
+            if not isinstance(val, str) or len(val) > 4000:
+                raise ValueError(f"Credential value for {key!r} is missing or too long")
+        return v
 
 
 class CertificateRequest(BaseModel):
@@ -38,6 +120,8 @@ class CertificateRequest(BaseModel):
     account_id: Optional[int] = None
     cluster_ids: List[int] = Field(default_factory=list)
     auto_renew: bool = True
+    # Issue #35: optional override; when None the account's default method is used.
+    challenge_type: Optional[str] = None
 
     @field_validator('domains')
     @classmethod
@@ -54,7 +138,30 @@ class CertificateRequest(BaseModel):
             if not _DOMAIN_REGEX.match(d_norm):
                 raise ValueError(f"Invalid domain format: '{d}'")
             normalized.append(d_norm)
-        return normalized
+        # De-duplicate (case/whitespace variants normalize to the same value) while preserving order,
+        # so we don't submit a redundant SAN to the CA or render duplicate-keyed tags in the UI.
+        return list(dict.fromkeys(normalized))
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v is not None and v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @model_validator(mode='after')
+    def _wildcard_requires_dns01(self):
+        # Static cross-field guard: a wildcard SAN can ONLY be issued via dns-01 (the CA rejects
+        # wildcard over http-01). The runtime dns01_enabled gate + provider resolution happen in the
+        # endpoint (validators can't do async/DB). When challenge_type is None here, the effective
+        # method is resolved from the account in the endpoint, which re-checks this.
+        if any((d or '').startswith('*.') for d in (self.domains or [])):
+            # Only reject when the caller EXPLICITLY chose a non-dns-01 method. When challenge_type is
+            # None, the effective method is resolved from the account in the endpoint, which re-checks
+            # wildcard-requires-dns-01 — so account-default dns-01 inheritance still works for wildcards.
+            if self.challenge_type is not None and self.challenge_type != 'dns-01':
+                raise ValueError("Wildcard certificates require challenge_type 'dns-01'")
+        return self
 
 
 # --- Account management ---
@@ -77,7 +184,7 @@ async def list_accounts(authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         rows = await conn.fetch(
-            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at FROM letsencrypt_accounts ORDER BY id"
+            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at, challenge_type, dns_provider FROM letsencrypt_accounts ORDER BY id"
         )
         return [dict(r) for r in rows]
     finally:
@@ -107,16 +214,36 @@ async def create_account(body: AccountCreate, authorization: str = Header(None))
         eab_kid = body.eab_kid or settings.get('eab_kid', '') or None
         eab_hmac_key = body.eab_hmac_key or settings.get('eab_hmac_key', '') or None
 
+        # Issue #35: a dns-01 account must name a supported DNS provider.
+        if body.challenge_type == 'dns-01':
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((body.dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {body.dns_provider}")
+
         result = await acme_service.register_account(
             email=body.email,
             directory_url=directory_url,
             tos_agreed=body.tos_agreed,
             eab_kid=eab_kid,
             eab_hmac_key=eab_hmac_key,
+            challenge_type=body.challenge_type,
+            dns_provider=(body.dns_provider or None),
         )
         return result
+    except HTTPException:
+        # Preserve deliberate status codes (e.g. 409 DNS-01 disabled, 422 unsupported provider) —
+        # the broad except below would otherwise downgrade them all to 400.
+        raise
     except Exception as e:
         logger.error(f"ACME account registration failed: {e}")
+        # Humanize the common EAB-required failure (ZeroSSL/Google). The ACME error propagates as a
+        # string ("Account registration failed: {<dict>}"), so match the URN substring in str(e).
+        if 'externalaccountrequired' in str(e).lower():
+            raise HTTPException(status_code=400, detail=(
+                "This CA requires External Account Binding (EAB). Enter the EAB Key ID and HMAC Key "
+                "from your ZeroSSL/Google account and retry."
+            ))
         raise HTTPException(status_code=400, detail=str(e))
 
 
@@ -186,6 +313,147 @@ async def remove_account(account_id: int, authorization: str = Header(None)):
         await close_database_connection(conn)
 
 
+# --- Issue #35: DNS-01 providers + per-account DNS credentials ---
+
+@router.get("/dns-providers")
+async def get_dns_providers(authorization: str = Header(None)):
+    """List supported DNS providers + their credential-field schema (for the UI). Also returns the
+    global dns01_enabled gate so a non-admin cert UI can read it without the admin-only settings API.
+    Authenticated (any user); not admin-only."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    return {"dns01_enabled": await _dns01_enabled(), "providers": list_providers()}
+
+
+@router.get("/accounts/{account_id}/dns-credentials")
+async def get_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Masked metadata only — provider + which credential fields are set + updated_at. NEVER returns
+    the ciphertext or any plaintext token. Read-only for any authenticated user (matches list_accounts)."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            "SELECT dns_provider, credentials_encrypted, updated_at FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+            account_id,
+        )
+        if not row:
+            return {"configured": False, "dns_provider": None, "credential_fields_present": [], "updated_at": None}
+        present = []
+        decrypted = decrypt_dns_credentials(row["credentials_encrypted"])
+        if isinstance(decrypted, dict):
+            present = sorted(decrypted.keys())
+        return {
+            "configured": True,
+            "dns_provider": row["dns_provider"],
+            "credential_fields_present": present,
+            "updated_at": row["updated_at"],
+        }
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/accounts/{account_id}/dns-credentials")
+async def upsert_dns_credentials(account_id: int, body: DnsCredentialsUpsert, authorization: str = Header(None)):
+    """Store (encrypted) DNS provider credentials for an account. Admin-only. Verifies the
+    credentials against the provider BEFORE persisting; returns a sanitized result (never the token)."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    provider_name = body.dns_provider.strip()
+    if not is_supported(provider_name):
+        raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {provider_name}")
+
+    conn = await get_database_connection()
+    try:
+        exists = await conn.fetchval("SELECT 1 FROM letsencrypt_accounts WHERE id = $1", account_id)
+        if not exists:
+            raise HTTPException(status_code=404, detail="ACME account not found")
+
+        # Verify credentials synchronously; only persist on success. The detail is user-safe.
+        try:
+            provider = get_provider(provider_name, dict(body.credentials))
+            verify = await provider.verify_credentials()
+        except DnsProviderError as exc:
+            raise HTTPException(status_code=422, detail=str(exc))
+        except HTTPException:
+            raise
+        except Exception:
+            # Defensive: never let a provider-internal exception string (which could echo creds in a
+            # future provider) reach the client. Always a sanitized 422.
+            raise HTTPException(status_code=422, detail="DNS provider credential verification failed")
+        if not verify.get("ok"):
+            raise HTTPException(status_code=422, detail=verify.get("detail") or "DNS provider credential verification failed")
+
+        token = encrypt_dns_credentials(dict(body.credentials))
+        await conn.execute(
+            """INSERT INTO letsencrypt_account_dns_credentials (account_id, dns_provider, credentials_encrypted, updated_at)
+               VALUES ($1, $2, $3, NOW())
+               ON CONFLICT (account_id) DO UPDATE SET
+                   dns_provider = EXCLUDED.dns_provider,
+                   credentials_encrypted = EXCLUDED.credentials_encrypted,
+                   updated_at = NOW()""",
+            account_id, provider_name, token,
+        )
+        # Keep the account's provider selection in sync.
+        await conn.execute(
+            "UPDATE letsencrypt_accounts SET dns_provider = $1, updated_at = NOW() WHERE id = $2",
+            provider_name, account_id,
+        )
+        return {"ok": True, "dns_provider": provider_name, "detail": verify.get("detail", "Credentials stored.")}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/accounts/{account_id}/dns-credentials")
+async def delete_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Remove an account's stored DNS credentials. Admin-only."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    conn = await get_database_connection()
+    try:
+        await conn.execute("DELETE FROM letsencrypt_account_dns_credentials WHERE account_id = $1", account_id)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/orders/{order_id}/dns-confirm")
+async def confirm_dns_order(order_id: int, authorization: str = Header(None)):
+    """Manual DNS-01 only: the user asserts the TXT record is published; tell the CA to validate.
+    Requires ssl.create + a per-user rate limit; acts only on a dns-01 + manual + pending order."""
+    from auth_middleware import get_current_user_from_token, check_user_permission
+    current_user = await get_current_user_from_token(authorization)
+    has_perm = await check_user_permission(current_user['id'], 'ssl', 'create')
+    if not has_perm:
+        raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.create required")
+    await _enforce_dns_confirm_rate_limit(current_user['id'])
+
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+    finally:
+        await close_database_connection(conn)
+    if not order:
+        raise HTTPException(status_code=404, detail="Order not found")
+    if order['challenge_type'] != 'dns-01' or (order['dns_provider'] or 'manual') != 'manual':
+        raise HTTPException(status_code=409, detail="This order is not a manual DNS-01 order")
+    if order['status'] not in ('pending', 'processing'):
+        raise HTTPException(status_code=409, detail=f"Order is '{order['status']}' and cannot be confirmed")
+
+    from services.dns01_orchestrator import confirm_manual_dns01
+    await confirm_manual_dns01(order_id)
+    return {"ok": True, "message": "DNS-01 confirmation submitted; the CA will validate shortly."}
+
+
 # --- Certificate operations ---
 
 @router.post("/certificates")
@@ -222,32 +490,61 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
         logger.info(f"ACME: Using account_id={account_id} for certificate request")
 
         warnings = []
-        # Audit Tur 5 / Commit 8c: empty cluster_ids in UI means "global certificate".
+
-        # Resolve to all ACME-enabled active clusters; only fail if NONE exist.
+        # Issue #35: resolve the effective challenge method (request override, else account default).
+        conn_acct = await get_database_connection()
+        try:
+            acct = await conn_acct.fetchrow(
+                "SELECT challenge_type, dns_provider FROM letsencrypt_accounts WHERE id = $1", account_id
+            )
+        finally:
+            await close_database_connection(conn_acct)
+        effective_challenge = (body.challenge_type or (acct['challenge_type'] if acct else None) or 'http-01')
+        dns_provider = (acct['dns_provider'] if acct else None)
+        is_dns01 = (effective_challenge == 'dns-01')
+        has_wildcard = any(d.startswith('*.') for d in body.domains)
+
+        if is_dns01:
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail="The selected ACME account has no DNS provider configured for DNS-01.")
+            if (dns_provider or 'manual') == 'manual':
+                # Manual DNS-01 cannot be renewed unattended; auto-renew is forced off on the issued
+                # certificate (see _complete_certificate). Tell the requester so it isn't a surprise.
+                warnings.append(
+                    "Manual DNS-01 certificates cannot auto-renew unattended. Auto-renew will be disabled; "
+                    "re-publish the TXT record and request renewal before expiry."
+                )
+        elif has_wildcard:
+            raise HTTPException(status_code=422, detail="Wildcard certificates require a DNS-01 account.")
+
+        # Empty cluster_ids = "global certificate". For DNS-01 no ACME Challenge Routing / port 80 is
+        # needed, so resolve to ALL active clusters; http-01 still requires acme_enabled clusters.
         if not body.cluster_ids:
             conn_resolve = await get_database_connection()
             try:
-                acme_clusters_resolved = await conn_resolve.fetch(
+                if is_dns01:
-                    "SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
-                )
+                else:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE")
             finally:
                 await close_database_connection(conn_resolve)
-            if not acme_clusters_resolved:
+            if not resolved:
+                if is_dns01:
+                    raise HTTPException(status_code=422, detail="Cannot issue certificate: no active clusters configured.")
                 raise HTTPException(
                     status_code=422,
                     detail="Cannot issue certificate: no ACME-enabled clusters configured. "
                            "Enable ACME Challenge Routing on at least one cluster in Cluster Management, "
                            "Apply the configuration change, then retry."
                 )
-            body.cluster_ids = [c['id'] for c in acme_clusters_resolved]
+            body.cluster_ids = [c['id'] for c in resolved]
             warnings.append(
-                f"No clusters specified — applied to all ACME-enabled cluster(s) ({len(body.cluster_ids)})"
+                f"No clusters specified — applied to all {'active' if is_dns01 else 'ACME-enabled'} cluster(s) ({len(body.cluster_ids)})"
             )
-            logger.warning(
+        elif not is_dns01:
-                f"ACME: Empty cluster_ids → global cert fallback to {len(body.cluster_ids)} ACME-enabled cluster(s)"
+            # http-01 only: warn if no cluster has ACME Challenge Routing enabled.
-            )
-        else:
-            # Validate that referenced clusters exist + are ACME-enabled (warn-only).
             try:
                 conn_warn = await get_database_connection()
                 try:
@@ -255,7 +552,6 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
                         "SELECT COUNT(*) FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
                     )
                     if acme_clusters == 0:
-                        logger.warning("ACME: No clusters with ACME Challenge Routing enabled - certificate validation will likely fail")
                         warnings.append(
                             "No clusters have ACME Challenge Routing enabled. "
                             "Certificate validation will fail. Enable it in Cluster Management and Apply Changes first."
@@ -269,14 +565,30 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             account_id=account_id,
             domains=body.domains,
             cluster_ids=body.cluster_ids,
+            challenge_type=effective_challenge,
+            created_by=current_user['id'],
         )
 
-        challenges = await acme_service.respond_to_challenges(order['order_id'])
+        # Audit trail: record who requested the certificate + the method (esp. for DNS-01/wildcard,
+        # which has a wider blast radius than http-01). Never raises into the request path.
+        try:
+            from utils.activity_log import record_event
+            await record_event(
+                order['order_id'], "acme.order.requested",
+                message=f"Certificate requested ({effective_challenge}) by user {current_user['id']}",
+                details={"user_id": current_user['id'], "challenge_type": effective_challenge,
+                         "dns_provider": dns_provider, "domains": body.domains},
+            )
+        except Exception:
+            pass
+
+        # http-01 posts the challenge response immediately (token served continuously). dns-01 is
+        # driven by the orchestrator AFTER the TXT is published (manual waits for dns-confirm), so we
+        # must NOT respond here.
+        challenges = []
+        if not is_dns01:
+            challenges = await acme_service.respond_to_challenges(order['order_id'])
 
-        # Commit 5b: re-fetch the order's *current* status from DB. The status
-        # returned by create_order() reflects the moment of creation; after
-        # respond_to_challenges() the CA may have already advanced it (e.g. to
-        # 'processing'). Surfacing stale status leads UI to under-poll.
         conn_status = await get_database_connection()
         try:
             fresh_status = await conn_status.fetchval(
@@ -287,14 +599,23 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             await close_database_connection(conn_status)
         effective_status = fresh_status or order['status']
 
-        logger.info(f"ACME: Order {order['order_id']} created, {len(challenges)} challenge(s) posted, status={effective_status}")
+        if is_dns01:
+            msg = ("Order created. Publish the DNS TXT record shown for each domain, then confirm."
+                   if dns_provider == 'manual'
+                   else "Order created. The DNS TXT record(s) will be published automatically; waiting for CA validation.")
+        else:
+            msg = "Order created. ACME challenges have been posted. Waiting for CA validation."
+
+        logger.info(f"ACME: Order {order['order_id']} created ({effective_challenge}), status={effective_status}")
 
         return {
             "order_id": order['order_id'],
             "status": effective_status,
             "domains": body.domains,
+            "challenge_type": effective_challenge,
+            "dns_provider": dns_provider,
             "challenges": challenges,
-            "message": "Order created. ACME challenges have been posted. Waiting for CA validation.",
+            "message": msg,
             "warnings": warnings,
         }
     except HTTPException:
@@ -316,7 +637,8 @@ async def list_orders(authorization: str = Header(None)):
         rows = await conn.fetch("""
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.ssl_certificate_id, o.cluster_ids, o.error_detail,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             ORDER BY o.created_at DESC
@@ -345,7 +667,8 @@ async def get_order(order_id: int, authorization: str = Header(None)):
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.certificate_url, o.finalize_url, o.expires_at,
                    o.error_detail, o.ssl_certificate_id, o.cluster_ids,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             WHERE o.id = $1
@@ -353,13 +676,23 @@ async def get_order(order_id: int, authorization: str = Header(None)):
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
 
+        # Issue #35: include challenge_type + dns_txt_value (PUBLIC DNS data — NOT key_authorization,
+        # NOT the API token) so the UI can render manual DNS-01 instructions. Explicit column list.
         challenges = await conn.fetch(
-            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
+            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at, "
+            "challenge_type, dns_txt_value FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
         )
         result = dict(order)
         result['domains'] = json.loads(result['domains']) if isinstance(result['domains'], str) else result['domains']
         result['cluster_ids'] = json.loads(result['cluster_ids']) if isinstance(result['cluster_ids'], str) else result['cluster_ids']
-        result['challenges'] = [dict(c) for c in challenges]
+        ch_list = []
+        for c in challenges:
+            cd = dict(c)
+            if cd.get('challenge_type') == 'dns-01':
+                # Server computes the record name (wildcard *.-stripping lives server-side).
+                cd['dns_record_name'] = acme_service._challenge_dns_name(cd['domain'])
+            ch_list.append(cd)
+        result['challenges'] = ch_list
         return result
     finally:
         await close_database_connection(conn)
@@ -470,18 +803,23 @@ async def renew_order(order_id: int, authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         order = await conn.fetchrow(
-            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1", order_id
+            "SELECT account_id, domains, cluster_ids, challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
         )
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
         domains = json.loads(order['domains']) if isinstance(order['domains'], str) else order['domains']
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
+        challenge_type = order['challenge_type'] or 'http-01'
 
         new_order = await acme_service.create_order(
-            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids
+            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids,
+            challenge_type=challenge_type, created_by=current_user['id'],
         )
-        challenges = await acme_service.respond_to_challenges(new_order['order_id'])
+        # dns-01 is driven by the orchestrator after the TXT is published; only http-01 responds here.
-        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenges": challenges}
+        challenges = []
+        if challenge_type != 'dns-01':
+            challenges = await acme_service.respond_to_challenges(new_order['order_id'])
+        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenge_type": challenge_type, "challenges": challenges}
     finally:
         await close_database_connection(conn)
 
@@ -611,11 +949,17 @@ async def get_renewal_schedule(authorization: str = Header(None)):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
     conn = await get_database_connection()
     try:
+        # Issue #35: expose the challenge method/provider (via the originating order/account) so the
+        # UI can distinguish manual DNS-01 certs, which cannot auto-renew unattended. LEFT JOINs keep
+        # legacy certs (no order link / pre-DNS-01 columns) rendering as http-01.
         certs = await conn.fetch("""
-            SELECT id, name, primary_domain, expiry_date, auto_renew, days_until_expiry
+            SELECT c.id, c.name, c.primary_domain, c.expiry_date, c.auto_renew, c.days_until_expiry,
-            FROM ssl_certificates
+                   o.challenge_type, a.dns_provider
-            WHERE source = 'letsencrypt' AND is_active = TRUE
+            FROM ssl_certificates c
-            ORDER BY expiry_date ASC NULLS LAST
+            LEFT JOIN letsencrypt_orders o ON o.id = c.letsencrypt_order_id
+            LEFT JOIN letsencrypt_accounts a ON a.id = o.account_id
+            WHERE c.source = 'letsencrypt' AND c.is_active = TRUE
+            ORDER BY c.expiry_date ASC NULLS LAST
         """)
         return [dict(c) for c in certs]
     finally:
@@ -663,6 +1007,18 @@ async def _complete_certificate(order_id: int) -> dict:
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
         primary_domain = domains[0] if domains else 'unknown'
 
+        # Issue #35: a manual DNS-01 certificate cannot be auto-renewed unattended (the renewal task
+        # skips it — see main.py), so persist auto_renew=FALSE rather than storing a misleading
+        # "Enabled" that the user trusts while the cert silently expires. http-01 and automated
+        # DNS-01 (e.g. Cloudflare) keep auto_renew=TRUE, preserving existing behaviour.
+        auto_renew_value = True
+        if order.get('challenge_type') == 'dns-01':
+            acct_provider = await conn.fetchval(
+                "SELECT dns_provider FROM letsencrypt_accounts WHERE id = $1", order['account_id']
+            )
+            if (acct_provider or 'manual') == 'manual':
+                auto_renew_value = False
+
         # Commit 5g: guard against empty cert_private_key.
         # Inserting an SSL certificate row with an empty private_key would silently
         # produce an unusable certificate (HAProxy would fail to load on Apply, or
@@ -749,12 +1105,13 @@ async def _complete_certificate(order_id: int) -> dict:
                     certificate_content = $1, private_key_content = $2, chain_content = $3,
                     all_domains = $4::jsonb, expiry_date = $5, days_until_expiry = $6,
                     issuer = $7, fingerprint = $8, letsencrypt_order_id = $9,
-                    auto_renew = TRUE, is_active = TRUE, last_config_status = 'PENDING',
+                    auto_renew = $11, is_active = TRUE, last_config_status = 'PENDING',
                     updated_at = NOW()
                 WHERE id = $10
             """, cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id,
+                auto_renew_value)
             logger.info(f"ACME RENEWAL: Updated certificate {cert_id} for {primary_domain}, status=PENDING")
         else:
             cert_row = await conn.fetchrow("""
@@ -764,11 +1121,12 @@ async def _complete_certificate(order_id: int) -> dict:
                      usage_type, source, letsencrypt_order_id, auto_renew, is_active,
                      last_config_status)
                 VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10,
-                        'frontend', 'letsencrypt', $11, TRUE, TRUE, 'PENDING')
+                        'frontend', 'letsencrypt', $11, $12, TRUE, 'PENDING')
                 RETURNING id
             """, f"le-{primary_domain}", cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), primary_domain, all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id,
+                auto_renew_value)
             cert_id = cert_row['id']
 
         await conn.execute(
@@ -815,12 +1173,18 @@ async def _complete_certificate(order_id: int) -> dict:
             if mapped:
                 effective_cluster_ids = [r['cluster_id'] for r in mapped]
             else:
-                # Audit Tur 6 / Commit 5j: only fall back to ACME-enabled clusters.
+                # Audit Tur 6 / Commit 5j: http-01 only falls back to ACME-enabled clusters.
-                # Applying renewal to ACME-disabled clusters could re-introduce Issue #11
+                # Issue #35: a dns-01 cert needs NO challenge routing / port 80, so it can renew on
-                # patterns and risks deploying certs to clusters where they can't be renewed.
+                # any active cluster — resolve to all active clusters for dns-01.
-                all_clusters = await conn.fetch(
+                ch_type = await conn.fetchval(
-                    "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                    "SELECT challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
                 )
+                if ch_type == 'dns-01':
+                    all_clusters = await conn.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    all_clusters = await conn.fetch(
+                        "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                    )
                 effective_cluster_ids = [r['id'] for r in all_clusters]
             if effective_cluster_ids:
                 logger.info(f"ACME RENEWAL: Resolved {len(effective_cluster_ids)} cluster(s) for global cert {cert_id}")

backend/routers/mfa.py

@@ -0,0 +1,727 @@
+"""MFA router — Issue #18, v1.6.0.
+
+All endpoints are additive; nothing here breaks existing JWT or apply_service flows.
+Authentication is JWT-based (Bearer). Admin endpoints additionally require
+``users.is_admin == True`` (canonical super-admin flag).
+"""
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from middleware.mfa_rate_limit_key import mfa_rate_limit_key
+from middleware.mfa_rate_limits import MFA_LIMITS
+from middleware.rate_limiter import limiter
+from models.mfa import (
+    MfaAdminResetAllRequest,
+    MfaAdminResetRequest,
+    MfaDisableRequest,
+    MfaEnrollConfirmRequest,
+    MfaEnrollConfirmResponse,
+    MfaEnrollStartResponse,
+    MfaRegenerateBackupRequest,
+    MfaRegenerateBackupResponse,
+    MfaStatusResponse,
+)
+from services import mfa_service
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/api/mfa", tags=["MFA"])
+
+# Lifecycle constants (Plan section 5)
+PENDING_ENROLLMENT_TTL_SECONDS = 600  # 10 minutes — QR scan + verify window
+PENDING_ENROLLMENT_MAX_ATTEMPTS = 5
+
+
+# ---------------------------------------------------------------------------
+# Authentication helpers
+# ---------------------------------------------------------------------------
+
+
+async def _require_user(authorization: Optional[str]) -> dict:
+    user = await get_current_user_from_token(authorization)
+    if not user:
+        raise HTTPException(status_code=401, detail="Not authenticated")
+    return user
+
+
+async def _require_admin(authorization: Optional[str]) -> dict:
+    user = await _require_user(authorization)
+    if not user.get("is_admin"):
+        raise HTTPException(status_code=403, detail="Admin privileges required")
+    return user
+
+
+async def _cleanup_expired_pending_enrollments(conn) -> None:
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_enrollments WHERE expires_at < NOW()"
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-enrollment cleanup skipped: {exc}")
+
+
+# ---------------------------------------------------------------------------
+# Self status
+# ---------------------------------------------------------------------------
+
+
+@router.get(
+    "/status",
+    summary="MFA status for the authenticated user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_status(authorization: str = Header(None)):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            current["id"],
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            current["id"],
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+@router.get(
+    "/admin/status/{user_id}",
+    summary="Admin: MFA status of any user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_admin_status(user_id: int, authorization: str = Header(None)):
+    await _require_admin(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            user_id,
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Enrollment
+# ---------------------------------------------------------------------------
+
+
+@router.post(
+    "/enroll/start",
+    summary="Begin TOTP enrollment (returns secret + otpauth URI)",
+    response_model=MfaEnrollStartResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_start, key_func=mfa_rate_limit_key)
+async def mfa_enroll_start(request: Request, authorization: str = Header(None)):
+    current = await _require_user(authorization)
+
+    # Round 7 audit fix — REFUSE re-enrollment if the user is already MFA-on.
+    # Without this guard a stolen JWT could silently rotate the victim's TOTP
+    # secret + invalidate all their backup codes via /enroll/start ->
+    # /enroll/confirm (overwriting `users.mfa_secret_encrypted` and replacing
+    # `mfa_backup_codes`). To re-enroll, the user must first call /api/mfa/disable
+    # (which requires a fresh TOTP) or an admin must run /api/mfa/admin-reset.
+    secret_plain = mfa_service.generate_totp_secret()
+    secret_encrypted = mfa_service.encrypt_secret(secret_plain)
+
+    conn = await get_database_connection()
+    blocked = False
+    try:
+        # Single transaction with SELECT FOR UPDATE closes the TOCTOU window
+        # between the mfa_enabled check and the pending_enrollment upsert.
+        async with conn.transaction():
+            row = await conn.fetchrow(
+                "SELECT mfa_enabled FROM users WHERE id = $1 FOR UPDATE",
+                current["id"],
+            )
+            if row and row["mfa_enabled"]:
+                blocked = True
+            else:
+                await _cleanup_expired_pending_enrollments(conn)
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_enrollments
+                        (user_id, secret_encrypted, attempts, expires_at)
+                    VALUES ($1, $2, 0, NOW() + ($3 || ' seconds')::interval)
+                    ON CONFLICT (user_id) DO UPDATE
+                      SET secret_encrypted = EXCLUDED.secret_encrypted,
+                          attempts = 0,
+                          expires_at = EXCLUDED.expires_at,
+                          created_at = CURRENT_TIMESTAMP
+                    """,
+                    current["id"],
+                    secret_encrypted,
+                    str(PENDING_ENROLLMENT_TTL_SECONDS),
+                )
+    finally:
+        await close_database_connection(conn)
+
+    if blocked:
+        raise HTTPException(
+            status_code=400,
+            detail="MFA is already enabled. Disable it first (via /api/mfa/disable or admin reset) to re-enroll.",
+        )
+
+    hostname_hint = request.url.hostname if request.url else None
+    label = mfa_service.build_account_label(current["username"], hostname_hint)
+    otpauth_uri = mfa_service.build_otpauth_uri(label, secret_plain)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.started",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"secret_len": len(secret_plain)},  # NEVER log the secret itself
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+
+    return MfaEnrollStartResponse(
+        secret=secret_plain,
+        otpauth_uri=otpauth_uri,
+        expires_in=PENDING_ENROLLMENT_TTL_SECONDS,
+    )
+
+
+@router.post(
+    "/enroll/confirm",
+    summary="Confirm enrollment with a TOTP code; returns 10 backup codes once",
+    response_model=MfaEnrollConfirmResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_confirm, key_func=mfa_rate_limit_key)
+async def mfa_enroll_confirm(
+    payload: MfaEnrollConfirmRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # Pre-generate plain codes & hashes outside the DB transaction so we
+    # never hold a row lock for ~2-3s of bcrypt work.
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    # failure: { reason, attempts, http_status, detail }; success when None.
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            # Lock the pending row so concurrent /enroll/confirm calls for the
+            # same user can't both consume the same pending enrollment.
+            pending = await conn.fetchrow(
+                """
+                SELECT secret_encrypted, attempts, expires_at
+                  FROM mfa_pending_enrollments
+                 WHERE user_id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+
+            if not pending:
+                failure = {
+                    "reason": "no_pending",
+                    "http_status": 410,
+                    "detail": "No pending enrollment; start again",
+                }
+            else:
+                from datetime import datetime as _dt
+                if pending["expires_at"] and pending["expires_at"] < _dt.utcnow():
+                    await conn.execute(
+                        "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                        current["id"],
+                    )
+                    failure = {
+                        "reason": "expired",
+                        "http_status": 410,
+                        "detail": "Enrollment expired; start again",
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(pending["secret_encrypted"])
+                    if not secret_plain:
+                        await conn.execute(
+                            "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                            current["id"],
+                        )
+                        failure = {
+                            "reason": "unreadable",
+                            "http_status": 500,
+                            "detail": "Pending enrollment unreadable; start again",
+                        }
+                    else:
+                        ok, _step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, None
+                        )
+                        if not ok:
+                            new_attempts = (pending["attempts"] or 0) + 1
+                            if new_attempts >= PENDING_ENROLLMENT_MAX_ATTEMPTS:
+                                await conn.execute(
+                                    "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "too_many_attempts",
+                                    "attempts": new_attempts,
+                                    "http_status": 410,
+                                    "detail": "Enrollment invalidated; start again",
+                                }
+                            else:
+                                await conn.execute(
+                                    "UPDATE mfa_pending_enrollments SET attempts = $1 WHERE user_id = $2",
+                                    new_attempts,
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "invalid_code",
+                                    "attempts": new_attempts,
+                                    "http_status": 401,
+                                    "detail": "Invalid code",
+                                }
+                        else:
+                            # Verified — finalize state inside the transaction.
+                            await conn.execute(
+                                """
+                                UPDATE users
+                                   SET mfa_enabled = TRUE,
+                                       mfa_method = 'totp',
+                                       mfa_secret_encrypted = $1,
+                                       mfa_enrolled_at = NOW(),
+                                       mfa_last_used_totp_step = NULL,
+                                       mfa_last_used_at = NULL
+                                 WHERE id = $2
+                                """,
+                                pending["secret_encrypted"],
+                                current["id"],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                current["id"],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        # Log AFTER commit so the audit row reflects what actually persisted.
+        await log_user_activity(
+            user_id=current["id"],
+            action="mfa.enrollment.failed",
+            resource_type="mfa",
+            resource_id=str(current["id"]),
+            details={
+                "reason": failure["reason"],
+                "attempts": failure.get("attempts"),
+            },
+            ip_address=ip,
+            user_agent=ua,
+        )
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.confirmed",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"method": "totp"},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaEnrollConfirmResponse(enabled=True, backup_codes=plain_codes, method="totp")
+
+
+# ---------------------------------------------------------------------------
+# Disable + regenerate
+# ---------------------------------------------------------------------------
+
+
+async def _verify_user_code(conn, user_row: dict, code: str) -> Optional[str]:
+    """Verify a TOTP-or-backup code against the user's stored secret.
+
+    Returns the method used ('totp' / 'backup') on success, None on failure.
+
+    On TOTP success the step counter is bumped *atomically* — the UPDATE
+    only succeeds if no other request consumed the same (or a newer) step
+    in between. On backup success the consumed row's used_at is set with
+    an atomic ``WHERE used_at IS NULL RETURNING id`` pattern.
+    """
+    secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+    if secret_plain:
+        ok, step = mfa_service.verify_totp_with_replay_guard(
+            secret_plain, code, user_row["mfa_last_used_totp_step"]
+        )
+        if ok:
+            bumped = await conn.fetchval(
+                """
+                UPDATE users
+                   SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                 WHERE id = $2
+                   AND (mfa_last_used_totp_step IS NULL
+                        OR mfa_last_used_totp_step < $1)
+                 RETURNING id
+                """,
+                step,
+                user_row["id"],
+            )
+            if bumped:
+                return "totp"
+
+    rows = await conn.fetch(
+        """
+        SELECT id, code_hash FROM mfa_backup_codes
+         WHERE user_id = $1 AND used_at IS NULL
+        """,
+        user_row["id"],
+    )
+    for row in rows:
+        if await mfa_service.check_backup_code(code, row["code_hash"]):
+            consumed = await conn.fetchval(
+                """
+                UPDATE mfa_backup_codes
+                   SET used_at = NOW()
+                 WHERE id = $1 AND used_at IS NULL
+                 RETURNING id
+                """,
+                row["id"],
+            )
+            if consumed:
+                return "backup"
+    return None
+
+
+@router.post("/disable", summary="Disable MFA (requires current TOTP or backup)")
+@limiter.limit(MFA_LIMITS.disable, key_func=mfa_rate_limit_key)
+async def mfa_disable(
+    payload: MfaDisableRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        user_row = await conn.fetchrow(
+            """
+            SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users WHERE id = $1
+            """,
+            current["id"],
+        )
+        if not user_row or not user_row["mfa_enabled"]:
+            raise HTTPException(status_code=400, detail="MFA is not enabled")
+
+        method_used = await _verify_user_code(conn, dict(user_row), payload.code)
+        if not method_used:
+            await log_user_activity(
+                user_id=current["id"],
+                action="mfa.disable.failed",
+                resource_type="mfa",
+                resource_id=str(current["id"]),
+                details={"reason": "invalid_code"},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get("user-agent"),
+            )
+            raise HTTPException(status_code=401, detail="Invalid code")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                current["id"],
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", current["id"]
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.disabled.self",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"verified_via": method_used},
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"enabled": False}
+
+
+@router.post(
+    "/backup-codes/regenerate",
+    summary="Issue 10 fresh backup codes (TOTP required)",
+    response_model=MfaRegenerateBackupResponse,
+)
+@limiter.limit(MFA_LIMITS.regenerate_backup_codes, key_func=mfa_rate_limit_key)
+async def mfa_regenerate_backup_codes(
+    payload: MfaRegenerateBackupRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # bcrypt-hash the new codes outside the DB transaction (~2-3s of CPU work).
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            user_row = await conn.fetchrow(
+                """
+                SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+                  FROM users WHERE id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+            if not user_row or not user_row["mfa_enabled"]:
+                failure = {"http_status": 400, "detail": "MFA is not enabled"}
+            else:
+                secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+                if not secret_plain:
+                    failure = {"http_status": 500, "detail": "MFA secret unreadable"}
+                else:
+                    ok, step = mfa_service.verify_totp_with_replay_guard(
+                        secret_plain, payload.code, user_row["mfa_last_used_totp_step"]
+                    )
+                    if not ok:
+                        failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                    else:
+                        bumped = await conn.fetchval(
+                            """
+                            UPDATE users
+                               SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                             WHERE id = $2
+                               AND (mfa_last_used_totp_step IS NULL
+                                    OR mfa_last_used_totp_step < $1)
+                             RETURNING id
+                            """,
+                            step,
+                            current["id"],
+                        )
+                        if not bumped:
+                            failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                        else:
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.backup_codes.regenerated",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"codes_count": len(plain_codes)},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaRegenerateBackupResponse(backup_codes=plain_codes)
+
+
+# ---------------------------------------------------------------------------
+# Admin operations
+# ---------------------------------------------------------------------------
+
+
+@router.post("/admin-reset/{user_id}", summary="Admin: reset a single user's MFA")
+@limiter.limit(MFA_LIMITS.admin_reset, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset(
+    user_id: int,
+    payload: MfaAdminResetRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    if user_id == admin["id"]:
+        raise HTTPException(status_code=400, detail="Use /api/mfa/disable for self-reset")
+
+    conn = await get_database_connection()
+    try:
+        target = await conn.fetchrow(
+            "SELECT id, username, mfa_enabled FROM users WHERE id = $1", user_id
+        )
+        if not target:
+            raise HTTPException(status_code=404, detail="User not found")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                user_id,
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", user_id
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_reset",
+        resource_type="mfa",
+        resource_id=str(user_id),
+        details={
+            "target_user_id": user_id,
+            "target_username": target["username"],
+            "admin_user_id": admin["id"],
+            "reason": payload.reason,
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset": True, "user_id": user_id}
+
+
+@router.post(
+    "/admin-reset-all",
+    summary="Admin: emergency reset of MFA for all users (double confirm)",
+)
+@limiter.limit(MFA_LIMITS.admin_reset_all, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset_all(
+    payload: MfaAdminResetAllRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    # Pydantic's Literal already enforces the magic string, but check defensively too.
+    if payload.confirm != "RESET ALL MFA":
+        raise HTTPException(status_code=400, detail="Invalid confirmation string")
+
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            reset_count = await conn.fetchval(
+                """
+                WITH affected AS (
+                    UPDATE users
+                       SET mfa_enabled = FALSE,
+                           mfa_method = NULL,
+                           mfa_secret_encrypted = NULL,
+                           mfa_enrolled_at = NULL,
+                           mfa_last_used_at = NULL,
+                           mfa_last_used_totp_step = NULL
+                     WHERE mfa_enabled = TRUE
+                     RETURNING id
+                )
+                SELECT COUNT(*) FROM affected
+                """
+            )
+            await conn.execute("DELETE FROM mfa_backup_codes")
+            await conn.execute("DELETE FROM mfa_pending_logins")
+            await conn.execute("DELETE FROM mfa_pending_enrollments")
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_bulk_reset",
+        resource_type="mfa",
+        resource_id=str(admin["id"]),
+        details={
+            "reset_count": int(reset_count or 0),
+            "reason": payload.reason,
+            "admin_user_id": admin["id"],
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset_count": int(reset_count or 0)}

backend/routers/site_wizard.py

@@ -1046,9 +1046,45 @@ async def suggest_defaults(
 
         slug = "newhost"
         if domain:
-            base = domain.replace("*.", "").split(".")
+            # Bulgu #93 (round-24 audit) — IDN / Unicode safety. PRE-FIX
-            slug = (base[0] or "newhost")[:32].lower()
+            # the suggest endpoint used `c.isalnum()`, which is Unicode-
-            slug = "".join(c if (c.isalnum() or c in ("-", "_")) else "-" for c in slug)
+            # aware and returns True for non-ASCII letters (ü, é, ñ, …).
+            # An operator who typed `bücher.example.com` got back
+            # `backend_name="be-bücher"`, dropped that into the wizard
+            # form, and then hit a hard 422 at create time because the
+            # backend/frontend name validator regex
+            # `^[a-zA-Z][a-zA-Z0-9_-]{0,63}$` is ASCII-only. The wizard
+            # CREATE path also forces the domain itself through punycode
+            # (the validator rejects raw Unicode with a "use 'xn--…'"
+            # hint). Make `suggest` honour the same on-the-wire ASCII
+            # contract: convert each label to its IDN/punycode form
+            # FIRST, then sanitise to the alphanumeric / `-_` set the
+            # entity-name regex permits. The result is a name the
+            # operator can submit to /api/sites without re-typing.
+            first_label = (
+                domain.replace("*.", "").split(".")[0]
+                if domain.replace("*.", "")
+                else ""
+            )
+            ascii_label = first_label
+            if first_label and not first_label.isascii():
+                try:
+                    ascii_label = first_label.encode("idna").decode("ascii")
+                except (UnicodeError, UnicodeDecodeError):
+                    # IDN encoding failed (empty label, invalid chars,
+                    # etc.) — fall back to stripping non-ASCII to '-'
+                    # so we still produce a usable slug.
+                    ascii_label = "".join(
+                        c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                        else "-"
+                        for c in first_label
+                    )
+            slug = (ascii_label or "newhost")[:32].lower()
+            slug = "".join(
+                c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                else "-"
+                for c in slug
+            )
             if slug.startswith("_"):
                 slug = "h-" + slug.lstrip("_")
             if not slug or not slug[0].isalpha():
@@ -1146,6 +1182,53 @@ async def preview_create(
     try:
         await _validate_user_cluster_access(current_user["id"], body.cluster_id, conn)
 
+        # Bulgu #88 / #89 (round-24 audit) — cluster-RBAC parity with
+        # `create_site` for SSL certificate references. PRE-FIX the
+        # preview path (`POST /api/sites/preview`) skipped the
+        # `select_existing_cert()` gate that `create_site` runs at
+        # lines 2298-2308 (per-server CA bundle) and 2398-2405
+        # (HTTPS bind cert). The omission let an authenticated wizard
+        # user pass `ssl.mode='existing', ssl_certificate_id=<X>` (or
+        # `servers[i].ssl_certificate_id=<X>`) where cert `X` belongs
+        # to a DIFFERENT cluster and receive the full rendered
+        # `would_create` envelope back — leaking cert id metadata
+        # across tenant boundaries. The actual submit (`POST /api/
+        # sites`) does enforce the gate, so this is a preview-only
+        # information leak, not a write-path escalation. The fix is
+        # to mirror the same `select_existing_cert(conn, id, cluster_
+        # id)` predicate (which already encodes the "global OR
+        # junction-bound" rule defined in `ssl_service.py:331-370`)
+        # so the preview returns 400 with a clear hint rather than
+        # 200 with a leaked render. We deliberately keep the error
+        # phrasing identical to the create-time message so wizard
+        # UI handlers that already match on "not found / inactive"
+        # need no client changes.
+        if body.ssl.mode == "existing" and body.ssl.ssl_certificate_id:
+            _preview_resolved = await select_existing_cert(
+                conn, body.ssl.ssl_certificate_id, body.cluster_id,
+            )
+            if not _preview_resolved:
+                raise HTTPException(
+                    status_code=400,
+                    detail=(
+                        f"ssl_certificate_id {body.ssl.ssl_certificate_id} "
+                        "not found / inactive / not bound to this cluster"
+                    ),
+                )
+        for _idx, _srv in enumerate(body.servers or []):
+            _srv_cert_id = getattr(_srv, "ssl_certificate_id", None)
+            if _srv_cert_id:
+                if not await select_existing_cert(
+                    conn, _srv_cert_id, body.cluster_id,
+                ):
+                    raise HTTPException(
+                        status_code=400,
+                        detail=(
+                            f"servers[{_idx}].ssl_certificate_id={_srv_cert_id} "
+                            "not found / inactive / not bound to this cluster"
+                        ),
+                    )
+
         # Phase K Phase C: rate-limit ONLY the dry-run code path so
         # legacy callers (e.g. `SiteDrafts.handlePreview` which never
         # sets the flag) keep their unrestricted preview budget. The
@@ -1848,6 +1931,28 @@ async def create_site(
     try:
         await _validate_user_cluster_access(user_id, body.cluster_id, conn)
 
+        # Bulgu #87 (round-23 audit) — rate-limit the actual create
+        # endpoint. Pre-fix /api/sites/preview and
+        # /api/sites/preflight-acme were rate-limited (5/min, see
+        # `_enforce_rate_limit` callers above) but POST /api/sites
+        # itself had NO cap. Round-3 live testing fired 10 wizard
+        # creates in a single second against demo-cluster1; without
+        # the cap an authenticated user / script can spam-create
+        # entities until the cluster apply queue clogs.
+        #
+        # Action name must match what the router itself logs at the
+        # end of create_site (`wizard_create_site` — see
+        # `_log_user_activity(... action="wizard_create_site", ...)`
+        # at the success path below). The activity_logger middleware
+        # intentionally skips this endpoint to avoid double-logging
+        # (see middleware/activity_logger.py:74-110), so we must
+        # rate-limit on the SAME action the router emits.
+        #
+        # 5/min matches the preview / preflight budget — bulk
+        # operators have the dedicated /api/sites/bulk-import path
+        # for larger batches.
+        await _enforce_rate_limit(conn, user_id, "wizard_create_site")
+
         # ----- Pre-create checks (must succeed before transaction)
 
         # Phase 3 (R11-audit follow-up): reserved-name check parity with
@@ -2133,7 +2238,31 @@ async def create_site(
         # rebrand. The reject path on `cluster.py` recognises BOTH
         # prefixes so historical APPLIED versions (created before this
         # rename) keep behaving correctly during reject/undo.
-        version_name = f"bulk-site-create-{ts}"
+        #
+        # Bulgu #86 (round-23 audit) — append a short UUID suffix so
+        # two wizard POST /api/sites calls submitted within the SAME
+        # epoch second cannot collide on the
+        # `config_versions(cluster_id, version_name)` UNIQUE
+        # constraint. Pre-fix `version_name = f"bulk-site-create-{ts}"`
+        # gave seconds resolution, so an operator who clicked "Create"
+        # twice in rapid succession (or any back-to-back API
+        # automation) saw the SECOND call 409 with the generic
+        # `UniqueViolationError` fall-through message:
+        #
+        #     "A wizard entity with this name already exists on the
+        #      cluster (UNIQUE constraint). Pick a different name."
+        #
+        # — even though the operator-chosen backend / frontend / SSL
+        # names were unique. The actual collision was on the
+        # auto-generated `version_name` and re-naming the wizard
+        # inputs did NOT help. The reject_pending_changes path on
+        # cluster.py:4179 prefix-matches `bulk-site-create-*` so
+        # appending a unique suffix preserves the historical
+        # rollback / undo semantics. 6 hex chars give 16M-room before
+        # birthday collisions, vastly more than the per-second
+        # request volume an operator can sustain through the wizard.
+        import uuid
+        version_name = f"bulk-site-create-{ts}-{uuid.uuid4().hex[:6]}"
         bulk_snapshots: List[dict] = []
         created_ids: Dict[str, Any] = {}
 
@@ -2929,22 +3058,68 @@ async def create_site(
         # broke".
         msg = str(uve)
         logger.info(f"WIZARD: name conflict on create: {msg}")
-        if "backends_name_cluster_id_key" in msg or 'backends_name' in msg:
+        # Bulgu #85 (round-23 audit) — extract the offending constraint
+        # name from the asyncpg message so the operator-visible detail
+        # can pin-point WHICH entity collided. Pre-fix the handler only
+        # recognised `backends_*_key` and `frontends_*_key`; any other
+        # constraint (ssl_certificates, backend_servers, config_versions
+        # …) fell through to the generic "wizard entity with this name"
+        # message which is useless for debugging — the operator has to
+        # open the server log and the engineer has to ssh-bounce to
+        # extract `constraint=<name>` from the exception detail.
+        #
+        # ``asyncpg.UniqueViolationError.constraint_name`` is the
+        # canonical structured field; ``str(uve)`` only contains the
+        # human-formatted DETAIL line. Prefer the attribute, fall back
+        # to substring scanning so we stay robust if asyncpg ever stops
+        # exposing it.
+        constraint = getattr(uve, "constraint_name", None) or ""
+        msg_lower = msg.lower()
+        if "backends_name_cluster_id_key" in msg or 'backends_name' in msg \
+                or constraint == "backends_name_cluster_id_key":
             detail = (
-                "A backend with this name already exists on the cluster "
+                f"A backend named '{body.backend.name}' already exists on "
-                "(possibly soft-deleted). Pick a different backend name "
+                "the cluster (possibly soft-deleted). Pick a different "
-                "or restore/permanently-delete the existing row."
+                "backend name or restore/permanently-delete the existing "
+                "row."
             )
-        elif "frontends_name_cluster_id_key" in msg or "frontends_name" in msg:
+        elif "frontends_name_cluster_id_key" in msg or "frontends_name" in msg \
+                or constraint == "frontends_name_cluster_id_key":
             detail = (
-                "A frontend with this name already exists on the cluster "
+                f"A frontend named '{body.frontend.name}' (or its auto-"
+                f"derived HTTPS sibling) already exists on the cluster "
                 "(possibly soft-deleted). Pick a different frontend name "
                 "or restore/permanently-delete the existing row."
             )
-        else:
+        elif "ssl_certificates" in msg_lower or "ssl_certificates" in constraint \
+                or "ssl_cert" in constraint:
+            ssl_name = getattr(body.ssl, "name", None) or "(unnamed)"
             detail = (
-                "A wizard entity with this name already exists on the "
+                f"An SSL certificate named '{ssl_name}' already exists on "
-                "cluster (UNIQUE constraint). Pick a different name."
+                "the cluster (possibly soft-deleted). Pick a different "
+                "ssl.name or restore/permanently-delete the existing "
+                "certificate row."
+            )
+        elif "backend_servers" in msg_lower or "backend_servers" in constraint:
+            detail = (
+                "Two servers in the same backend share a server_name. "
+                "HAProxy requires `server <name>` tokens to be unique "
+                "within a backend block. Rename the duplicate(s) and "
+                "resubmit."
+            )
+        else:
+            # Echo the constraint name (a stable, non-secret schema
+            # identifier) in the detail so a human reading the toast
+            # can grep the codebase for the matching CREATE TABLE
+            # without needing server-log access. Names like
+            # `proxied_hosts_pkey` or `config_versions_unique` are
+            # safe to surface — they're public schema info.
+            con_hint = f" (constraint={constraint})" if constraint else ""
+            detail = (
+                f"A wizard entity with this name already exists on the "
+                f"cluster (UNIQUE constraint{con_hint}). Pick a different "
+                "name or retry; if the conflict persists contact the "
+                "platform team."
             )
         raise HTTPException(status_code=409, detail=detail)
     except Exception as e:

backend/routers/user.py

@@ -38,18 +38,19 @@ async def get_users(authorization: str = Header(None)):
         # Get users with their roles (only active users)
         try:
             users = await conn.fetch("""
-                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active, 
+                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active,
-                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at
+                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at,
+                       COALESCE(u.mfa_enabled, FALSE) AS mfa_enabled
                 FROM users u
                 WHERE u.is_active = TRUE
                 ORDER BY u.username
             """)
         except Exception as schema_error:
             logger.warning(f"Schema error in users query, using fallback: {schema_error}")
-            # Fallback query with minimal columns
+            # Fallback query with minimal columns (pre-MFA-migration deploys)
             users = await conn.fetch("""
                 SELECT id, username, email, is_active, is_admin, created_at
-                FROM users 
+                FROM users
                 WHERE is_active = TRUE
                 ORDER BY username
             """)

backend/routers/vip.py

@@ -0,0 +1,987 @@
+"""Issue #27 — HA/VIP (Keepalived) management API (v1.7.0).
+
+Isolated router: it owns vip_instances / vip_members only. It NEVER imports or calls
+the global HAProxy apply flow (cluster.py::apply_pending_changes) or the haproxy.cfg
+generator — VIP "Apply" is a standalone verb that renders per-node keepalived.conf
+snapshots into vip_members and flips the VIP to APPLIED.
+
+For Apply-Management consistency it ALSO stages a standard `config_versions` row per
+change (version_name `vip-{id}-{action}`, status PENDING, is_active=FALSE — exactly like
+ssl-* versions) so VIP changes appear in the right-panel Pending Versions list with the
+product's standard "View Change" diff. is_active stays FALSE so a VIP version can never
+be served to an agent as haproxy.cfg (the agent config query requires is_active=TRUE);
+cluster.py keeps vip-* versions out of the generic haproxy apply/reject (NOT LIKE 'vip-%').
+
+Every DB access is wrapped so a (pathological) missing vip_* relation degrades to an
+empty/None result instead of a 500 (B-7) — preserving the fleet-wide no-op guarantee.
+"""
+import hashlib
+import json
+import logging
+import re
+import time
+import uuid
+from typing import List, Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import check_user_permission, get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from models.vip import VIPCreate, VIPUpdate
+from services.keepalived_config import (
+    build_haproxy_check_script,
+    decrypt_vrrp_secret,
+    encrypt_vrrp_secret,
+    render_keepalived_conf,
+)
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/vip", tags=["HA / VIP"])
+
+
+def _client_ip(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.client.host if request and request.client else None
+    except Exception:  # noqa: BLE001
+        return None
+
+
+def _user_agent(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.headers.get("user-agent") if request else None
+    except Exception:  # noqa: BLE001
+        return None
+
+
+async def _require(authorization: Optional[str], action: str):
+    """Authenticate + enforce vip.<action>; returns current_user or raises 401/403."""
+    current_user = await get_current_user_from_token(authorization)
+    ok = await check_user_permission(current_user["id"], "vip", action, current_user=current_user)
+    if not ok:
+        raise HTTPException(status_code=403, detail=f"vip.{action} permission required")
+    return current_user
+
+
+async def _alloc_free_vrid(conn, pool_id: int, requested: Optional[int]) -> int:
+    """Use the requested VRID if free in the pool, else the lowest free 1..255."""
+    used = {r["virtual_router_id"] for r in await conn.fetch(
+        "SELECT virtual_router_id FROM vip_instances WHERE pool_id=$1 AND is_active=TRUE", pool_id)}
+    if requested is not None:
+        if requested in used:
+            raise HTTPException(status_code=409, detail=f"VRID {requested} already used in this pool")
+        return requested
+    for cand in range(1, 256):
+        if cand not in used:
+            return cand
+    raise HTTPException(status_code=409, detail="No free VRID (1-255) left in this pool")
+
+
+def _md5(text: str) -> str:
+    return hashlib.md5(text.encode("utf-8")).hexdigest()
+
+
+# The VRRP secret is rendered into keepalived.conf as `auth_pass <value>`. The "secret
+# never leaves the server in cleartext" rule means any config we hand back to the UI
+# (e.g. the version diff) must mask it — only the at-rest Fernet token and the
+# agent-delivery endpoint ever see the real value. Mask the WHOLE remainder of the line
+# (not just up to the first space) so a secret containing whitespace can't partially leak.
+_AUTH_PASS_RE = re.compile(r"(auth_pass\s+).*")
+
+
+def _redact_secret(conf: Optional[str]) -> str:
+    return _AUTH_PASS_RE.sub(r"\1********", conf or "")
+
+
+def _derive_deploy_status(last_config_status: str, members, pending_delete: bool = False,
+                          is_active: bool = True) -> tuple:
+    """Display status that reflects ACTUAL agent convergence, not just the staging flag.
+
+    Other entities only read APPLIED once their agents acknowledge the new config; the
+    VIP now mirrors that so the table never claims a VIP is live before its member nodes
+    have deployed keepalived and acked the current config hash (issue #27 follow-up).
+
+        PENDING_DELETE — a deletion is STAGED and awaiting approval in Apply Management; the
+                     VIP keeps running until approved (nothing is torn down)
+        DELETING   — deletion APPROVED; member nodes are tearing keepalived down (not yet acked)
+        PENDING    — staged (created/edited); apply from Apply Management
+        SYNCING    — applied; an ONLINE member is still converging (deploying/acking)
+        AWAITING   — applied, but the un-converged members' agents are all OFFLINE, so
+                     nothing can deploy yet (bring the node's agent online) — not a hang
+        ACTIVE     — applied AND every member deployed & acked the current config hash
+        ERROR      — a member reported a deploy error
+        ATTENTION  — a member found a hand-managed keepalived (externally_managed)
+
+    Returns (status, synced_count, total_count).
+    """
+    # Approval-gated delete: staged (still running) vs approved (tearing down).
+    if pending_delete:
+        return "PENDING_DELETE", 0, len(members)
+    if not is_active:
+        total = len(members)
+        torn = sum(1 for m in members if m["last_deploy_state"] == "disabled")
+        return ("DELETED" if total and torn == total else "DELETING"), torn, total
+    if last_config_status == "PENDING":
+        return "PENDING", 0, len(members)
+    total = len(members)
+    if total == 0:
+        return "APPLIED", 0, 0
+
+    def _in_sync(m) -> bool:
+        return (m["last_deploy_state"] == "enabled"
+                and m["applied_config_hash"] is not None
+                and m["last_deploy_hash"] == m["applied_config_hash"])
+
+    synced = sum(1 for m in members if _in_sync(m))
+    if any(m["last_deploy_state"] == "error" for m in members):
+        return "ERROR", synced, total
+    if any(m["last_deploy_state"] == "externally_managed" for m in members):
+        return "ATTENTION", synced, total
+    if synced == total:
+        return "ACTIVE", synced, total
+    # Not fully converged: distinguish "actively converging" (an online agent will deploy
+    # on its next poll) from "waiting on offline agents" (nothing will happen until the
+    # operator brings the node's agent online) — the latter must NOT read as a live spinner.
+    not_synced = [m for m in members if not _in_sync(m)]
+    if not_synced and all((m["agent_status"] or "offline") != "online" for m in not_synced):
+        return "AWAITING", synced, total
+    return "SYNCING", synced, total
+
+
+async def render_vip_config_masked(conn, vip_id: int):
+    """Render the keepalived.conf each member node would deploy, as ONE masked text block
+    (VRRP secret never in cleartext). Used by the standard config-version diff for vip-*
+    versions (cluster.py) and to populate the staged version's config_content — so VIP
+    changes show the product's standard "View Change" instead of a bespoke screen.
+
+    Returns (text, {"name": ...}) or (None, None) if the VIP is gone. Best-effort: a
+    per-node render error becomes an inline comment rather than raising.
+    """
+    v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1", vip_id)
+    if not v:
+        return None, None
+    members = await conn.fetch("""
+        SELECT m.agent_id, m.network_interface, m.role, m.priority,
+               a.name AS agent_name, a.ip_address
+        FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+        WHERE m.vip_id=$1 ORDER BY m.priority DESC
+    """, vip_id)
+    auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
+    vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
+                "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                "track_haproxy": v["track_haproxy"]}
+    member_dicts = [{"role": m["role"], "priority": m["priority"],
+                     "network_interface": m["network_interface"], "agent_id": m["agent_id"],
+                     "ip_address": str(m["ip_address"]) if m["ip_address"] else ""} for m in members]
+    blocks = []
+    for m in members:
+        this_agent = next(d for d in member_dicts if d["agent_id"] == m["agent_id"])
+        peer_ips = [d["ip_address"] for d in member_dicts
+                    if d["agent_id"] != m["agent_id"] and d["ip_address"]]
+        try:
+            conf = render_keepalived_conf(vip=vip_dict, members=member_dicts,
+                                          this_agent=this_agent, peer_ips=peer_ips,
+                                          auth_pass_plain=auth_plain)
+        except Exception as exc:  # noqa: BLE001 — best-effort preview
+            conf = f"# cannot render this node yet: {exc}\n"
+        header = (f"# ===== node: {m['agent_name'] or m['agent_id']} "
+                  f"({m['ip_address'] or 'no IP'}) — {m['role']} priority {m['priority']} =====")
+        blocks.append(header + "\n" + _redact_secret(conf))
+    text = "\n\n".join(blocks) if blocks else "# (no participating nodes selected yet)\n"
+    return text, {"name": v["name"]}
+
+
+async def _stage_vip_version(conn, vip_id: int, action: str, created_by: Optional[int]):
+    """Stage a standard PENDING config_versions row for a VIP change so it shows in Apply
+    Management exactly like other entities. One row per cluster in the VIP's pool;
+    is_active=FALSE so it is NEVER delivered as haproxy.cfg. Best-effort — a staging
+    failure never fails the VIP operation (logged); the VIP still applies via its own flow.
+    """
+    try:
+        cluster_ids = [r["id"] for r in await conn.fetch(
+            "SELECT id FROM haproxy_clusters WHERE pool_id = "
+            "(SELECT pool_id FROM vip_instances WHERE id=$1)", vip_id)]
+        if not cluster_ids:
+            return
+        content, _meta = await render_vip_config_masked(conn, vip_id)
+        content = content or f"# VIP {vip_id} ({action})\n"
+        checksum = _md5(content)
+        # uuid suffix makes the name collision-proof even for sub-second same-action re-edits
+        # (UNIQUE(cluster_id, version_name) would otherwise reject a same-second retry; review LOW-1).
+        version_name = f"vip-{vip_id}-{action}-{int(time.time())}-{uuid.uuid4().hex[:6]}"
+        # Capture this change's PENDING state so undo-reject can faithfully re-stage it
+        # (restore the VIP — reactivating it if a rejected create soft-deleted it).
+        pending_state = await _capture_vip_pending_state(conn, vip_id)
+        metadata = json.dumps({"pending_state": pending_state}) if pending_state else None
+        for cid in cluster_ids:
+            # Collapse repeated pre-apply edits to a single PENDING row per VIP+cluster.
+            await conn.execute(
+                "DELETE FROM config_versions WHERE cluster_id=$1 AND status='PENDING' "
+                "AND version_name LIKE $2", cid, f"vip-{vip_id}-%")
+            await conn.execute("""
+                INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                VALUES ($1,$2,$3,$4,$5,FALSE,'PENDING',$6::jsonb)
+            """, cid, version_name, content, checksum, created_by, metadata)
+    except Exception as e:  # noqa: BLE001 — versioning is a UI convenience, never block the op
+        logger.warning(f"_stage_vip_version({vip_id},{action}) failed: {e}")
+
+
+async def _capture_vip_pending_state(conn, vip_id: int):
+    """Snapshot a VIP's current (pending) field + member state for undo-reject. The VRRP
+    secret travels only as its encrypted token (never plaintext)."""
+    v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1", vip_id)
+    if not v:
+        return None
+    members = await conn.fetch(
+        "SELECT agent_id, network_interface, role, priority FROM vip_members WHERE vip_id=$1", vip_id)
+    return {
+        "vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
+                "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                "track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
+        "members": [{"agent_id": m["agent_id"], "network_interface": m["network_interface"],
+                     "role": m["role"], "priority": m["priority"]} for m in members],
+    }
+
+
+async def restore_vip_from_rejected_version(conn, version_id: int):
+    """Undo-reject for a vip-* version: re-stage the rejected change as PENDING from the
+    version's captured `pending_state`. Reactivates the VIP if a rejected create soft-deleted
+    it, or re-applies a rejected edit's members — preserving each remaining member's
+    last-applied snapshot so a running VIP is never torn down (T-1). Returns None on success
+    or a human-readable error string (e.g. the name/address/VRID was reused since reject).
+    """
+    row = await conn.fetchrow("SELECT version_name, metadata FROM config_versions WHERE id=$1", version_id)
+    if not row:
+        return "version not found"
+    m = re.match(r"vip-(\d+)-", row["version_name"] or "")
+    if not m:
+        return "not a VIP version"
+    vip_id = int(m.group(1))
+    # Undo the reject of a DELETE request → re-arm the staged deletion (the VIP keeps running
+    # until re-approved; nothing on the node changes). Requires the VIP to still be active.
+    if "-delete-" in (row["version_name"] or ""):
+        if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id):
+            return "the VIP is no longer active — nothing to re-stage for deletion"
+        await conn.execute(
+            "UPDATE vip_instances SET pending_delete=TRUE, last_config_status='PENDING', "
+            "updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+        await conn.execute(
+            "UPDATE config_versions SET status='PENDING', is_active=FALSE, updated_at=CURRENT_TIMESTAMP "
+            "WHERE version_name=$1 AND status='REJECTED'", row["version_name"])
+        return None
+    meta = row["metadata"]
+    meta = json.loads(meta) if isinstance(meta, str) else meta
+    ps = (meta or {}).get("pending_state")
+    if not ps:
+        return ("this VIP change predates undo support — re-create or re-edit the VIP "
+                "from the HA/VIP page")
+    if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1", vip_id):
+        return "the VIP no longer exists — re-create it from the HA/VIP page"
+    sv, sm = ps["vip"], ps.get("members", [])
+    # One-VIP-per-agent must STILL hold after reactivation: if a member was meanwhile added
+    # to another active VIP (while this one was rejected/soft-deleted), undoing here would
+    # create a double-membership — and since the delivery endpoint serves one VIP per agent,
+    # reactivating this (never-applied → not_configured) VIP would tear down the other live
+    # VIP on that node. Block it with a clear message (review round-3 FINDING 2).
+    member_ids = [m["agent_id"] for m in sm]
+    if member_ids:
+        dup = await conn.fetchrow(
+            "SELECT a.name AS agent_name, v.name AS vip_name FROM vip_members vm "
+            "JOIN vip_instances v ON v.id = vm.vip_id JOIN agents a ON a.id = vm.agent_id "
+            "WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE AND v.id <> $2 LIMIT 1",
+            member_ids, vip_id)
+        if dup:
+            return (f"node '{dup['agent_name']}' now belongs to active VIP '{dup['vip_name']}' — "
+                    f"remove it there first, then undo")
+    # Preserve the running (applied) snapshot for members that remain, so undo of an edit
+    # never momentarily flips a live node to not_configured (T-1).
+    prev = {r["agent_id"]: r for r in await conn.fetch(
+        "SELECT agent_id, applied_config_content, applied_config_hash, last_deploy_state, "
+        "last_deploy_message, last_deploy_hash, last_deploy_at FROM vip_members WHERE vip_id=$1", vip_id)}
+    try:
+        async with conn.transaction():
+            await conn.execute("""
+                UPDATE vip_instances SET name=$2, description=$3, virtual_ip=$4, prefix_length=$5,
+                    virtual_router_id=$6, advert_int=$7, use_unicast=$8, track_haproxy=$9,
+                    auth_pass_encrypted=$10, is_active=TRUE, last_config_status='PENDING',
+                    updated_at=CURRENT_TIMESTAMP
+                WHERE id=$1
+            """, vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
+                sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
+                sv.get("auth_pass_encrypted"))
+            await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+            for mm in sm:
+                o = prev.get(mm["agent_id"])
+                await conn.execute("""
+                    INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                        applied_config_content, applied_config_hash,
+                        last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at)
+                    VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
+                """, vip_id, mm["agent_id"], mm["network_interface"], mm["role"], mm["priority"],
+                    o["applied_config_content"] if o else None, o["applied_config_hash"] if o else None,
+                    o["last_deploy_state"] if o else None, o["last_deploy_message"] if o else None,
+                    o["last_deploy_hash"] if o else None, o["last_deploy_at"] if o else None)
+            # A multi-cluster pool stages one row per cluster under the SAME version_name;
+            # flip them all back to PENDING so every affected cluster's Apply Management shows
+            # the restored change (mirrors the SSL auto-undo). is_active stays FALSE (review MED-2).
+            await conn.execute(
+                "UPDATE config_versions SET status='PENDING', is_active=FALSE, "
+                "updated_at=CURRENT_TIMESTAMP WHERE version_name=$1 AND status='REJECTED'",
+                row["version_name"])
+    except Exception as e:  # noqa: BLE001
+        if "unique" in str(e).lower() or "duplicate" in str(e).lower():
+            return "the VIP's name, address or VRID was reused after it was rejected"
+        raise
+    return None
+
+
+async def _transition_vip_versions(conn, vip_id: int, new_status: str):
+    """Move this VIP's PENDING config_versions to APPLIED/REJECTED (is_active stays FALSE).
+    Called by the VIP apply/reject endpoints so the right-panel version tracks the VIP."""
+    try:
+        await conn.execute(
+            "UPDATE config_versions SET status=$2, is_active=FALSE, updated_at=CURRENT_TIMESTAMP "
+            "WHERE status='PENDING' AND version_name LIKE $1", f"vip-{vip_id}-%", new_status)
+    except Exception as e:  # noqa: BLE001
+        logger.warning(f"_transition_vip_versions({vip_id},{new_status}) failed: {e}")
+
+
+def _jsonb_list(v):
+    """agents.capabilities/network_interfaces are JSONB; asyncpg returns them as a raw
+    JSON string (no codec). Normalize to a Python list."""
+    if isinstance(v, list):
+        return v
+    if isinstance(v, str):
+        try:
+            parsed = json.loads(v)
+            return parsed if isinstance(parsed, list) else []
+        except Exception:  # noqa: BLE001
+            return []
+    return []
+
+
+def _capable(capabilities) -> bool:
+    return "keepalived_management" in _jsonb_list(capabilities)
+
+
+# ---------------------------------------------------------------------------
+# List / read
+# ---------------------------------------------------------------------------
+@router.get("")
+async def list_vips(cluster_id: Optional[int] = None, authorization: str = Header(None)):
+    """List VIPs with their members + live MASTER/BACKUP state (never the secret).
+
+    Optional cluster_id scopes to VIPs in that cluster's pool — used by the Apply
+    Management page (which is cluster-scoped) to surface pending VIP changes.
+    """
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        # Show active VIPs PLUS approved-but-still-tearing-down ones (is_active=FALSE with a
+        # member that hasn't acked 'disabled' yet) so the operator can TRACK a deletion through
+        # to completion; a VIP drops off only once every member has torn keepalived down.
+        # The teardown-tracking clause is gated on last_config_status='APPLIED' so it ONLY shows
+        # deletions APPROVED via the new flow — a VIP soft-deleted under the old immediate-delete
+        # (pre-1.7.2: is_active=FALSE, last_config_status='PENDING') is NOT resurfaced (backward
+        # compat). Rejected never-applied creates (also PENDING) are likewise excluded.
+        _visible = ("(v.is_active = TRUE OR (v.last_config_status = 'APPLIED' AND EXISTS ("
+                    "SELECT 1 FROM vip_members mm WHERE mm.vip_id = v.id "
+                    "AND mm.applied_config_hash IS NOT NULL "
+                    "AND mm.last_deploy_state IS DISTINCT FROM 'disabled')))")
+        if cluster_id is not None:
+            vips = await conn.fetch(f"""
+                SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                       v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                       v.is_active, v.last_config_status, v.pending_delete,
+                       (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                       v.created_at, v.updated_at, p.name AS pool_name
+                FROM vip_instances v
+                LEFT JOIN haproxy_cluster_pools p ON p.id = v.pool_id
+                WHERE {_visible}
+                  AND v.pool_id = (SELECT pool_id FROM haproxy_clusters WHERE id = $1)
+                ORDER BY v.name
+            """, cluster_id)
+        else:
+            vips = await conn.fetch(f"""
+                SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                       v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                       v.is_active, v.last_config_status, v.pending_delete,
+                       (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                       v.created_at, v.updated_at, p.name AS pool_name
+                FROM vip_instances v
+                LEFT JOIN haproxy_cluster_pools p ON p.id = v.pool_id
+                WHERE {_visible}
+                ORDER BY v.name
+            """)
+        result = []
+        for v in vips:
+            members = await conn.fetch("""
+                SELECT m.id, m.agent_id, m.network_interface, m.role, m.priority,
+                       m.applied_config_hash, m.last_deploy_state, m.last_deploy_hash, m.last_deploy_at,
+                       a.name AS agent_name, a.status AS agent_status,
+                       a.keepalive_state, a.keepalive_ip, a.capabilities,
+                       a.ip_address
+                FROM vip_members m
+                LEFT JOIN agents a ON a.id = m.agent_id
+                WHERE m.vip_id = $1
+                ORDER BY m.priority DESC
+            """, v["id"])
+            deploy_status, deploy_synced, deploy_total = _derive_deploy_status(
+                v["last_config_status"], members, v["pending_delete"], v["is_active"])
+            result.append({
+                **dict(v),
+                # Convergence-aware status for the table (issue #27 follow-up); the raw
+                # last_config_status is kept above for the PENDING gate / Apply button.
+                "deploy_status": deploy_status,
+                "deploy_synced": deploy_synced,
+                "deploy_total": deploy_total,
+                "members": [{
+                    "id": m["id"], "agent_id": m["agent_id"], "agent_name": m["agent_name"],
+                    "network_interface": m["network_interface"], "role": m["role"],
+                    "priority": m["priority"], "agent_status": m["agent_status"],
+                    "keepalive_state": m["keepalive_state"], "keepalive_ip": m["keepalive_ip"],
+                    "ip_address": str(m["ip_address"]) if m["ip_address"] else None,
+                    "keepalived_capable": _capable(m["capabilities"]),
+                    "last_deploy_state": m["last_deploy_state"],
+                    "last_deploy_at": m["last_deploy_at"].isoformat() if m["last_deploy_at"] else None,
+                } for m in members],
+            })
+        return {"vips": result}
+    except Exception as e:  # noqa: BLE001 — degrade to empty rather than 500 (B-7)
+        logger.error(f"list_vips failed: {e}")
+        return {"vips": []}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.get("/{vip_id}")
+async def get_vip(vip_id: int, authorization: str = Header(None)):
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("""
+            SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                   v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                   v.is_active, v.last_config_status,
+                   (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                   v.created_at, v.updated_at
+            FROM vip_instances v WHERE v.id = $1 AND v.is_active = TRUE
+        """, vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        members = await conn.fetch("""
+            SELECT m.agent_id, m.network_interface, m.role, m.priority,
+                   m.last_deploy_state, m.last_deploy_at,
+                   a.name AS agent_name, a.keepalive_state, a.keepalive_ip
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id = $1 ORDER BY m.priority DESC
+        """, vip_id)
+        return {**dict(v), "members": [dict(m) for m in members]}
+    finally:
+        await close_database_connection(conn)
+
+
+# ---------------------------------------------------------------------------
+# Create / update / delete
+# ---------------------------------------------------------------------------
+async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_virtual_ip: str,
+                                         exclude_vip_id: Optional[int] = None):
+    """Members must belong to the VIP's pool; VIP must not collide with an agent IP
+    or another live VIP (T-4/SQL-1)."""
+    # pool membership
+    for m in members:
+        row = await conn.fetchrow("SELECT pool_id FROM agents WHERE id=$1", m.agent_id)
+        if not row:
+            raise HTTPException(status_code=400, detail=f"agent {m.agent_id} not found")
+        if row["pool_id"] != pool_id:
+            raise HTTPException(status_code=400,
+                                detail=f"agent {m.agent_id} is not in pool {pool_id}")
+    # One active VIP per agent (v1): a node deploys a single keepalived.conf, and the
+    # delivery endpoint serves one VIP per agent — a second active membership would never
+    # converge (stuck SYNCING) instead of erroring. Reject it up front (review MED-2).
+    agent_ids = [m.agent_id for m in members]
+    if agent_ids:
+        dq = ("SELECT a.name AS agent_name, v.name AS vip_name "
+              "FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+              "JOIN agents a ON a.id = vm.agent_id "
+              "WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE")
+        dparams = [agent_ids]
+        if exclude_vip_id is not None:
+            dq += " AND v.id <> $2"
+            dparams.append(exclude_vip_id)
+        dup = await conn.fetchrow(dq + " LIMIT 1", *dparams)
+        if dup:
+            raise HTTPException(status_code=409,
+                                detail=f"node '{dup['agent_name']}' is already a member of VIP "
+                                       f"'{dup['vip_name']}' — a node can belong to only one VIP")
+    # VIP must not be an existing agent's primary IP (INET cast, SQL-1)
+    clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", vip_virtual_ip)
+    if clash:
+        raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already a node's IP")
+    # …or another live VIP's address
+    q = "SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE"
+    params = [vip_virtual_ip]
+    if exclude_vip_id is not None:
+        q += " AND id <> $2"
+        params.append(exclude_vip_id)
+    if await conn.fetchval(q, *params):
+        raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already used by another VIP")
+
+
+@router.post("")
+async def create_vip(payload: VIPCreate, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "create")
+    conn = await get_database_connection()
+    try:
+        pool = await conn.fetchrow("SELECT id FROM haproxy_cluster_pools WHERE id=$1", payload.pool_id)
+        if not pool:
+            raise HTTPException(status_code=400, detail=f"pool {payload.pool_id} not found")
+        await _validate_members_against_pool(conn, payload.pool_id, payload.members, payload.virtual_ip)
+
+        enc = encrypt_vrrp_secret(payload.auth_pass) if payload.auth_pass else None
+        # Allocate VRID + insert, retrying once on a unique-violation race (B-1).
+        last_err = None
+        for _attempt in range(2):
+            vrid = await _alloc_free_vrid(conn, payload.pool_id, payload.virtual_router_id)
+            try:
+                async with conn.transaction():
+                    vip_id = await conn.fetchval("""
+                        INSERT INTO vip_instances
+                          (name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
+                           advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
+                           is_active, last_config_status, created_by)
+                        VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11)
+                        RETURNING id
+                    """, payload.name, payload.description, payload.pool_id, payload.virtual_ip,
+                        payload.prefix_length, vrid, payload.advert_int, enc,
+                        payload.use_unicast, payload.track_haproxy, current_user["id"])
+                    for m in payload.members:
+                        await conn.execute("""
+                            INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority)
+                            VALUES ($1,$2,$3,$4,$5)
+                        """, vip_id, m.agent_id, m.network_interface, m.role, m.priority)
+                last_err = None
+                break
+            except Exception as ie:  # noqa: BLE001
+                if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                    last_err = ie
+                    if payload.virtual_router_id is not None:
+                        raise HTTPException(status_code=409, detail="VIP name/address/VRID already in use")
+                    continue  # auto-VRID race → retry allocation
+                raise
+        if last_err is not None:
+            raise HTTPException(status_code=409, detail="VIP create conflict (name/address/VRID)")
+
+        # Stage a standard PENDING config_version so the change shows in Apply Management.
+        await _stage_vip_version(conn, vip_id, "create", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="create", resource_type="vip",
+            resource_id=str(vip_id),
+            details={"name": payload.name, "virtual_ip": payload.virtual_ip,
+                     "pool_id": payload.pool_id, "vrid": vrid, "members": len(payload.members)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"id": vip_id, "message": "VIP created (PENDING — apply from Apply Management)"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/{vip_id}")
+async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "update")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        new_ip = payload.virtual_ip or v["virtual_ip"]
+        members = payload.members if payload.members is not None else None
+        if members is not None:
+            await _validate_members_against_pool(conn, v["pool_id"], members, new_ip, exclude_vip_id=vip_id)
+        elif payload.virtual_ip:
+            # Address-only change: re-check the VIP isn't a node IP or another live VIP.
+            clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address=$1::inet LIMIT 1", new_ip)
+            if clash:
+                raise HTTPException(status_code=409, detail=f"{new_ip} is already a node's IP")
+            dup = await conn.fetchval(
+                "SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE AND id<>$2", new_ip, vip_id)
+            if dup:
+                raise HTTPException(status_code=409, detail=f"{new_ip} is already used by another VIP")
+
+        enc_set = payload.auth_pass is not None and payload.auth_pass != ""
+        try:
+            async with conn.transaction():
+                await conn.execute("""
+                    UPDATE vip_instances SET
+                        name = COALESCE($2, name),
+                        description = COALESCE($3, description),
+                        virtual_ip = COALESCE($4, virtual_ip),
+                        prefix_length = COALESCE($5, prefix_length),
+                        virtual_router_id = COALESCE($6, virtual_router_id),
+                        advert_int = COALESCE($7, advert_int),
+                        use_unicast = COALESCE($8, use_unicast),
+                        track_haproxy = COALESCE($9, track_haproxy),
+                        auth_pass_encrypted = CASE WHEN $10 THEN $11 ELSE auth_pass_encrypted END,
+                        last_config_status = 'PENDING',
+                        -- Editing a VIP means you are KEEPING and changing it, so it cancels any
+                        -- staged deletion (otherwise a later Apply would delete instead of applying
+                        -- the edit). The edit then becomes the pending change to approve.
+                        pending_delete = FALSE,
+                        purge_on_teardown = FALSE,
+                        updated_at = CURRENT_TIMESTAMP
+                    WHERE id = $1
+                """, vip_id, payload.name, payload.description, payload.virtual_ip,
+                    payload.prefix_length, payload.virtual_router_id, payload.advert_int,
+                    payload.use_unicast, payload.track_haproxy,
+                    enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None)
+                if members is not None:
+                    # T-1: preserve the APPLIED snapshot + deploy ack for members that REMAIN,
+                    # so a pending member edit never momentarily flips a running node to
+                    # not_configured (which would self-heal-teardown a live VIP). The new
+                    # config only goes live on the next Apply. A REMOVED member loses its row
+                    # → delivery returns not_configured → marker self-heal teardown (correct:
+                    # it's no longer part of the VIP). A NEW member starts with a NULL snapshot.
+                    prev = {r["agent_id"]: r for r in await conn.fetch(
+                        "SELECT agent_id, applied_config_content, applied_config_hash, "
+                        "last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at "
+                        "FROM vip_members WHERE vip_id=$1", vip_id)}
+                    await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+                    for m in members:
+                        o = prev.get(m.agent_id)
+                        await conn.execute("""
+                            INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                                applied_config_content, applied_config_hash,
+                                last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at)
+                            VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
+                        """, vip_id, m.agent_id, m.network_interface, m.role, m.priority,
+                            o["applied_config_content"] if o else None,
+                            o["applied_config_hash"] if o else None,
+                            o["last_deploy_state"] if o else None,
+                            o["last_deploy_message"] if o else None,
+                            o["last_deploy_hash"] if o else None,
+                            o["last_deploy_at"] if o else None)
+        except HTTPException:
+            raise
+        except Exception as ie:  # noqa: BLE001
+            if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                raise HTTPException(status_code=409, detail="VIP name/address/VRID already in use")
+            raise
+        # Re-stage the standard PENDING config_version reflecting the edited config.
+        await _stage_vip_version(conn, vip_id, "update", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="update", resource_type="vip",
+            resource_id=str(vip_id), details={"vip_id": vip_id},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": "VIP updated (PENDING — apply from Apply Management)"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/{vip_id}")
+async def delete_vip(vip_id: int, request: Request, purge_package: bool = False,
+                     authorization: str = Header(None)):
+    """Request VIP deletion — APPROVAL-GATED for safety.
+
+    Deleting a *running* (already-applied) VIP does NOT take effect immediately: it is STAGED
+    for Apply Management (pending_delete=TRUE + a vip-*-delete version) and the VIP keeps
+    running — is_active stays TRUE, agents keep serving it, NOTHING is torn down — until the
+    operator APPROVES the deletion. Rejecting it leaves the VIP running, untouched. Only the
+    approval flips is_active=FALSE and lets the agents tear keepalived down. So a misclick can
+    never tear down a production VIP, and an agent never deletes without an explicit approval.
+
+    A VIP that was NEVER applied (not deployed to any node) is removed immediately — there is
+    nothing running to tear down. purge_package opts into uninstalling the keepalived package
+    on teardown (default keeps it) and is honoured only once the deletion is approved, and only
+    on nodes where WE installed it (the agent's install marker), never an admin's package.
+    """
+    current_user = await _require(authorization, "delete")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow(
+            "SELECT id, name, applied_snapshot FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+
+        if v["applied_snapshot"] is None:
+            # Never deployed to any node — removing it affects nothing, so do it at once.
+            await conn.execute(
+                "UPDATE vip_instances SET is_active=FALSE, last_config_status='PENDING', "
+                "pending_delete=FALSE, purge_on_teardown=$2, updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+                vip_id, bool(purge_package))
+            await conn.execute(
+                "DELETE FROM config_versions WHERE status='PENDING' AND version_name LIKE $1",
+                f"vip-{vip_id}-%")
+            await log_user_activity(
+                user_id=current_user["id"], action="delete", resource_type="vip",
+                resource_id=str(vip_id), details={"name": v["name"], "never_applied": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "VIP removed — it was never applied, so no node was affected.",
+                    "staged": False}
+
+        # Running VIP → STAGE the deletion for approval. is_active stays TRUE (no teardown yet);
+        # the agent keeps serving the VIP until the operator approves in Apply Management.
+        await conn.execute(
+            "UPDATE vip_instances SET pending_delete=TRUE, purge_on_teardown=$2, "
+            "last_config_status='PENDING', updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+            vip_id, bool(purge_package))
+        await _stage_vip_version(conn, vip_id, "delete", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="delete-requested", resource_type="vip",
+            resource_id=str(vip_id), details={"name": v["name"], "purge_package": bool(purge_package)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": ("Deletion staged for approval — the VIP keeps running until you APPROVE it "
+                            "in Apply Management; reject to keep it. Nothing changes on the node until "
+                            "you approve."),
+                "staged": True, "purge_package": bool(purge_package)}
+    finally:
+        await close_database_connection(conn)
+
+
+# ---------------------------------------------------------------------------
+# Apply (isolated) + status
+# ---------------------------------------------------------------------------
+@router.post("/{vip_id}/apply")
+async def apply_vip(vip_id: int, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "apply")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        # APPROVED DELETION: if a deletion was staged for this VIP, approving it here performs the
+        # actual delete — flip is_active=FALSE so the agents tear keepalived down on their next
+        # poll (honouring purge_on_teardown). Until this moment the VIP kept running untouched, so
+        # the teardown happens ONLY after this explicit human approval.
+        if v["pending_delete"]:
+            async with conn.transaction():
+                await conn.execute(
+                    "UPDATE vip_instances SET is_active=FALSE, pending_delete=FALSE, "
+                    "last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await _transition_vip_versions(conn, vip_id, "APPLIED")
+            await log_user_activity(
+                user_id=current_user["id"], action="delete", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "approved_delete": True,
+                         "purge_package": bool(v["purge_on_teardown"])},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            msg = "VIP deletion approved — member nodes will stop keepalived and release the VIP on their next poll"
+            if v["purge_on_teardown"]:
+                msg += "; the keepalived package will be uninstalled on nodes where we installed it"
+            return {"message": msg, "deleted": True}
+        members = await conn.fetch("""
+            SELECT m.id, m.agent_id, m.network_interface, m.role, m.priority, a.ip_address
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id = $1
+        """, vip_id)
+        if len(members) < 1:
+            raise HTTPException(status_code=400, detail="VIP needs at least 1 member")
+        masters = [m for m in members if m["role"] == "MASTER"]
+        if len(masters) != 1:
+            raise HTTPException(status_code=400, detail="exactly one member must be MASTER")
+        if any(m["ip_address"] is None for m in members):
+            raise HTTPException(status_code=400,
+                                detail="every member must have a reported IP before apply (unicast peers)")
+        master_prio = masters[0]["priority"]
+        if any(m["role"] == "BACKUP" and m["priority"] >= master_prio for m in members):
+            raise HTTPException(status_code=400, detail="MASTER priority must exceed every BACKUP")
+
+        auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
+        # If a secret is set but can't be decrypted (SECRET_KEY/VIP_ENCRYPTION_KEY rotated or
+        # drifted between pods), FAIL the apply rather than silently rendering a config with NO
+        # VRRP authentication — that would be a silent security downgrade and a guaranteed
+        # MASTER/BACKUP auth mismatch with any node still holding the old config (review MED-1).
+        if v["auth_pass_encrypted"] and not auth_plain:
+            raise HTTPException(status_code=409,
+                                detail="VRRP secret could not be decrypted (encryption key changed?) — "
+                                       "re-enter the VRRP secret on the VIP, then apply again")
+        check_script = build_haproxy_check_script() if v["track_haproxy"] else ""
+        member_dicts = [{"role": m["role"], "priority": m["priority"],
+                         "network_interface": m["network_interface"],
+                         "agent_id": m["agent_id"],
+                         "ip_address": str(m["ip_address"])} for m in members]
+        vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
+                    "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                    "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                    "track_haproxy": v["track_haproxy"]}
+
+        snapshot_members = []
+        async with conn.transaction():
+            for m in members:
+                this_agent = next(d for d in member_dicts if d["agent_id"] == m["agent_id"])
+                peer_ips = [d["ip_address"] for d in member_dicts if d["agent_id"] != m["agent_id"]]
+                conf = render_keepalived_conf(vip=vip_dict, members=member_dicts,
+                                              this_agent=this_agent, peer_ips=peer_ips,
+                                              auth_pass_plain=auth_plain)
+                chash = _md5(conf)
+                await conn.execute("""
+                    UPDATE vip_members
+                    SET applied_config_content=$2, applied_config_hash=$3, updated_at=CURRENT_TIMESTAMP
+                    WHERE id=$1
+                """, m["id"], conf, chash)
+                snapshot_members.append({
+                    "agent_id": m["agent_id"], "network_interface": m["network_interface"],
+                    "role": m["role"], "priority": m["priority"],
+                    "applied_config_content": conf, "applied_config_hash": chash})
+            # Capture the field-level applied state so a later pending edit can be REJECTED
+            # and fully reverted to exactly this state (auth secret stored encrypted, never plain).
+            applied_snapshot = {
+                "vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
+                        "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                        "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                        "track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
+                "members": snapshot_members}
+            await conn.execute(
+                "UPDATE vip_instances SET last_config_status='APPLIED', "
+                "applied_snapshot=$2::jsonb, updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+                vip_id, json.dumps(applied_snapshot))
+
+        # Move the standard config_version PENDING → APPLIED (stays is_active=FALSE so it is
+        # never served as haproxy.cfg). Keeps the right-panel version in lockstep with the VIP.
+        await _transition_vip_versions(conn, vip_id, "APPLIED")
+        await log_user_activity(
+            user_id=current_user["id"], action="apply", resource_type="vip",
+            resource_id=str(vip_id),
+            details={"name": v["name"], "virtual_ip": v["virtual_ip"], "members": len(members)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        # check_script is rendered but not stored on the vip row; the agent gets it via
+        # the delivery endpoint (which rebuilds it). Returned here only for visibility.
+        return {"message": "VIP applied — agents will converge on next poll",
+                "members_rendered": len(members), "tracks_haproxy": bool(check_script)}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/{vip_id}/reject")
+async def reject_vip(vip_id: int, request: Request, authorization: str = Header(None)):
+    """Discard a VIP's PENDING changes and fully restore the last-APPLIED state — the
+    isolated equivalent of the product's reject -> restore-to-previous. Restores both the
+    vip_instances fields and the exact member set (with their delivered config snapshots)
+    from `applied_snapshot`, so the agents keep running what they already have (no churn).
+    A never-applied PENDING VIP (no snapshot) is SOFT-deleted (is_active=FALSE) and its
+    staged version marked REJECTED — so the change is reversible from the Rejected tab
+    (undo-reject reactivates it). Never touches the global haproxy.cfg apply flow.
+    """
+    current_user = await _require(authorization, "update")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        # REJECT A STAGED DELETION: cancel it — the VIP keeps running exactly as before (it was
+        # never touched; is_active was never flipped). Clear the delete + purge intent and mark
+        # the staged version REJECTED. This is the "nothing happened" path the operator expects.
+        if v["pending_delete"]:
+            await conn.execute(
+                "UPDATE vip_instances SET pending_delete=FALSE, purge_on_teardown=FALSE, "
+                "last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await _transition_vip_versions(conn, vip_id, "REJECTED")
+            await log_user_activity(
+                user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "delete_cancelled": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "Deletion rejected — the VIP keeps running unchanged."}
+        if v["last_config_status"] != "PENDING":
+            return {"message": "Nothing to reject — no pending changes"}
+
+        snap_raw = v["applied_snapshot"]
+        snap = json.loads(snap_raw) if isinstance(snap_raw, str) else snap_raw
+        if not snap:
+            # Created but never applied -> reject SOFT-deletes the VIP (is_active=FALSE) and
+            # marks its staged version REJECTED. The row + members are kept so undo-reject can
+            # reactivate the exact VIP (no orphan). The partial unique indexes free its
+            # name/address/VRID for reuse while it's inactive.
+            await _transition_vip_versions(conn, vip_id, "REJECTED")
+            await conn.execute(
+                "UPDATE vip_instances SET is_active=FALSE, last_config_status='PENDING', "
+                "updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await log_user_activity(
+                user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "discarded": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "Pending VIP rejected (undo from the Rejected tab to restore it)"}
+
+        sv = snap["vip"]
+        sm = snap.get("members", [])
+        try:
+            async with conn.transaction():
+                await conn.execute("""
+                    UPDATE vip_instances SET
+                        name=$2, description=$3, virtual_ip=$4, prefix_length=$5, virtual_router_id=$6,
+                        advert_int=$7, use_unicast=$8, track_haproxy=$9, auth_pass_encrypted=$10,
+                        last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP
+                    WHERE id=$1
+                """, vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
+                    sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
+                    sv.get("auth_pass_encrypted"))
+                await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+                for m in sm:
+                    await conn.execute("""
+                        INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                            applied_config_content, applied_config_hash)
+                        VALUES ($1,$2,$3,$4,$5,$6,$7)
+                    """, vip_id, m["agent_id"], m["network_interface"], m["role"], m["priority"],
+                        m.get("applied_config_content"), m.get("applied_config_hash"))
+        except HTTPException:
+            raise
+        except Exception as ie:  # noqa: BLE001
+            if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                raise HTTPException(status_code=409,
+                                    detail="Cannot restore — the previous address/VRID was taken in the meantime")
+            raise
+        # Mark the standard config_version REJECTED (history); the VIP is back to APPLIED.
+        await _transition_vip_versions(conn, vip_id, "REJECTED")
+        await log_user_activity(
+            user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+            details={"name": v["name"], "restored": True},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": "Pending changes rejected — VIP restored to its last applied state"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.get("/{vip_id}/status")
+async def vip_status(vip_id: int, authorization: str = Header(None)):
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT id, name, is_active, last_config_status FROM vip_instances WHERE id=$1", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        members = await conn.fetch("""
+            SELECT m.agent_id, m.role, m.priority, m.network_interface,
+                   m.last_deploy_state, m.last_deploy_message, m.last_deploy_at, m.applied_config_hash,
+                   a.name AS agent_name, a.keepalive_state, a.keepalive_ip, a.status AS agent_status,
+                   a.capabilities
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id=$1 ORDER BY m.priority DESC
+        """, vip_id)
+        out = []
+        for m in members:
+            applied = m["applied_config_hash"]
+            deploy = m["last_deploy_state"]
+            capable = _capable(m["capabilities"])
+            if applied and not deploy:
+                converge = "awaiting agent (upgrade may be required)" if not capable else "converging"
+            else:
+                converge = deploy or "pending"
+            out.append({
+                "agent_name": m["agent_name"], "role": m["role"], "priority": m["priority"],
+                "network_interface": m["network_interface"],
+                "agent_status": m["agent_status"], "keepalive_state": m["keepalive_state"],
+                "keepalive_ip": m["keepalive_ip"], "keepalived_capable": capable,
+                "deploy_state": deploy, "deploy_message": m["last_deploy_message"],
+                "deploy_at": m["last_deploy_at"].isoformat() if m["last_deploy_at"] else None,
+                "convergence": converge,
+            })
+        return {"id": v["id"], "name": v["name"], "is_active": v["is_active"],
+                "last_config_status": v["last_config_status"], "members": out}
+    finally:
+        await close_database_connection(conn)
+
+
+# NOTE: the keepalived config preview/diff is served by the STANDARD config-version diff
+# endpoint (cluster.py get_config_version_diff, vip-* branch) via render_vip_config_masked
+# above — there is no bespoke VIP preview endpoint, so VIP changes use the product's
+# standard "View Change" like every other entity (issue #27 follow-up).

backend/services/acme_diagnostics.py

@@ -116,6 +116,10 @@ _PROBLEM_HUMANIZED: Dict[str, Dict[str, str]] = {
         "title": "HTTP-01 challenge response mismatch",
         "hint": "The CA fetched the challenge URL but received the wrong key authorization. Confirm the challenge was served from the right backend.",
     },
+    "urn:ietf:params:acme:error:externalAccountRequired": {
+        "title": "External Account Binding (EAB) required",
+        "hint": "This CA (e.g. ZeroSSL, Google) requires EAB. Enter the EAB Key ID and HMAC Key from your CA account when registering.",
+    },
     "urn:ietf:params:acme:error:invalidContact": {
         "title": "Invalid contact email",
         "hint": "The ACME account email is malformed. Update the LE account email.",
@@ -199,6 +203,23 @@ def humanize_error_detail(error_detail: Any) -> Dict[str, Any]:
     status = parsed.get("status")
     subproblems = parsed.get("subproblems") or []
 
+    # Issue #35: DNS-01 failures are recorded as {stage, reason, timestamp} (no RFC8555 "type"),
+    # so without this fallback the humanized alert would show a bare "ACME error" with no message.
+    # Surface the reason and a targeted hint so the operator knows exactly what to fix.
+    if not problem_type and parsed.get("reason"):
+        reason = str(parsed.get("reason"))
+        message = message or reason
+        title = "DNS-01 validation failed"
+        rlow = reason.lower()
+        if "decrypt" in rlow or "credential" in rlow:
+            hint = hint or "Re-enter the DNS provider credentials for this account in ACME Automation."
+        elif "zone" in rlow:
+            hint = hint or "Confirm the domain's DNS zone is managed by the configured provider and the token has access to it."
+        elif "deadline" in rlow or "expired" in rlow or "confirm" in rlow:
+            hint = hint or "The manual confirmation window passed. Create a new certificate request and publish the TXT record promptly."
+        else:
+            hint = hint or "Check the DNS TXT record and provider credentials, then retry."
+
     out = {
         "title": title,
         "message": message,
@@ -573,9 +594,18 @@ async def check_agents(conn, cluster_ids: List[int]) -> Dict[str, Any]:
             severity="warn",
             duration_ms=int((time.time() - started) * 1000),
         )
+    # Bulgu #84 (round-23 audit) — the canonical timestamp column on the
+    # `agents` table is `last_seen`. Pre-fix this query referenced a
+    # non-existent `a.last_heartbeat`, so every ACME preflight call
+    # (`POST /api/sites/preflight-acme`) crashed at the `check_agents`
+    # stage with `UndefinedColumnError: column a.last_heartbeat does
+    # not exist`, blocking the entire wizard's ACME pre-validation
+    # gate. Every other agents.last_seen reader in the codebase
+    # (routers/cluster.py:695-702, routers/agent.py, routers/dashboard
+    # *.py) uses `last_seen`; aligning here.
     rows = await conn.fetch(
         """
-        SELECT a.id, a.hostname, a.status, a.last_heartbeat, hc.id AS cluster_id, hc.name AS cluster_name
+        SELECT a.id, a.hostname, a.status, a.last_seen, hc.id AS cluster_id, hc.name AS cluster_name
         FROM agents a
         JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
         WHERE hc.id = ANY($1::int[])
@@ -623,6 +653,61 @@ async def check_agents(conn, cluster_ids: List[int]) -> Dict[str, Any]:
 CHECK_IDS = ("dns", "port80", "routing", "account", "agents")
 
 
+def _coerce_cluster_ids(raw) -> List[int]:
+    """Coerce a cluster_ids list to ints, dropping non-integer-compatible
+    values. JSONB-stored lists occasionally land as ["1", "2"] (string form)
+    due to legacy paths; asyncpg's `$1::int[]` cast then fails the diagnostic
+    query with InvalidTextRepresentationError. We normalise here so the
+    diagnostic surface is the same regardless of how the order was written.
+    """
+    out: List[int] = []
+    if not raw:
+        return out
+    for v in raw:
+        try:
+            out.append(int(v))
+        except (TypeError, ValueError):
+            continue
+    return out
+
+
+# Bulgu #94 (Round-25 audit) — the entire point of the diagnostic panel
+# is to SHOW the operator what went wrong. Pre-fix, a single check raising
+# an uncaught exception (e.g. an asyncpg cast error from a malformed
+# cluster_ids JSONB, a DNS resolver outage, an SSRF-guard glitch) would
+# propagate up to the router's `try/finally` block, which had no `except`
+# clause, and return HTTP 500 with no body. The operator saw only
+# "Internal Server Error" in DevTools — the inverse of what a diagnostic
+# panel should ever produce. We now wrap every check inside `run_checks`
+# so that a check crash becomes a structured `fail` row instead of
+# bubbling up; the operator gets the exception type + message in the
+# UI and can carry it forward, and the rest of the panel still renders.
+async def _safe_check(check_id: str, label: str, coro):
+    """Run an awaitable that produces a check result; swallow exceptions
+    and convert them to a structured `fail` result so the diagnostic
+    response is never short-circuited by a single broken check."""
+    started = time.time()
+    try:
+        return await coro
+    except Exception as exc:  # noqa: BLE001 — diagnostic boundary
+        duration_ms = int((time.time() - started) * 1000)
+        logger.exception(
+            "ACME diagnostic check %s raised", check_id
+        )
+        return _check_result(
+            check_id,
+            label,
+            "fail",
+            f"Diagnostic check crashed: {exc.__class__.__name__}: {exc}",
+            severity="error",
+            details={
+                "exception_type": exc.__class__.__name__,
+                "exception_message": str(exc),
+            },
+            duration_ms=duration_ms,
+        )
+
+
 async def run_checks(
     conn,
     *,
@@ -630,22 +715,53 @@ async def run_checks(
     cluster_ids: List[int],
     account_id: Optional[int],
     only: Optional[List[str]] = None,
+    challenge_type: str = "http-01",
 ) -> List[Dict[str, Any]]:
     """Execute the full pre-flight check suite. `only` lets callers re-run a
     subset (per-check rerun in the UI).
+
+    Every individual check is wrapped in `_safe_check` so the diagnostic
+    endpoint NEVER 500s because of one broken check — the operator gets
+    a structured `fail` row identifying which check crashed and why.
     """
     selected = set(only) if only else set(CHECK_IDS)
     results: List[Dict[str, Any]] = []
 
+    # Normalise inputs once so the per-check error stays in the right
+    # bucket (a malformed cluster_ids should not crash routing/agents).
+    safe_domains = [d for d in (domains or []) if isinstance(d, str) and d]
+    safe_cluster_ids = _coerce_cluster_ids(cluster_ids)
+    try:
+        safe_account_id = int(account_id) if account_id is not None else None
+    except (TypeError, ValueError):
+        safe_account_id = None
+
+    # Issue #35: DNS-01 validates via a TXT record, so the HTTP-01 reachability checks
+    # (public A record, inbound port 80, ACME Challenge Routing) do not apply — report them
+    # as `skipped` rather than failing an internal/isolated host that is actually fine.
+    is_dns01 = (challenge_type == "dns-01")
     if "dns" in selected:
-        results.append(await check_dns(domains))
+        if is_dns01:
+            results.append(_check_result("dns", "DNS resolution", "skipped",
+                                         "DNS-01: a public A record is not required (validation is via a TXT record).",
+                                         severity="info"))
+        else:
+            results.append(await _safe_check("dns", "DNS resolution", check_dns(safe_domains)))
     if "port80" in selected:
-        results.append(await check_port80(domains))
+        if is_dns01:
+            results.append(_check_result("port80", "Port 80 reachability", "skipped",
+                                         "DNS-01: inbound port 80 is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("port80", "Port 80 reachability", check_port80(safe_domains)))
     if "routing" in selected:
-        results.append(await check_routing(conn, domains, cluster_ids))
+        if is_dns01:
+            results.append(_check_result("routing", "HAProxy routing", "skipped",
+                                         "DNS-01: ACME Challenge Routing is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("routing", "HAProxy routing", check_routing(conn, safe_domains, safe_cluster_ids)))
     if "account" in selected:
-        results.append(await check_account(conn, account_id))
+        results.append(await _safe_check("account", "ACME account", check_account(conn, safe_account_id)))
     if "agents" in selected:
-        results.append(await check_agents(conn, cluster_ids))
+        results.append(await _safe_check("agents", "HAProxy agents", check_agents(conn, safe_cluster_ids)))
 
     return results

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -71,17 +75,21 @@ class ACMEService:
                     raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
                 data = await resp.json()
                 if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
+                    self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
                 data['_fetched_at'] = time.time()
                 self._directory_cache[directory_url] = data
                 return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
-            nonce = self._nonce
+        cached = self._nonce_by_dir.pop(directory_url, None)
-            self._nonce = None
+        if cached:
-            return nonce
+            return cached
         directory = await self.get_directory(directory_url)
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         async with aiohttp.ClientSession() as session:
             async with session.head(directory['newNonce']) as resp:
                 return resp.headers['Replay-Nonce']
@@ -128,6 +136,20 @@ class ACMEService:
         digest = hashlib.sha256(ordered.encode('utf-8')).digest()
         return _b64url(digest)
 
+    @staticmethod
+    def _dns_txt_value(key_authorization: str) -> str:
+        """RFC 8555 §8.4: the DNS-01 TXT value is base64url(SHA256(key_authorization)) over the
+        RAW 32-byte digest (NOT the hexdigest)."""
+        return _b64url(hashlib.sha256(key_authorization.encode('utf-8')).digest())
+
+    @staticmethod
+    def _challenge_dns_name(identifier: str) -> str:
+        """The `_acme-challenge.<base>` record name for an ACME identifier. A leading wildcard
+        `*.` is stripped, so both `*.example.com` and bare `example.com` map to the SAME name
+        `_acme-challenge.example.com` (which is why apex+wildcard need two coexisting TXT values)."""
+        base = identifier[2:] if identifier.startswith('*.') else identifier
+        return f"_acme-challenge.{base}"
+
     def _sign_jws(self, private_key, protected: dict, payload: Any) -> dict:
         protected_b64 = _b64url(json.dumps(protected).encode('utf-8'))
         if payload == "":
@@ -174,11 +196,17 @@ class ACMEService:
                     timeout=aiohttp.ClientTimeout(total=30),
                 ) as resp:
                     if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
+                    if resp.status == 400 and attempt < 2:
                         err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
+                        etype = (err.get('type') or '')
+                        edetail = (err.get('detail') or '').lower()
+                        # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                        # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                        # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                        # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                        if etype.endswith('badNonce') or 'nonce' in edetail:
                             nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
                             protected['nonce'] = nonce
                             body = self._sign_jws(private_key, protected, payload)
@@ -208,6 +236,8 @@ class ACMEService:
         tos_agreed: bool = True,
         eab_kid: Optional[str] = None,
         eab_hmac_key: Optional[str] = None,
+        challenge_type: str = 'http-01',
+        dns_provider: Optional[str] = None,
     ) -> dict:
         directory = await self.get_directory(directory_url)
         pem, jwk = self._generate_account_key()
@@ -250,13 +280,14 @@ class ACMEService:
         conn = await get_database_connection()
         try:
             row = await conn.fetchrow("""
-                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid)
+                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid, challenge_type, dns_provider)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 ON CONFLICT (email, directory_url) DO UPDATE SET
-                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6, updated_at = NOW()
+                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6,
-                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at
+                    challenge_type = $8, dns_provider = $9, updated_at = NOW()
+                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at, challenge_type, dns_provider
             """, email, directory_url, account_url, pem,
-                data.get('status') or 'valid', tos_agreed, eab_kid)
+                data.get('status') or 'valid', tos_agreed, eab_kid, challenge_type, dns_provider)
             return dict(row)
         finally:
             await close_database_connection(conn)
@@ -302,8 +333,10 @@ class ACMEService:
         account_id: int,
         domains: List[str],
         cluster_ids: Optional[List[int]] = None,
+        challenge_type: str = 'http-01',
+        created_by: Optional[int] = None,
     ) -> dict:
-        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}")
+        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}, challenge_type={challenge_type}")
         conn = await get_database_connection()
         try:
             account = await conn.fetchrow(
@@ -341,12 +374,12 @@ class ACMEService:
 
             order_row = await conn.fetchrow("""
                 INSERT INTO letsencrypt_orders
-                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids)
+                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids, challenge_type, created_by)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 RETURNING id
             """, account_id, order_url, data.get('status') or 'pending',
                 json.dumps(domains), data.get('finalize') or '', expires_at,
-                json.dumps(cluster_ids or []))
+                json.dumps(cluster_ids or []), challenge_type, created_by)
 
             order_id = order_row['id']
 
@@ -383,18 +416,24 @@ class ACMEService:
                 domain = (auth_data.get('identifier') or {}).get('value', '')
                 http01_for_domain = False
                 for challenge in (auth_data.get('challenges') or []):
-                    if challenge.get('type') == 'http-01':
+                    # Store only the challenge of the CHOSEN method (default 'http-01' keeps the
+                    # existing behaviour byte-identical; 'dns-01' selects the TXT challenge instead).
+                    if challenge.get('type') == challenge_type:
                         token = challenge['token']
                         jwk = self._get_jwk(private_key)
                         thumbprint = self._jwk_thumbprint(jwk)
                         key_auth = f"{token}.{thumbprint}"
+                        dns_txt = self._dns_txt_value(key_auth) if challenge_type == 'dns-01' else None
 
                         await conn.execute("""
-                            INSERT INTO acme_challenges (order_id, domain, token, key_authorization, challenge_url, status)
+                            INSERT INTO acme_challenges
-                            VALUES ($1, $2, $3, $4, $5, $6)
+                                (order_id, domain, token, key_authorization, challenge_url, status,
+                                 challenge_type, dns_txt_value)
+                            VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
                         """, order_id, domain, token, key_auth,
-                            challenge.get('url') or '', challenge.get('status') or 'pending')
+                            challenge.get('url') or '', challenge.get('status') or 'pending',
-                        logger.info(f"ACME: Challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
+                            challenge_type, dns_txt)
+                        logger.info(f"ACME: {challenge_type} challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
                         http01_for_domain = True
                 if http01_for_domain and domain:
                     domains_with_http01.add(domain)
@@ -413,7 +452,7 @@ class ACMEService:
                     error_payload, order_id
                 )
                 raise Exception(
-                    f"ACME order {order_id} created but no http-01 challenges available "
+                    f"ACME order {order_id} created but no {challenge_type} challenges available "
                     f"(auth fetch failures: {len(auth_fetch_failures)}). See order.error_detail for diagnostics."
                 )
             elif auth_fetch_failures:
@@ -449,10 +488,15 @@ class ACMEService:
         try:
             # Issue #12 / Commit 5a: include 'failed' challenges so they can be retried,
             # but rate-limit per challenge: max 5 attempts in last 5 minutes.
+            # DNS-01 skip-gate (the single safe choke point): never POST a challenge response for a
+            # dns-01 row whose TXT record has not been published yet — that would make the CA validate
+            # against a missing record and burn the order. http-01 rows (challenge_type 'http-01'/NULL)
+            # are never excluded, so the existing flow is byte-identical.
             challenges = await conn.fetch(
                 """SELECT * FROM acme_challenges
                    WHERE order_id = $1
-                     AND (status IN ('pending', 'failed') OR status IS NULL)""",
+                     AND (status IN ('pending', 'failed') OR status IS NULL)
+                     AND NOT (COALESCE(challenge_type, 'http-01') = 'dns-01' AND COALESCE(dns_record_published, FALSE) = FALSE)""",
                 order_id
             )
             order = await conn.fetchrow(

backend/services/dns01_orchestrator.py

@@ -0,0 +1,321 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): non-blocking per-cycle orchestration.
+
+Driven by the existing `complete_pending_acme_orders` background task (which already claims
+in-progress orders with `FOR UPDATE SKIP LOCKED`). For a dns-01 order this module advances AT MOST
+ONE step per 60s cycle — publish TXT, then (after a short min-age) respond — so the serial claim
+loop is never blocked by a multi-minute wait, and NO DNS library is needed (the CA is the source of
+truth; a propagation-lag `invalid` is recovered by a bounded fresh-order chain).
+
+Design invariants (from the hardening review):
+- Additive at the RRset level: publish/cleanup operate on a single (name, value), so wildcard+apex
+  (two values at one name) coexist.
+- No new order-status value: a failed order stays `invalid`; a boolean `dns01_retry_claimed` does the
+  winner-only CAS + claim exclusion, so existing `status` consumers are untouched.
+- Secrets (provider API tokens) are NEVER logged or written to error_detail/events.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta, timezone
+from typing import Dict, List, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+from services.acme_service import acme_service as acme_svc, ACMEService
+from services.dns_providers import get_provider, is_supported, DnsProviderError
+from utils.dns_credentials import decrypt_dns_credentials
+from utils.activity_log import record_event
+
+logger = logging.getLogger(__name__)
+
+# Tunables (kept conservative vs Let's Encrypt rate limits: 5 failed-validations/host/hour,
+# 300 new-orders/account/3h).
+PROPAGATION_GRACE_SECONDS = 25          # min age before we tell the CA to validate
+MANUAL_CONFIRM_TTL = timedelta(hours=48)
+MAX_RETRIES = 3                          # bounded fresh-order chain (1 original + 3 retries = 4 orders)
+# Retry backoff floor (minutes) indexed by the order's current dns01_attempts: [15, 30, 60].
+# The AUTHORITATIVE implementation is the SQL CASE in main.py's claim query
+# (complete_pending_acme_orders), so the backoff is evaluated atomically with the
+# FOR UPDATE SKIP LOCKED claim. Documented here only — do not reintroduce a second copy.
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _aware(dt) -> Optional[datetime]:
+    if dt is None:
+        return None
+    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+
+
+async def _load_credentials(conn, account_id: int) -> Tuple[Optional[Dict[str, str]], bool]:
+    """Return (credentials_dict_or_None, row_exists). credentials None + row_exists True means the
+    stored token could not be decrypted (e.g. SECRET_KEY rotated)."""
+    row = await conn.fetchrow(
+        "SELECT credentials_encrypted FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+        account_id,
+    )
+    if not row:
+        return None, False
+    return decrypt_dns_credentials(row["credentials_encrypted"]), True
+
+
+async def _fail_order(conn, order_id: int, reason: str) -> None:
+    """Mark an order invalid with a sanitized reason (no secrets) + event."""
+    import json
+    payload = json.dumps({"stage": "dns01", "reason": reason, "timestamp": _now().isoformat()})
+    await conn.execute(
+        "UPDATE letsencrypt_orders SET status = 'invalid', error_detail = $1, updated_at = NOW() WHERE id = $2",
+        payload, order_id,
+    )
+    await record_event(order_id, "acme.dns01.validation", severity="ERROR", message=reason, conn=conn)
+
+
+async def advance_dns01_order(order_id: int) -> None:
+    """One non-blocking step for a claimed pending/processing dns-01 order. No-op for http-01 or
+    orders not in a publishable state. Safe to call every cycle (idempotent via CAS flags)."""
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, o.account_id, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if order["status"] not in ("pending", "processing"):
+            return
+
+        challenges = await conn.fetch(
+            "SELECT * FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'", order_id
+        )
+        if not challenges:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+
+        # --- Manual provider: the user publishes + confirms; we only enforce the deadline. ---
+        if provider_name == "manual" or not is_supported(provider_name):
+            deadline = _aware(challenges[0]["manual_confirm_deadline"])
+            if deadline is None:
+                new_deadline = _now() + MANUAL_CONFIRM_TTL
+                await conn.execute(
+                    "UPDATE acme_challenges SET manual_confirm_deadline = $1 WHERE order_id = $2 AND manual_confirm_deadline IS NULL",
+                    new_deadline, order_id,
+                )
+            elif _now() > deadline:
+                await _fail_order(conn, order_id,
+                                  "Manual DNS-01 confirmation deadline passed without confirmation.")
+            return  # respond happens via the dns-confirm endpoint
+
+        # --- Automated provider (e.g. Cloudflare). ---
+        creds, row_exists = await _load_credentials(conn, order["account_id"])
+        if not row_exists:
+            await _fail_order(conn, order_id,
+                              f"No DNS provider credentials configured for provider '{provider_name}'.")
+            return
+        if creds is None:
+            await _fail_order(conn, order_id,
+                              "DNS provider credentials could not be decrypted; re-enter them in Settings.")
+            return
+
+        provider = get_provider(provider_name, creds)
+
+        # Publish any not-yet-published challenge (CAS so two replicas can't double-publish).
+        for ch in challenges:
+            if ch["dns_record_published"]:
+                continue
+            flipped = await conn.fetchval(
+                """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = NOW()
+                   WHERE id = $1 AND dns_record_published = FALSE RETURNING id""",
+                ch["id"],
+            )
+            if not flipped:
+                continue
+            name = ACMEService._challenge_dns_name(ch["domain"])
+            try:
+                await provider.add_txt_record(name, ch["dns_txt_value"])
+                await record_event(order_id, "acme.dns01.publish",
+                                   message=f"Published TXT {name}; waiting for DNS propagation before asking the CA to validate.",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+            except DnsProviderError as exc:
+                # Revert so the next cycle retries the publish; keep the order pending. exc is sanitized.
+                await conn.execute(
+                    "UPDATE acme_challenges SET dns_record_published = FALSE, dns_published_at = NULL WHERE id = $1",
+                    ch["id"],
+                )
+                await record_event(order_id, "acme.dns01.publish", severity="WARNING",
+                                   message=f"Publish failed for {name}: {exc}",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+                return
+
+        # All published? Then respond once the min-age gate has elapsed (across cycles, no sleep).
+        rows = await conn.fetch(
+            "SELECT dns_record_published, dns_published_at FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+            order_id,
+        )
+        if any(not r["dns_record_published"] for r in rows):
+            return
+        published_ats = [_aware(r["dns_published_at"]) for r in rows if r["dns_published_at"]]
+        if not published_ats:
+            return
+        if (_now() - min(published_ats)).total_seconds() < PROPAGATION_GRACE_SECONDS:
+            return  # wait one more cycle
+
+        # Only POST the challenge response if something still needs validating — avoids re-POSTing
+        # every cycle (and bumping dns01_last_attempt_at) once the CA already has them processing.
+        still_pending = await conn.fetchval(
+            """SELECT 1 FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'
+               AND (status IN ('pending', 'failed') OR status IS NULL) LIMIT 1""",
+            order_id,
+        )
+        if not still_pending:
+            return
+
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Told the CA to validate the DNS-01 challenge(s).", conn=conn)
+    finally:
+        await close_database_connection(conn)
+
+
+async def confirm_manual_dns01(order_id: int) -> Dict:
+    """Called by POST /orders/{id}/dns-confirm for the manual provider: mark the TXT published and
+    tell the CA to validate. Returns a small status dict."""
+    conn = await get_database_connection()
+    try:
+        await conn.execute(
+            """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = COALESCE(dns_published_at, NOW())
+               WHERE order_id = $1 AND challenge_type = 'dns-01'""",
+            order_id,
+        )
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Manual DNS-01 confirmed; told the CA to validate.", conn=conn)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+async def retry_invalid_dns01(order_id: int) -> None:
+    """Bounded fresh-order recovery for a dns-01 order that went `invalid` (e.g. propagation lag).
+    Winner-only CAS on `dns01_retry_claimed`; cleans the old TXT, mints a child order. No-op for
+    http-01 or when the budget is exhausted."""
+    conn = await get_database_connection()
+    child_created = False
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.*, a.dns_provider FROM letsencrypt_orders o
+               JOIN letsencrypt_accounts a ON o.account_id = a.id WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if (order["dns01_attempts"] or 0) >= MAX_RETRIES:
+            return  # budget exhausted; stays terminal invalid
+
+        # Winner-only claim (closes the cross-replica double-mint race).
+        claimed = await conn.fetchval(
+            """UPDATE letsencrypt_orders SET dns01_retry_claimed = TRUE, updated_at = NOW()
+               WHERE id = $1 AND status = 'invalid' AND dns01_retry_claimed = FALSE RETURNING id""",
+            order_id,
+        )
+        if not claimed:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+        # Best-effort cleanup of this order's TXT before minting the replacement.
+        if provider_name != "manual" and is_supported(provider_name):
+            creds, _exists = await _load_credentials(conn, order["account_id"])
+            if creds:
+                provider = get_provider(provider_name, creds)
+                chs = await conn.fetch(
+                    "SELECT domain, dns_txt_value FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+                    order_id,
+                )
+                for ch in chs:
+                    try:
+                        await provider.remove_txt_record(ACMEService._challenge_dns_name(ch["domain"]), ch["dns_txt_value"])
+                    except DnsProviderError:
+                        pass  # tolerate; the reconcile sweep will retry
+        await conn.execute(
+            "UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE order_id = $1 AND dns_record_published = TRUE",
+            order_id,
+        )
+
+        import json
+        domains = json.loads(order["domains"]) if isinstance(order["domains"], str) else (order["domains"] or [])
+        cluster_ids = json.loads(order["cluster_ids"]) if isinstance(order["cluster_ids"], str) else (order["cluster_ids"] or [])
+        next_attempts = (order["dns01_attempts"] or 0) + 1
+        child = await acme_svc.create_order(
+            order["account_id"], domains, cluster_ids, challenge_type="dns-01",
+            created_by=order["created_by"],
+        )
+        child_created = True
+        await conn.execute(
+            """UPDATE letsencrypt_orders
+               SET dns01_attempts = $1, dns01_parent_order_id = $2, dns01_last_attempt_at = NOW()
+               WHERE id = $3""",
+            next_attempts, order_id, child["order_id"],
+        )
+        await record_event(order_id, "acme.dns01.validation", severity="WARNING",
+                           message=f"DNS-01 order invalid; minted retry #{next_attempts} (order {child['order_id']}).",
+                           details={"child_order_id": child["order_id"], "attempt": next_attempts}, conn=conn)
+    except Exception as exc:  # noqa: BLE001
+        logger.error(f"[DNS01-RETRY] order {order_id}: {exc}")
+        # A transient failure (e.g. CA rate limit) BEFORE the child was minted must NOT permanently
+        # burn the retry slot — reset the claim so the next cycle can retry. If the child was already
+        # created, leave the claim set (resetting would double-mint).
+        if not child_created:
+            try:
+                await conn.execute(
+                    "UPDATE letsencrypt_orders SET dns01_retry_claimed = FALSE WHERE id = $1 AND status = 'invalid'",
+                    order_id,
+                )
+            except Exception:
+                pass
+    finally:
+        await close_database_connection(conn)
+
+
+async def reconcile_dns01_cleanup() -> None:
+    """Best-effort sweep that removes any TXT records left published for terminal orders (covers a
+    cleanup that failed, or the kill-switch being flipped off mid-flight). NOT gated by the
+    kill-switch. Runs once per completion cycle."""
+    conn = await get_database_connection()
+    try:
+        rows = await conn.fetch(
+            """SELECT c.id AS chal_id, c.order_id, c.domain, c.dns_txt_value, o.account_id, a.dns_provider
+               FROM acme_challenges c
+               JOIN letsencrypt_orders o ON c.order_id = o.id
+               JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE c.challenge_type = 'dns-01'
+                 AND c.dns_record_published = TRUE
+                 AND COALESCE(c.dns_record_cleaned, FALSE) = FALSE
+                 AND o.status IN ('valid', 'invalid', 'cancelled')
+               LIMIT 50""",
+        )
+        for r in rows:
+            provider_name = (r["dns_provider"] or "manual").strip()
+            if provider_name == "manual" or not is_supported(provider_name):
+                # Manual: nothing to call; mark cleaned so we stop revisiting.
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                continue
+            creds, _exists = await _load_credentials(conn, r["account_id"])
+            if creds is None:
+                continue  # can't clean without creds; leave for a later pass
+            provider = get_provider(provider_name, creds)
+            try:
+                await provider.remove_txt_record(ACMEService._challenge_dns_name(r["domain"]), r["dns_txt_value"])
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                await record_event(r["order_id"], "acme.dns01.cleanup",
+                                   message=f"Cleaned up TXT for {r['domain']}", conn=conn)
+            except DnsProviderError:
+                pass  # retry next sweep
+    except Exception as exc:  # noqa: BLE001
+        logger.debug(f"[DNS01-RECONCILE] skipped: {exc}")
+    finally:
+        await close_database_connection(conn)

backend/services/dns_providers/__init__.py

@@ -0,0 +1,14 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): pluggable DNS provider package.
+
+A small adapter layer so DNS-01 challenges can publish/clean up the
+`_acme-challenge.<domain>` TXT record via different DNS providers. The interface is
+additive at the RRset level (add/remove a single value by name+content, never
+overwrite-by-name) so multiple coexisting values at one name (wildcard + apex) work.
+
+MVP providers: manual (user publishes the TXT themselves) and Cloudflare. New providers
+plug in via the registry without touching the orchestration.
+"""
+from .base import DnsProvider, DnsProviderError
+from .registry import get_provider, list_providers, is_supported
+
+__all__ = ["DnsProvider", "DnsProviderError", "get_provider", "list_providers", "is_supported"]

backend/services/dns_providers/base.py

@@ -0,0 +1,53 @@
+"""Abstract DNS provider interface for ACME DNS-01 (Issue #35)."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Dict, List
+
+
+class DnsProviderError(Exception):
+    """A DNS provider failure with a SANITIZED, user-safe message.
+
+    The message must NEVER contain API tokens, request headers, or other secrets — it is
+    persisted to acme_order_events / order error_detail and shown in the UI. Raise this (not a
+    raw aiohttp/json error) so credentials can't leak into logs or the order timeline.
+    """
+
+
+class DnsProvider(ABC):
+    """Base class for a pluggable DNS provider.
+
+    RRset semantics are ADDITIVE: ``add_txt_record`` ensures a (name, value) TXT exists WITHOUT
+    removing other values at the same name, and ``remove_txt_record`` deletes ONLY the record
+    matching (name, value). This is required because a cert for ``example.com`` + ``*.example.com``
+    publishes two distinct values at the SAME name ``_acme-challenge.example.com``.
+    """
+
+    # Stable machine name (used in DB + API); human label; whether the provider automates publishing.
+    name: str = "base"
+    label: str = "Base"
+    automated: bool = True
+
+    # Declarative schema the UI renders to collect credentials. Each field:
+    #   {"key", "label", "type" ("text"|"password"), "required" (bool), "max_length" (int), "help" (str)}
+    credential_fields: List[Dict] = []
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        self.credentials = credentials or {}
+
+    @abstractmethod
+    async def verify_credentials(self) -> Dict:
+        """Validate the stored credentials against the provider. Returns
+        ``{"ok": bool, "detail": str}`` (detail is user-safe). Must not raise on auth failure —
+        return ``ok=False`` with a sanitized detail; may raise DnsProviderError on transport errors.
+        """
+
+    @abstractmethod
+    async def add_txt_record(self, name: str, value: str) -> None:
+        """Ensure a TXT record (name, value) exists. Idempotent; must not remove other values
+        at the same name. Raise DnsProviderError (sanitized) on failure."""
+
+    @abstractmethod
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        """Remove ONLY the TXT record matching (name, value). Tolerate 'already gone'.
+        Raise DnsProviderError (sanitized) on a real failure."""

backend/services/dns_providers/cloudflare.py

@@ -0,0 +1,179 @@
+"""Cloudflare DNS provider for ACME DNS-01 (Issue #35).
+
+Uses the Cloudflare API v4 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name influences which zone is used). Errors are wrapped in DnsProviderError with a
+sanitized message so the API token never reaches logs / order events.
+
+Token scope required: Zone:DNS:Edit + Zone:Read.
+"""
+from __future__ import annotations
+
+import logging
+import re
+from typing import Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+CLOUDFLARE_API_BASE = "https://api.cloudflare.com/client/v4"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# Characters NOT valid in an HTTP bearer credential (RFC 6750 token68: A-Za-z0-9-._~+/=).
+# Cloudflare API tokens are a strict subset of this set, so removing anything outside it can
+# never corrupt a valid token, but it does strip the paste artifacts that make Cloudflare
+# reject the Authorization header with HTTP 400 "Invalid request headers" (CF code 6003):
+# surrounding/embedded quotes, interior spaces/tabs, zero-width/unicode chars, and CR/LF
+# (the latter would otherwise make aiohttp raise client-side before the request is even sent).
+_NON_TOKEN68 = re.compile(r"[^A-Za-z0-9._~+/=-]")
+
+
+def _strip_quotes(s: str) -> str:
+    s = (s or "").strip()
+    if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
+        return s[1:-1]
+    return s
+
+
+def _sanitize_token(s: str) -> str:
+    """Strip surrounding quotes/whitespace, then drop every character outside the token68 set."""
+    return _NON_TOKEN68.sub("", _strip_quotes(s))
+
+
+class CloudflareDNSProvider(DnsProvider):
+    name = "cloudflare"
+    label = "Cloudflare"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_token",
+            "label": "API Token",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": "Scoped API token with Zone:DNS:Edit and Zone:Read permissions.",
+        }
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        self._raw_token = (self.credentials.get("api_token") or "").strip()
+        # Sanitize to the token68 set so a pasted token with quotes/spaces/control/unicode chars
+        # cannot produce an invalid Authorization header (CF 6003 "Invalid request headers").
+        self._token = _sanitize_token(self._raw_token)
+
+    def _headers(self) -> Dict[str, str]:
+        return {"Authorization": f"Bearer {self._token}", "Content-Type": "application/json"}
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> dict:
+        """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
+        DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
+        url = f"{CLOUDFLARE_API_BASE}{path}"
+        try:
+            async with session.request(
+                method, url, headers=self._headers(), allow_redirects=False, **kwargs
+            ) as resp:
+                try:
+                    body = await resp.json()
+                except Exception:  # noqa: BLE001
+                    body = {}
+                if resp.status in (401, 403):
+                    raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                if resp.status >= 400 or not body.get("success", False):
+                    # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                    # human message text, never the request (which carries the token header).
+                    msgs = "; ".join(
+                        str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                    )
+                    raise DnsProviderError(
+                        f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                    )
+                return body
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Do NOT include exc verbatim everywhere; aiohttp client errors are URL/transport only
+            # (no token), but keep the message generic and stable.
+            raise DnsProviderError(f"Could not reach the Cloudflare API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected Cloudflare API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._token:
+            return {"ok": False, "detail": "No Cloudflare API token provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                body = await self._request(session, "GET", "/zones?per_page=1")
+            total = ((body.get("result_info") or {}).get("total_count"))
+            detail = "Cloudflare token valid."
+            if isinstance(total, int):
+                detail = f"Cloudflare token valid; {total} zone(s) visible."
+            return {"ok": True, "detail": detail}
+        except DnsProviderError as exc:
+            # Always surface the real Cloudflare reason (e.g. token scope). If sanitizing also changed
+            # the token, append a hint that stray characters were stripped (never echo the token).
+            detail = str(exc)
+            if self._raw_token != self._token:
+                detail += (" Note: the token contained characters that were stripped; if it still "
+                           "fails, re-copy it from Cloudflare without quotes or spaces.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the Cloudflare token."}
+
+    async def _resolve_zone(self, session: aiohttp.ClientSession, record_name: str) -> Tuple[str, str]:
+        """Find the most-specific (longest-suffix) managed zone for a record name.
+        Returns (zone_id, zone_name). Raises DnsProviderError if no zone matches."""
+        labels = record_name.split(".")
+        # Walk suffixes from longest to shortest; a zone needs at least 2 labels.
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break
+            body = await self._request(
+                session, "GET", f"/zones?name={quote(candidate)}&status=active&per_page=50"
+            )
+            results = body.get("result") or []
+            if results:
+                return results[0]["id"], candidate
+        raise DnsProviderError(f"No managed Cloudflare zone found for {record_name}.")
+
+    async def _find_record_id(
+        self, session: aiohttp.ClientSession, zone_id: str, name: str, value: str
+    ) -> Optional[str]:
+        body = await self._request(
+            session, "GET", f"/zones/{zone_id}/dns_records?type=TXT&name={quote(name)}&per_page=100"
+        )
+        for rec in body.get("result") or []:
+            if _strip_quotes(rec.get("content", "")) == value:
+                return rec.get("id")
+        return None
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            zone_id, _zone_name = await self._resolve_zone(session, name)
+            # Idempotent: only create if (name, value) is not already present (preserves coexisting values).
+            existing = await self._find_record_id(session, zone_id, name, value)
+            if existing:
+                return
+            await self._request(
+                session,
+                "POST",
+                f"/zones/{zone_id}/dns_records",
+                json={"type": "TXT", "name": name, "content": value, "ttl": 120},
+            )
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            try:
+                zone_id, _zone_name = await self._resolve_zone(session, name)
+            except DnsProviderError:
+                # Zone gone / not resolvable — nothing we can clean up.
+                return
+            record_id = await self._find_record_id(session, zone_id, name, value)
+            if not record_id:
+                return  # already gone — tolerate
+            await self._request(session, "DELETE", f"/zones/{zone_id}/dns_records/{record_id}")

backend/services/dns_providers/manual.py

@@ -0,0 +1,31 @@
+"""Manual DNS provider for ACME DNS-01 (Issue #35).
+
+The user publishes the `_acme-challenge` TXT record in their own DNS (any provider, including
+fully internal/isolated DNS that no API can reach) and then confirms via the UI. There is no API
+to call, so add/remove are no-ops and the orchestration waits for an explicit `dns-confirm`.
+CNAME delegation works implicitly here: the CA follows a CNAME, so a user who delegates
+`_acme-challenge` elsewhere just publishes the value there and confirms.
+"""
+from __future__ import annotations
+
+from typing import Dict, List
+
+from .base import DnsProvider
+
+
+class ManualDNSProvider(DnsProvider):
+    name = "manual"
+    label = "Manual (publish the TXT record yourself)"
+    automated = False
+    credential_fields: List[Dict] = []  # no credentials needed
+
+    async def verify_credentials(self) -> Dict:
+        return {"ok": True, "detail": "Manual mode needs no credentials. You will publish the TXT record yourself."}
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user publishes the record and confirms via the UI.
+        return None
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user may remove the record manually after issuance.
+        return None

backend/services/dns_providers/registry.py

@@ -0,0 +1,44 @@
+"""DNS provider registry for ACME DNS-01 (Issue #35).
+
+Single source of truth mapping a provider name -> class. The API serves the credential-field
+schema from here (so the UI has no hardcoded provider fields) and validates inbound provider
+names against this allow-list. Adding a provider = add it here; nothing else changes.
+"""
+from __future__ import annotations
+
+from typing import Dict, List, Type
+
+from .base import DnsProvider
+from .cloudflare import CloudflareDNSProvider
+from .manual import ManualDNSProvider
+
+_PROVIDERS: Dict[str, Type[DnsProvider]] = {
+    ManualDNSProvider.name: ManualDNSProvider,
+    CloudflareDNSProvider.name: CloudflareDNSProvider,
+}
+
+
+def is_supported(name: str) -> bool:
+    return name in _PROVIDERS
+
+
+def get_provider(name: str, credentials: Dict[str, str] | None = None) -> DnsProvider:
+    cls = _PROVIDERS.get(name)
+    if cls is None:
+        raise ValueError(f"Unsupported DNS provider: {name}")
+    return cls(credentials or {})
+
+
+def list_providers() -> List[Dict]:
+    """Return the UI-facing provider catalog: name, label, automated flag, and credential schema."""
+    out: List[Dict] = []
+    for name, cls in _PROVIDERS.items():
+        out.append(
+            {
+                "name": cls.name,
+                "label": cls.label,
+                "automated": cls.automated,
+                "credential_fields": cls.credential_fields,
+            }
+        )
+    return out

backend/services/haproxy_config.py

@@ -908,6 +908,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 "redirect": [], "use_be": [], "default_be": [],
             }
             _stick_table_emitted = False
+            # Whether this frontend emits ANY stick-counter usage (`track-sc<N>` or an
+            # `sc_*_rate(...)` fetch). If it does but no `stick-table` is declared, HAProxy
+            # fatally rejects the WHOLE cluster config with "table '<frontend>' used but not
+            # configured". This happens with rate-limit directives baked into a frontend's
+            # stored fields (request_headers/options) by an older version or a config import.
+            # We track it here and inject a default stick-table before flushing if needed.
+            _sc_counter_used = False
             # Phase K Phase D follow-up (Bulgu #13) — same dedup
             # contract for `http-request track-sc<N> <fetch>` lines.
             # HAProxy only NEEDS one tracking call per
@@ -931,9 +938,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 correct frontend-block bucket. Idempotent for stick-table
                 lines (R3.3 dedup) AND http-request track-sc<N> lines
                 (Bulgu #13 dedup)."""
-                nonlocal _stick_table_emitted
+                nonlocal _stick_table_emitted, _sc_counter_used
                 cat = _categorize_haproxy_directive(line)
                 stripped = line.strip()
+                # Any stick-counter usage (track-sc<N> write, or an sc_*_rate(...) fetch like
+                # sc_http_req_rate(0)) requires a stick-table in this frontend.
+                if "track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped):
+                    _sc_counter_used = True
                 if cat == "stick":
                     if _stick_table_emitted and stripped.startswith("stick-table"):
                         logger.debug(
@@ -1188,6 +1199,22 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 else:
                     logger.warning(f"Config Generation: No config lines generated for WAF rule '{waf_rule['name']}' (ID: {waf_rule['id']}, Type: {waf_rule['rule_type']})")
 
+            # Robustness fix: if this frontend uses a stick counter (track-sc<N> or an
+            # sc_*_rate(...) fetch) but declared NO stick-table, inject a default one so HAProxy
+            # doesn't fatally reject the whole cluster config with "table '<frontend>' used but
+            # not configured". This rescues rate-limit directives baked into a frontend's stored
+            # request_headers/options by an older version or import. Purely additive — it only
+            # fires when a counter is used AND no table exists (a config that is invalid today),
+            # so it never changes a frontend that already has a stick-table or doesn't rate-limit.
+            if _sc_counter_used and not _stick_table_emitted:
+                _fe_buckets["stick"].insert(
+                    0, "    stick-table type ip size 100k expire 30s store http_req_rate(10s)")
+                _stick_table_emitted = True
+                logger.info(
+                    f"STICK-TABLE AUTO-INJECT: frontend '{frontend['name']}' uses a stick "
+                    f"counter (track-sc/sc_*_rate) but declared no stick-table; injected a "
+                    f"default so the config stays valid.")
+
             # ─────────────────────────────────────────────────────────────
             # Flush the per-frontend buckets in canonical HAProxy order.
             # The order below is the single source of truth for emit

backend/services/keepalived_config.py

@@ -0,0 +1,192 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Standalone, DB-free renderer for a node's /etc/keepalived/keepalived.conf and the
+HAProxy health-check script, plus Fernet at-rest encryption for the VRRP secret.
+
+The router fetches DB rows and calls these pure functions; nothing here touches the
+database or logs secrets. Trivially unit-testable (see tests/test_keepalived_config.py).
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import re
+from typing import List, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# Written into every file we manage so the agent can tell "ours" from a
+# hand-maintained keepalived setup (ownership guard, B-3/T-2). Must match the
+# string the agent greps for in linux_install.sh.
+OWNERSHIP_MARKER = "# Managed by HAProxy OpenManager"
+CHECK_SCRIPT_PATH = "/etc/keepalived/check_haproxy.sh"
+
+
+# ---------------------------------------------------------------------------
+# VRRP secret at rest (mirrors backend/services/mfa_service.py)
+# ---------------------------------------------------------------------------
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit VIP_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so the secret survives restarts, like MFA)."""
+    explicit = os.getenv("VIP_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("VIP_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"vip-vrrp-secret-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_vrrp_secret(secret_plain: str) -> str:
+    return _get_fernet().encrypt(secret_plain.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_vrrp_secret(secret_encrypted: str) -> Optional[str]:
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt VRRP secret (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting VRRP secret: %s", exc)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Renderers
+# ---------------------------------------------------------------------------
+def build_haproxy_check_script(*, bin_path: Optional[str] = None,
+                               config_path: Optional[str] = None) -> str:
+    """Render the health-check the agent writes to CHECK_SCRIPT_PATH.
+
+    Derives the process name from the HAProxy binary basename (B-4) rather than a
+    blind hardcoded 'haproxy'. Returns non-zero when HAProxy isn't running so the
+    VRRP track_script lowers this node's priority and the VIP fails over.
+    """
+    proc = "haproxy"
+    if bin_path:
+        base = os.path.basename(bin_path.strip())
+        if re.match(r'^[A-Za-z0-9._-]{1,64}$', base):
+            proc = base
+    return (
+        "#!/bin/sh\n"
+        f"{OWNERSHIP_MARKER} — DO NOT EDIT\n"
+        "# Exits 0 while HAProxy is up; non-zero triggers VRRP failover.\n"
+        f"pidof {proc} >/dev/null 2>&1 || exit 1\n"
+        "exit 0\n"
+    )
+
+
+def _vrrp_instance_name(vip_id: int) -> str:
+    return f"VI_{int(vip_id)}"
+
+
+def _failover_weight(members: List[dict]) -> int:
+    """Negative weight so a failed MASTER drops strictly below every healthy BACKUP
+    (B-6). master_priority + weight < min(backup_priority)."""
+    master = next((m for m in members if str(m.get("role", "")).upper() == "MASTER"), None)
+    backups = [int(m["priority"]) for m in members if str(m.get("role", "")).upper() != "MASTER"]
+    if not master or not backups:
+        return -20
+    return -((int(master["priority"]) - min(backups)) + 1)
+
+
+def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
+                           peer_ips: List[str], auth_pass_plain: Optional[str]) -> str:
+    """Render one node's keepalived.conf from the VIP + member rows.
+
+    `vip` keys: id, name, virtual_ip, prefix_length, virtual_router_id, advert_int,
+                use_unicast, track_haproxy.
+    `this_agent` keys: role, priority, network_interface, ip_address (str).
+    `peer_ips`: the OTHER members' ip_address strings (already str()'d by the caller).
+    Caller must never log the returned string (it may contain auth_pass).
+    """
+    role = str(this_agent["role"]).upper()
+    iface = this_agent["network_interface"]
+    prio = int(this_agent["priority"])
+    track = bool(vip.get("track_haproxy", True))
+    use_unicast = bool(vip.get("use_unicast", True))
+    vrid = int(vip["virtual_router_id"])
+    advert = int(vip.get("advert_int", 1))
+    name = str(vip.get("name", ""))
+    inst = _vrrp_instance_name(vip["id"])
+
+    lines: List[str] = []
+    lines.append(f"{OWNERSHIP_MARKER} — DO NOT EDIT")
+    lines.append(f'# VIP "{name}" (id={vip["id"]}) — role {role}')
+    lines.append("global_defs {")
+    lines.append("    enable_script_security")
+    lines.append("    script_user root")
+    lines.append("}")
+    lines.append("")
+
+    if track:
+        weight = _failover_weight(members)
+        lines.append("vrrp_script chk_haproxy {")
+        lines.append(f'    script "{CHECK_SCRIPT_PATH}"')
+        lines.append("    interval 2")
+        lines.append("    fall 2")
+        lines.append("    rise 2")
+        lines.append(f"    weight {weight}")
+        lines.append("}")
+        lines.append("")
+
+    lines.append(f"vrrp_instance {inst} {{")
+    lines.append(f"    state {role}")
+    lines.append(f"    interface {iface}")
+    lines.append(f"    virtual_router_id {vrid}")
+    lines.append(f"    priority {prio}")
+    lines.append(f"    advert_int {advert}")
+    if auth_pass_plain:
+        lines.append("    authentication {")
+        lines.append("        auth_type PASS")
+        lines.append(f"        auth_pass {auth_pass_plain}")
+        lines.append("    }")
+    # Unicast only makes sense with at least one peer. For a single-node VIP (no peers)
+    # we deliberately omit the unicast block: keepalived treats a bare `unicast_src_ip`
+    # with no `unicast_peer` as deprecated, warns, and silently falls back to multicast —
+    # and `keepalived -t` flags it. Omitting it yields a clean multicast config that holds
+    # the VIP with no peer to talk to. Multi-node behaviour (peers present) is unchanged.
+    if use_unicast and peer_ips:
+        src = this_agent.get("ip_address")
+        if src:
+            lines.append(f"    unicast_src_ip {src}")
+        lines.append("    unicast_peer {")
+        for p in peer_ips:
+            lines.append(f"        {p}")
+        lines.append("    }")
+    lines.append("    virtual_ipaddress {")
+    lines.append(f'        {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
+    lines.append("    }")
+    if track:
+        lines.append("    track_script {")
+        lines.append("        chk_haproxy")
+        lines.append("    }")
+    lines.append("}")
+    return "\n".join(lines) + "\n"

backend/services/mfa_service.py

@@ -0,0 +1,238 @@
+"""MFA (TOTP + backup codes) service layer — Issue #18, v1.6.0.
+
+Owns the cryptographic and persistence-shape concerns of multi-factor auth:
+  - TOTP secret generation / verification with replay protection (RFC 6238)
+  - Backup code generation, hashing (bcrypt) and atomic single-use consumption
+  - Fernet-based encryption of TOTP secrets at rest
+
+Strictly no logging of secrets — only metadata (lengths, counts) is logged.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import secrets as _secrets
+import time
+from typing import List, Optional, Tuple
+from urllib.parse import quote
+
+import bcrypt
+import pyotp
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# RFC 6238 parameters — kept conservative for widest authenticator app compatibility.
+TOTP_DIGITS = 6
+TOTP_PERIOD = 30
+TOTP_DIGEST = "sha1"
+TOTP_VALID_WINDOW_STEPS = 1  # ±1 step (±30s) tolerance
+
+# Backup code spec (Plan section 5).
+# Alphabet drops the confusing pairs: 0/O, 1/I, L. Resulting size is 31, which
+# still yields 31**8 ≈ 8.5×10^11 combinations per half — far beyond brute-force.
+BACKUP_CODE_COUNT = 10
+BACKUP_CODE_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"
+BACKUP_CODE_HALF_LEN = 4  # XXXX-YYYY
+
+# OTP URI defaults.
+DEFAULT_ISSUER = "HAProxy OpenManager"
+ACCOUNT_LABEL_DOMAIN_FALLBACK = "haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Fernet key resolution
+# ---------------------------------------------------------------------------
+
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Resolve the Fernet key, preferring the explicit env var.
+
+    Falls back to HKDF over SECRET_KEY with a versioned info string so a future
+    rotation can be expressed by bumping the version suffix.
+    """
+    explicit = os.getenv("MFA_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:
+            logger.error("MFA_ENCRYPTION_KEY env var present but invalid: %s", exc)
+            # fall through to HKDF derivation rather than crashing the app
+
+    logger.warning(
+        "MFA_ENCRYPTION_KEY env var not set or invalid; deriving from SECRET_KEY (v1). "
+        "Set an explicit MFA_ENCRYPTION_KEY in production to enable key rotation."
+    )
+    hkdf = HKDF(
+        algorithm=hashes.SHA256(),
+        length=32,
+        salt=None,
+        info=b"mfa-totp-secret-v1",
+    )
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution of the Fernet key after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+# ---------------------------------------------------------------------------
+# TOTP secrets
+# ---------------------------------------------------------------------------
+
+
+def generate_totp_secret() -> str:
+    """Return a fresh base32 TOTP secret (32 chars)."""
+    return pyotp.random_base32()
+
+
+def encrypt_secret(secret_plain: str) -> str:
+    """Fernet-encrypt the base32 secret. Returns str for direct DB storage."""
+    token = _get_fernet().encrypt(secret_plain.encode("utf-8"))
+    return token.decode("utf-8")
+
+
+def decrypt_secret(secret_encrypted: str) -> Optional[str]:
+    """Decrypt a previously stored secret. Returns None when the token can't be
+    decrypted (e.g. key rotated without re-enroll). Never raises to the caller.
+    """
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt MFA secret (invalid Fernet token)")
+        return None
+    except Exception as exc:
+        logger.error("Unexpected error decrypting MFA secret: %s", exc)
+        return None
+
+
+def build_otpauth_uri(account_label: str, secret_plain: str, issuer: str = DEFAULT_ISSUER) -> str:
+    """Build an otpauth:// URI that all major authenticator apps accept.
+
+    Format: otpauth://totp/<issuer>:<account>?secret=<b32>&issuer=<issuer>&algorithm=SHA1&digits=6&period=30
+    """
+    issuer_q = quote(issuer, safe="")
+    label = f"{issuer}:{account_label}"
+    label_q = quote(label, safe=":@")
+    return (
+        f"otpauth://totp/{label_q}?secret={secret_plain}"
+        f"&issuer={issuer_q}&algorithm=SHA1&digits={TOTP_DIGITS}&period={TOTP_PERIOD}"
+    )
+
+
+def build_account_label(username: str, hostname_hint: Optional[str] = None) -> str:
+    """Compose the per-user otpauth label, respecting env > hostname > fallback."""
+    domain = (
+        os.getenv("MFA_ACCOUNT_LABEL_DOMAIN", "").strip()
+        or (hostname_hint or "").strip()
+        or ACCOUNT_LABEL_DOMAIN_FALLBACK
+    )
+    return f"{username}@{domain}"
+
+
+def verify_totp_with_replay_guard(
+    secret_plain: str,
+    code: str,
+    last_used_step: Optional[int],
+) -> Tuple[bool, Optional[int]]:
+    """Verify a 6-digit TOTP code with explicit per-step replay protection.
+
+    Returns (success, step_consumed). Caller persists the consumed step on success.
+
+    Implementation notes:
+      - pyotp.TOTP.at(seconds_since_epoch) — to target step N we pass step*PERIOD.
+      - secrets.compare_digest is used for constant-time comparison.
+      - Replay guard rejects codes whose step is <= the previously consumed step.
+    """
+    if not secret_plain or not code:
+        return (False, None)
+    code = code.strip()
+    if len(code) != TOTP_DIGITS or not code.isdigit():
+        return (False, None)
+
+    totp = pyotp.TOTP(secret_plain, digits=TOTP_DIGITS, interval=TOTP_PERIOD, digest=TOTP_DIGEST)
+    now = int(time.time())
+    current_step = now // TOTP_PERIOD
+
+    for offset in (0, -1, 1):
+        step = current_step + offset
+        expected = totp.at(step * TOTP_PERIOD)
+        if len(expected) == len(code) and _secrets.compare_digest(expected, code):
+            if last_used_step is not None and step <= last_used_step:
+                return (False, None)
+            return (True, step)
+    return (False, None)
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+def generate_backup_codes(count: int = BACKUP_CODE_COUNT) -> List[str]:
+    """Return ``count`` plain-text backup codes formatted as ``XXXX-YYYY``."""
+    codes: List[str] = []
+    for _ in range(count):
+        left = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        right = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        codes.append(f"{left}-{right}")
+    return codes
+
+
+def normalize_backup_code(user_input: str) -> str:
+    """Canonical form for comparison: uppercase, strip dashes/spaces."""
+    if not user_input:
+        return ""
+    return user_input.strip().upper().replace("-", "").replace(" ", "")
+
+
+async def _hash_one_backup_code(code_plain: str) -> str:
+    """Bcrypt-hash a single backup code on a worker thread."""
+    normalized = normalize_backup_code(code_plain)
+    hashed = await asyncio.to_thread(bcrypt.hashpw, normalized.encode("utf-8"), bcrypt.gensalt())
+    return hashed.decode("utf-8")
+
+
+async def hash_backup_codes(codes_plain: List[str]) -> List[str]:
+    """Hash backup codes in parallel (each bcrypt op runs in its own thread)."""
+    return await asyncio.gather(*(_hash_one_backup_code(c) for c in codes_plain))
+
+
+async def check_backup_code(user_input: str, code_hash: str) -> bool:
+    """Run a single bcrypt verify on the worker pool."""
+    normalized = normalize_backup_code(user_input)
+    if not normalized:
+        return False
+    return await asyncio.to_thread(
+        bcrypt.checkpw, normalized.encode("utf-8"), code_hash.encode("utf-8")
+    )
+
+
+# ---------------------------------------------------------------------------
+# Misc helpers
+# ---------------------------------------------------------------------------
+
+
+def generate_challenge_token() -> str:
+    """64-char hex challenge token for /api/auth/login → /mfa-verify hand-off."""
+    return _secrets.token_hex(32)

backend/tests/test_acme_diagnostics.py

@@ -433,7 +433,7 @@ async def test_check_agents_fail_when_none_registered():
 async def test_check_agents_warn_when_none_active():
     conn = AsyncMock()
     conn.fetch.return_value = [
-        {"id": 1, "hostname": "h1", "status": "offline", "last_heartbeat": None,
+        {"id": 1, "hostname": "h1", "status": "offline", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
     ]
     out = await check_agents(conn, [1])
@@ -444,9 +444,9 @@ async def test_check_agents_warn_when_none_active():
 async def test_check_agents_ok_with_active():
     conn = AsyncMock()
     conn.fetch.return_value = [
-        {"id": 1, "hostname": "h1", "status": "active", "last_heartbeat": None,
+        {"id": 1, "hostname": "h1", "status": "active", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
-        {"id": 2, "hostname": "h2", "status": "offline", "last_heartbeat": None,
+        {"id": 2, "hostname": "h2", "status": "offline", "last_seen": None,
          "cluster_id": 1, "cluster_name": "c1"},
     ]
     out = await check_agents(conn, [1])
@@ -454,6 +454,49 @@ async def test_check_agents_ok_with_active():
     assert "1 of 2" in out["message"]
 
 
+@pytest.mark.asyncio
+async def test_bulgu84_check_agents_uses_last_seen_column():
+    """Bulgu #84 (round-23 audit) — pin the SQL column name.
+
+    Pre-fix the query referenced a non-existent ``a.last_heartbeat``
+    column. AsyncMock returns whatever dict the test sets without
+    re-validating the SQL string, so the pre-fix test suite was
+    GREEN while every live ACME preflight call (the
+    ``POST /api/sites/preflight-acme`` endpoint that the wizard
+    runs before showing the ACME step) crashed with
+    ``UndefinedColumnError: column a.last_heartbeat does not
+    exist``. The crash blocked the entire wizard ACME preview
+    page on a production cluster, but unit tests never noticed
+    because they only assert on the helper's return shape, not
+    on the SQL string the helper sends to Postgres.
+
+    This pin inspects ``conn.fetch.call_args`` to assert the SQL
+    body actually queries ``a.last_seen`` (the canonical column
+    name used everywhere else in the codebase — see
+    routers/cluster.py:695-702 and routers/agent.py:281+ for
+    sibling readers). A future ``last_heartbeat`` typo would
+    re-fail this test without anyone noticing the live impact.
+    """
+    conn = AsyncMock()
+    conn.fetch.return_value = [
+        {"id": 1, "hostname": "h1", "status": "active", "last_seen": None,
+         "cluster_id": 1, "cluster_name": "c1"},
+    ]
+    await check_agents(conn, [1])
+    conn.fetch.assert_awaited_once()
+    sql_query = conn.fetch.call_args[0][0]
+    assert "a.last_seen" in sql_query, (
+        f"check_agents SQL must select a.last_seen (the canonical "
+        f"agents-table timestamp column); got: {sql_query!r}"
+    )
+    assert "a.last_heartbeat" not in sql_query, (
+        f"check_agents SQL still references a.last_heartbeat — this "
+        f"column does NOT exist on the agents table and the query "
+        f"will 500 with UndefinedColumnError at runtime. SQL: "
+        f"{sql_query!r}"
+    )
+
+
 # ----------------------------------------------------------------------------
 # run_checks orchestration
 # ----------------------------------------------------------------------------
@@ -506,3 +549,180 @@ async def test_run_checks_unknown_only_returns_empty():
         account_id=None, only=["bogus"],
     )
     assert out == []
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #94 / #95 (Round-25 audit) — diagnostic-runner robustness
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_swallows_single_check_crash(monkeypatch):
+    """Bulgu #94 — a single check exception must NOT collapse the suite.
+
+    Pre-fix, an asyncpg UndefinedColumnError from check_agents (e.g. the
+    Bulgu #84 ``a.last_heartbeat`` typo on an older deploy) propagated
+    up to the FastAPI router which had no `except`, so the operator saw
+    HTTP 500 with no body. The diagnostic panel is precisely the place
+    that should SURFACE this — never opaque-500 it. We now wrap each
+    check; the failing one becomes a structured `fail` row and the
+    other four still render.
+    """
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+
+    # check_routing + check_agents both call conn.fetch; explode on the
+    # FIRST call (which is check_routing) and return rows on the second.
+    call_count = {"n": 0}
+
+    async def _fetch(*args, **kwargs):
+        call_count["n"] += 1
+        if call_count["n"] == 1:
+            raise RuntimeError("simulated: column a.last_heartbeat does not exist")
+        return []
+
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # All five checks must still appear in the response.
+    ids = [c["id"] for c in out]
+    assert ids == ["dns", "port80", "routing", "account", "agents"]
+    routing = next(c for c in out if c["id"] == "routing")
+    assert routing["status"] == "fail"
+    assert "Diagnostic check crashed" in routing["message"]
+    assert routing["details"]["exception_type"] == "RuntimeError"
+    assert "last_heartbeat" in routing["details"]["exception_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_coerces_string_cluster_ids(monkeypatch):
+    """Bulgu #94 — cluster_ids stored as JSONB strings (legacy paths)
+    must not crash check_routing / check_agents with
+    ``invalid input syntax for type integer: "1"``."""
+    monkeypatch.setattr(socket, "gethostbyname_ex",
+                        lambda d: (d, [], ["10.0.0.1"]))
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    captured_args = []
+
+    async def _fetch(*args, **kwargs):
+        captured_args.append(args)
+        return []
+
+    conn = AsyncMock()
+    conn.fetch = _fetch
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com"],
+        cluster_ids=["1", "2", "garbage", None, 3],
+        account_id=None,
+    )
+    # The list passed to asyncpg should already be a pure-int list.
+    # check_routing is the first call that uses cluster_ids.
+    routing_call_args = [a for a in captured_args if "frontends" in a[0]]
+    assert routing_call_args, "check_routing should have queried frontends"
+    cluster_ids_arg = routing_call_args[0][1]
+    assert cluster_ids_arg == [1, 2, 3], (
+        f"cluster_ids must be coerced to ints before being passed to "
+        f"asyncpg's ::int[] cast; got: {cluster_ids_arg!r}"
+    )
+    assert all(c["status"] != "fail" or c["id"] != "routing"
+               for c in out
+               if c["id"] == "routing" and "Diagnostic check crashed" in (c.get("message") or "")
+              ), "routing should not have crashed on coerced cluster_ids"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_does_not_swallow_cancellation(monkeypatch):
+    """`_safe_check` must catch `Exception` but NOT `BaseException`.
+
+    asyncio.CancelledError is a BaseException (Python 3.8+) so it must
+    propagate out of `_safe_check` — otherwise a request that the
+    client cancelled mid-flight would silently keep running diagnostic
+    checks instead of unwinding cleanly. We hand `_safe_check` a coro
+    that raises CancelledError and assert it bubbles up.
+    """
+    from services.acme_diagnostics import _safe_check
+
+    async def _cancelled_coro():
+        raise asyncio.CancelledError()
+
+    with pytest.raises(asyncio.CancelledError):
+        await _safe_check("dns", "DNS resolution", _cancelled_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_safe_check_handles_keyboardinterrupt(monkeypatch):
+    """`_safe_check` must also not swallow `KeyboardInterrupt`."""
+    from services.acme_diagnostics import _safe_check
+
+    async def _interrupt_coro():
+        raise KeyboardInterrupt()
+
+    with pytest.raises(KeyboardInterrupt):
+        await _safe_check("dns", "DNS resolution", _interrupt_coro())
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_coerce_cluster_ids_handles_none():
+    """Coercion must handle None input without raising."""
+    from services.acme_diagnostics import _coerce_cluster_ids
+    assert _coerce_cluster_ids(None) == []
+    assert _coerce_cluster_ids([]) == []
+    assert _coerce_cluster_ids([1, 2, 3]) == [1, 2, 3]
+    assert _coerce_cluster_ids(["1", "2"]) == [1, 2]
+    assert _coerce_cluster_ids([1.5]) == [1]  # int() truncates floats
+    assert _coerce_cluster_ids(["abc", None, "5"]) == [5]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_run_checks_filters_invalid_domains(monkeypatch):
+    """Non-string entries in `domains` must not reach the DNS resolver."""
+    seen_domains = []
+
+    def fake_gethostbyname_ex(domain):
+        seen_domains.append(domain)
+        return (domain, [], ["10.0.0.1"])
+
+    monkeypatch.setattr(socket, "gethostbyname_ex", fake_gethostbyname_ex)
+
+    def _ctor(*args, **kwargs):
+        return _FakeSession(statuses=[200])
+    monkeypatch.setattr("aiohttp.ClientSession", _ctor)
+
+    conn = AsyncMock()
+    conn.fetch.return_value = []
+    conn.fetchrow.return_value = None
+
+    out = await run_checks(
+        conn,
+        domains=["a.example.com", None, "", 42, "b.example.com"],
+        cluster_ids=[1],
+        account_id=None,
+    )
+    # Both check_dns and check_port80 resolve DNS, so each valid domain
+    # may appear multiple times — but invalid entries (None, "", 42)
+    # must never reach the resolver.
+    assert set(seen_domains) == {"a.example.com", "b.example.com"}
+    assert None not in seen_domains
+    assert "" not in seen_domains
+    assert 42 not in seen_domains
+    dns_check = next(c for c in out if c["id"] == "dns")
+    assert dns_check["status"] == "ok"

backend/tests/test_acme_diagnostics_router_round25.py

@@ -0,0 +1,554 @@
+"""Router-level tests for the ACME diagnostic endpoints (Round-25 audit).
+
+These tests pin the contract introduced by Bulgu #94 / #95:
+
+* ``POST /api/letsencrypt/orders/{order_id}/diagnostics`` must NEVER return
+  HTTP 500 for an in-suite failure. Authentication / authorisation /
+  rate-limit / not-found errors still raise the appropriate 4xx, but any
+  unexpected exception during check execution is converted to HTTP 200
+  with a structured failure envelope so the UI can render the cause.
+
+* ``GET /api/letsencrypt/orders/{order_id}/events`` must NEVER return
+  HTTP 500 because of schema drift in ``user_activity_logs`` (the
+  original 500 cause: SELECTing a non-existent ``status`` column). A
+  partial failure is reported via ``meta.errors[]``.
+
+The tests use AsyncMock-based fake connections rather than spinning up a
+real Postgres so they run hermetically inside CI.
+"""
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from routers import acme_diagnostics as router_mod
+
+
+# ----------------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------------
+
+
+class _FakeUser(dict):
+    pass
+
+
+def _patch_auth_and_db(monkeypatch, conn, user_id=1):
+    """Patch the auth / db helpers used by both endpoints so the tests
+    don't have to construct a real FastAPI request stack."""
+    async def _fake_user(_auth):
+        return _FakeUser(id=user_id, username="t", email="t@x")
+
+    async def _fake_perm(_uid, *_a, **_k):
+        return True
+
+    async def _fake_get_conn():
+        return conn
+
+    async def _fake_close_conn(_c):
+        return None
+
+    async def _fake_rate_limit(*_a, **_kw):
+        return None
+
+    monkeypatch.setattr(router_mod, "get_current_user_from_token", _fake_user)
+    monkeypatch.setattr(router_mod, "check_user_permission", _fake_perm)
+    monkeypatch.setattr(router_mod, "get_database_connection", _fake_get_conn)
+    monkeypatch.setattr(router_mod, "close_database_connection", _fake_close_conn)
+    monkeypatch.setattr(router_mod, "_enforce_rate_limit", _fake_rate_limit)
+
+
+# ----------------------------------------------------------------------------
+# /diagnostics endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_load_order_crash_returns_envelope_not_500(monkeypatch):
+    """A DB crash during load_order must surface a 200 envelope, not 500.
+
+    Pre-fix, any RuntimeError between auth and run_checks bubbled out of
+    the bare ``try/finally`` block and FastAPI returned an opaque HTTP 500.
+    The Round-25 fix wraps load_order so the operator sees the cause
+    inside the diagnostic panel.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB connectivity loss")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.run_diagnostics(order_id=42, authorization="Bearer x")
+
+    assert out["order_id"] == 42
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["checks"][0]["status"] == "fail"
+    assert out["checks"][0]["id"] == "diagnostics_runner"
+    assert "simulated DB connectivity loss" in out["checks"][0]["message"]
+    assert out["meta"]["correlation_id"]
+    assert out["meta"]["error_stage"] == "load_order"
+    assert "humanized_error" in out
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_run_checks_crash_returns_envelope(monkeypatch):
+    """A crash inside run_checks (after order is loaded) must also stay 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated check orchestrator crash")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["status"] == "diagnostics_unavailable"
+    assert out["meta"]["error_stage"] == "run_checks"
+    assert out["meta"]["error_type"] == "RuntimeError"
+    assert "simulated check orchestrator crash" in out["meta"]["error_message"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_diagnostics_returns_meta_summary_on_success(monkeypatch):
+    """Successful diagnostics responses carry a meta summary the UI uses
+    to surface 'N checks failed, M warnings' without recomputing."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _fake_checks(*_a, **_kw):
+        return [
+            {"id": "dns", "label": "DNS", "status": "ok", "severity": "info", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "routing", "label": "Routing", "status": "fail", "severity": "error", "message": "", "details": {}, "duration_ms": 1},
+            {"id": "port80", "label": "Port 80", "status": "warn", "severity": "warn", "message": "", "details": {}, "duration_ms": 1},
+        ]
+
+    monkeypatch.setattr(router_mod, "run_checks", _fake_checks)
+
+    out = await router_mod.run_diagnostics(order_id=5, authorization="Bearer x")
+
+    assert out["meta"]["checks_total"] == 3
+    assert out["meta"]["checks_failed"] == 1
+    assert out["meta"]["checks_warn"] == 1
+    assert out["meta"]["correlation_id"]
+
+
+# ----------------------------------------------------------------------------
+# /events endpoint
+# ----------------------------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_missing_status_column_returns_partial_envelope(monkeypatch):
+    """Bulgu #95 — user_activity_logs lacks a `status` column.
+
+    Pre-fix, the SELECT pulled `status` directly and the endpoint
+    returned HTTP 500 for every order that had any correlated
+    user-activity rows. The Round-25 fix introspects the schema; here
+    we simulate a deployment with no `status` column AND a JOIN /
+    query that would otherwise crash — the endpoint must stay 200,
+    return whatever it could collect from acme_order_events, and
+    record the user_activity_logs section as degraded but recoverable.
+    """
+    conn = AsyncMock()
+
+    # _load_order
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    fetchval_calls = {"n": 0}
+
+    async def _fetchval(sql, *args):
+        fetchval_calls["n"] += 1
+        # First call: existence check for acme_order_events table
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    columns_no_status = [
+        {"column_name": "id"},
+        {"column_name": "user_id"},
+        {"column_name": "action"},
+        {"column_name": "resource_type"},
+        {"column_name": "resource_id"},
+        {"column_name": "details"},
+        {"column_name": "created_at"},
+        # NOTE: no "status" column — this is the canonical schema.
+    ]
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql and "user_activity_logs" in sql:
+            return columns_no_status
+        if "FROM acme_order_events" in sql:
+            return []  # empty timeline is fine for this test
+        if "FROM user_activity_logs" in sql:
+            # If the projection includes `status` we will fail loudly.
+            assert "status" not in sql, (
+                "SELECT must not include `status` when the column is absent; "
+                f"SQL was: {sql!r}"
+            )
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["order_id"] == 5
+    assert out["count"] == 0
+    assert out["meta"]["correlation_id"]
+    # No section errors expected — schema-aware projection silently
+    # adapted, the panel just got an empty event list.
+    assert out["meta"]["errors"] == []
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_acme_order_events_query_crash_returns_partial(monkeypatch):
+    """A crash in the acme_order_events sub-query must NOT kill the
+    whole endpoint — the user_activity_logs section should still run
+    and the failure must appear in meta.errors."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5,
+        "account_id": 1,
+        "status": "invalid",
+        "domains": '["a.example.com"]',
+        "cluster_ids": "[1]",
+        "error_detail": None,
+        "post_completion_actions": None,
+        "pending_apply_version_name": None,
+        "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+            ]
+        if "FROM acme_order_events" in sql:
+            raise RuntimeError("simulated acme_order_events index corruption")
+        if "FROM user_activity_logs" in sql:
+            return []
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert out["count"] == 0
+    error_sections = [e["section"] for e in out["meta"]["errors"]]
+    assert "acme_order_events" in error_sections
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_load_order_404_still_raises(monkeypatch):
+    """The 404 HTTPException raised by `_load_order` for an unknown order
+    must remain a 404 — the Round-25 envelope is only for *unexpected*
+    failures, not for client-supplied invalid order IDs."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    conn.fetchrow.return_value = None  # no order found
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.get_order_events(order_id=9999, authorization="Bearer x")
+    assert exc_info.value.status_code == 404
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_setup_crash_returns_check_envelope(monkeypatch):
+    """The single-check re-run path must also envelope, never 500.
+
+    Pre-fix the rerun handler used the same bare ``try/finally`` shape
+    as the suite POST. If `_load_order` / `_enforce_rate_limit` raised,
+    the operator clicking the row's "Re-run" button got an opaque
+    toast and the row never updated. Now the rerun handler returns a
+    `fail` check shaped the same way the table already renders, so
+    the row updates in place with the cause + correlation_id.
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = RuntimeError("simulated DB drop during rerun")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="dns", authorization="Bearer x",
+    )
+    assert out["order_id"] == 5
+    assert out["check"]["status"] == "fail"
+    assert out["check"]["id"] == "dns"
+    assert "simulated DB drop during rerun" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "setup"
+    assert out["meta"]["correlation_id"]
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_run_checks_crash_returns_check_envelope(monkeypatch):
+    """A crash inside run_checks during a re-run also stays 200."""
+    conn = AsyncMock()
+    conn.fetchrow.return_value = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+    _patch_auth_and_db(monkeypatch, conn)
+
+    async def _boom(*_a, **_kw):
+        raise RuntimeError("simulated run_checks failure")
+
+    monkeypatch.setattr(router_mod, "run_checks", _boom)
+
+    out = await router_mod.rerun_diagnostic_check(
+        order_id=5, check_id="agents", authorization="Bearer x",
+    )
+    assert out["check"]["id"] == "agents"
+    assert out["check"]["status"] == "fail"
+    assert "simulated run_checks failure" in out["check"]["message"]
+    assert out["meta"]["error_stage"] == "run_checks"
+
+
+@pytest.mark.asyncio
+async def test_bulgu94_rerun_invalid_check_id_still_400(monkeypatch):
+    """An unknown check_id must remain a 400, not an envelope. The
+    envelope is only for *unexpected* server-side failures, not for
+    client-supplied invalid identifiers."""
+    from fastapi import HTTPException
+
+    conn = AsyncMock()
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(HTTPException) as exc_info:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5, check_id="bogus", authorization="Bearer x",
+        )
+    assert exc_info.value.status_code == 400
+
+
+@pytest.mark.asyncio
+async def test_bulgu95_events_status_column_is_used_when_present(monkeypatch):
+    """If a deployment DID add a `status` column (e.g. via a private
+    schema extension), the projection picks it up and the resulting
+    severity reflects it."""
+    conn = AsyncMock()
+    order_row = {
+        "id": 5, "account_id": 1, "status": "invalid",
+        "domains": '["a.example.com"]', "cluster_ids": "[1]",
+        "error_detail": None, "post_completion_actions": None,
+        "pending_apply_version_name": None, "wizard_staged_until": None,
+        "created_by": 1,
+    }
+
+    async def _fetchval(sql, *args):
+        if "acme_order_events" in sql:
+            return True
+        return False
+
+    async def _fetchrow(sql, *args):
+        return order_row
+
+    captured_sql = {"ua": None}
+    from datetime import datetime, timezone
+
+    async def _fetch(sql, *args):
+        if "information_schema.columns" in sql:
+            return [
+                {"column_name": "id"}, {"column_name": "action"},
+                {"column_name": "resource_type"}, {"column_name": "resource_id"},
+                {"column_name": "details"}, {"column_name": "created_at"},
+                {"column_name": "status"},
+            ]
+        if "FROM acme_order_events" in sql:
+            return []
+        if "FROM user_activity_logs" in sql:
+            captured_sql["ua"] = sql
+            return [{
+                "id": 100,
+                "action": "letsencrypt.order.create",
+                "resource_type": "letsencrypt_order",
+                "resource_id": "5",
+                "details": '{"foo":"bar"}',
+                "created_at": datetime(2026, 5, 13, 19, 0, 0, tzinfo=timezone.utc),
+                "user_id": 7,
+                "status": "failure",
+            }]
+        return []
+
+    conn.fetchval = _fetchval
+    conn.fetchrow = _fetchrow
+    conn.fetch = _fetch
+    _patch_auth_and_db(monkeypatch, conn)
+
+    out = await router_mod.get_order_events(order_id=5, authorization="Bearer x")
+
+    assert "status" in captured_sql["ua"]
+    assert out["count"] == 1
+    ev = out["events"][0]
+    assert ev["source"] == "user_activity_log"
+    assert ev["severity"] == "warn"  # status="failure" → warn
+    assert ev["details"] == {"foo": "bar"}
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #96 (prod-canary follow-up): int4 overflow on order_id must NOT
+# leak the asyncpg DataError message ("invalid input for query argument
+# $1: ... value out of int32 range") into the operator-facing response
+# body. Same shape as the "row not found" path: clean HTTPException(404).
+# ----------------------------------------------------------------------------
+
+
+import asyncpg as _asyncpg  # noqa: E402 — late import so the test module
+# can still be collected even if asyncpg has changed its exception module.
+
+
+def _data_error(msg: str) -> _asyncpg.exceptions.DataError:
+    """Construct an asyncpg DataError that mirrors what Postgres returns
+    when a path-param order_id overflows int4. We can't easily build the
+    real instance from the binary protocol, so we synthesize one with the
+    same class so the router's `except asyncpg.exceptions.DataError`
+    branch is exercised."""
+    return _asyncpg.exceptions.DataError(msg)
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_diagnostics_int4_overflow_returns_clean_404(monkeypatch):
+    """``order_id`` outside the int4 range must surface as a clean 404,
+    NOT as a `diagnostics_unavailable` envelope leaking the asyncpg
+    DataError message ("query argument $1", "int32 range").
+    """
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 2147483648 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(
+            order_id=2_147_483_648,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    # The operator must see the canonical "not found" detail, NOT the
+    # raw asyncpg error message.
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+    assert "query argument" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_events_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the events endpoint — out-of-range order_id is a
+    clean 404, not a `meta.errors[]` envelope leaking SQL detail."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 9999999999 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.get_order_events(
+            order_id=9_999_999_999,
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_rerun_int4_overflow_returns_clean_404(monkeypatch):
+    """Same contract on the per-check rerun endpoint — out-of-range
+    order_id is a clean 404, not a structured ``check.fail`` envelope
+    leaking the asyncpg DataError message."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error(
+        "invalid input for query argument $1: 5000000000 (value out of int32 range)"
+    )
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.rerun_diagnostic_check(
+            order_id=5_000_000_000,
+            check_id="dns",
+            authorization="Bearer x",
+        )
+
+    assert excinfo.value.status_code == 404
+    assert "not found" in str(excinfo.value.detail).lower()
+    assert "int32" not in str(excinfo.value.detail).lower()
+
+
+@pytest.mark.asyncio
+async def test_bulgu96_load_order_dataerror_does_not_leak_correlation_envelope(monkeypatch):
+    """Belt-and-braces: even when the DataError carries other kinds of
+    invalid-input strings (e.g. type coercion failure on an int4
+    column), `_load_order` must still answer with the canonical 404
+    envelope and NOT route it through `_diagnostic_failure_envelope`
+    (which would surface the raw SQL detail to the UI)."""
+    conn = AsyncMock()
+    conn.fetchrow.side_effect = _data_error("invalid integer literal: 'NaN'")
+    _patch_auth_and_db(monkeypatch, conn)
+
+    with pytest.raises(router_mod.HTTPException) as excinfo:
+        await router_mod.run_diagnostics(order_id=123, authorization="Bearer x")
+
+    assert excinfo.value.status_code == 404
+    # No SQL detail leak
+    assert "invalid integer literal" not in str(excinfo.value.detail).lower()
+    assert "NaN" not in str(excinfo.value.detail)

backend/tests/test_acme_humanizer.py

@@ -81,6 +81,7 @@ def test_legacy_plain_string_with_brace_but_invalid_json_falls_back():
     ("urn:ietf:params:acme:error:rejectedIdentifier", "blacklisted",    "rejected"),
     ("urn:ietf:params:acme:error:serverInternal", "internal err",       "ACME server"),
     ("urn:ietf:params:acme:error:userActionRequired", "agree to ToS",   "User action"),
+    ("urn:ietf:params:acme:error:externalAccountRequired", "EAB required", "External Account Binding"),
 ])
 def test_known_problem_types_are_humanized(problem_type, detail_text, expected_title_contains):
     payload = json.dumps({"type": problem_type, "detail": detail_text, "status": 400})

backend/tests/test_acme_pydantic_validation.py

@@ -8,7 +8,31 @@ from pydantic import ValidationError
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from routers.letsencrypt import CertificateRequest
+from routers.letsencrypt import CertificateRequest, AccountCreate
+
+
+class TestAccountCreateEAB:
+    """Issue #35 follow-up: EAB HMAC key must be valid base64url; empty/None passes through
+    (falls back to global Settings) so non-EAB accounts (HTTP-01 / LE / Cloudflare) are unaffected."""
+
+    def test_no_eab_is_allowed(self):
+        acc = AccountCreate(email="a@b.com")
+        assert acc.eab_hmac_key is None and acc.eab_kid is None
+
+    def test_valid_base64url_hmac_accepted(self):
+        # urlsafe base64, unpadded and padded — both accepted.
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZGVmZ2g")
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZA==")
+
+    def test_invalid_base64_hmac_rejected(self):
+        # 5 base64 chars (count ≡ 1 mod 4) is undecodable — the exact shape that would otherwise
+        # make register_account's _b64url_decode raise a cryptic binascii error.
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="AAAAA")
+
+    def test_oversized_hmac_rejected(self):
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="A" * 600)
 
 
 class TestCertificateRequestDomains:

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_dns01.py

@@ -0,0 +1,220 @@
+"""Issue #35 — ACME DNS-01: focused unit tests for the pure logic (no DB/network).
+
+Covers the TXT-value math (RFC 8555 §8.4 — raw SHA-256 digest, base64url, NOT hex),
+the _acme-challenge record-name derivation (wildcard stripping), credential encryption
+round-trip + tamper handling, and the DNS provider registry/allow-list.
+"""
+import base64
+import hashlib
+import os
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-dns01-unit-tests")
+
+from services.acme_service import ACMEService
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import (
+    encrypt_dns_credentials, decrypt_dns_credentials, reset_fernet_for_tests,
+)
+
+
+def _b64url(b: bytes) -> str:
+    return base64.urlsafe_b64encode(b).rstrip(b"=").decode("ascii")
+
+
+def test_dns_txt_value_is_raw_sha256_base64url():
+    key_auth = "token123.thumbprintABC"
+    expected = _b64url(hashlib.sha256(key_auth.encode("utf-8")).digest())
+    assert ACMEService._dns_txt_value(key_auth) == expected
+    # Must NOT be the (classic-mistake) base64url of the HEX digest.
+    hex_based = _b64url(hashlib.sha256(key_auth.encode("utf-8")).hexdigest().encode("utf-8"))
+    assert ACMEService._dns_txt_value(key_auth) != hex_based
+
+
+def test_challenge_dns_name_derivation():
+    assert ACMEService._challenge_dns_name("example.com") == "_acme-challenge.example.com"
+    # Wildcard: the '*.' is stripped, so apex + wildcard share the SAME record name.
+    assert ACMEService._challenge_dns_name("*.example.com") == "_acme-challenge.example.com"
+    assert ACMEService._challenge_dns_name("foo.bar.example.com") == "_acme-challenge.foo.bar.example.com"
+
+
+def test_credential_encryption_roundtrip():
+    reset_fernet_for_tests()
+    creds = {"api_token": "super-secret-token-value"}
+    token = encrypt_dns_credentials(creds)
+    assert token != "super-secret-token-value"
+    assert "super-secret-token-value" not in token  # ciphertext, not plaintext
+    assert decrypt_dns_credentials(token) == creds
+
+
+def test_decrypt_invalid_token_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_dns_credentials("not-a-valid-fernet-token") is None
+
+
+def test_provider_registry_and_allow_list():
+    names = {p["name"] for p in list_providers()}
+    assert {"manual", "cloudflare"} <= names
+    assert is_supported("manual") and is_supported("cloudflare")
+    assert not is_supported("route53")  # not in MVP allow-list
+
+    assert get_provider("manual").automated is False
+    cf = get_provider("cloudflare", {"api_token": "x"})
+    assert cf.automated is True
+    assert any(f["key"] == "api_token" for f in cf.credential_fields)
+
+    raised = False
+    try:
+        get_provider("definitely-not-a-provider")
+    except ValueError:
+        raised = True
+    assert raised
+
+
+def test_cloudflare_token_sanitize():
+    # Issue #35 follow-up: a pasted token with quotes/spaces/control/unicode chars produced an
+    # invalid Authorization header (CF 6003 "Invalid request headers"). The sanitizer strips them.
+    from services.dns_providers.cloudflare import _sanitize_token, CloudflareDNSProvider
+
+    # Surrounding double quotes stripped.
+    assert _sanitize_token('"abc123-_def"') == 'abc123-_def'
+    # Interior spaces / tabs / newlines removed.
+    assert _sanitize_token('abc 123\tdef\n') == 'abc123def'
+    # A clean token68 string is unchanged (cannot corrupt a valid Cloudflare token).
+    clean = 'A1b2-_C3.d4~e5+f6/g7=='
+    assert _sanitize_token(clean) == clean
+    # Single quotes and a zero-width char removed.
+    assert _sanitize_token("'tok" + chr(0x200b) + "en'") == 'token'
+
+    # The provider constructor sanitizes into _token and keeps the raw input for diagnostics.
+    p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
+    assert p._token == 'my-token_123'
+    assert p._raw_token == '"my-token_123"'
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/tests/test_haproxy_validator_bulgu12.py

@@ -5328,7 +5328,22 @@ def test_bulgu62_update_path_grandfathers_unchanged_rule():
         fe, grandfathered_signatures=grand,
     )
     assert len(warnings) == 1
-    assert "grandfathered" in warnings[0].lower() or "pre-dated" in warnings[0].lower()
+    # Bulgu #83 (round-23 audit) — the warning was reworded from
+    # "Grandfathered ... pre-dated this validation" to a clearer
+    # "rule was not modified by this edit" phrasing that also
+    # echoes the verbatim rule body. Accept either the legacy
+    # markers or the new ones so the contract is signal-not-
+    # wording.
+    w_lower = warnings[0].lower()
+    assert (
+        "not modified by this edit" in w_lower
+        or "grandfathered" in w_lower
+        or "pre-dated" in w_lower
+    ), warnings[0]
+    # The verbatim rule must appear in the warning body so the
+    # operator can identify the offending entry without opening
+    # the ACL Builder cards.
+    assert stale in warnings[0], warnings[0]
 
 
 def test_bulgu62_update_path_still_rejects_new_contradiction():
@@ -6267,3 +6282,623 @@ def test_bulgu82_generate_install_script_validates_cluster():
     window = src[fn_start:fn_start + 5000]
     assert "validate_user_cluster_access" in window
     assert "Bulgu #82 (round-22 audit)" in window
+
+
+# ---- Bulgu #83 — grandfathered contradiction warning wording ----
+
+
+def test_bulgu83_warning_includes_verbatim_rule_text():
+    """The PUT-path grandfather warning must echo the verbatim
+    offending rule string so the operator can identify the
+    offender without opening the ACL Builder cards. Pre-fix the
+    warning only said "1 legacy rule" via the FE toast and
+    "Grandfathered <label> entry contains a self-contradictory
+    X !X condition that pre-dated this validation" via the server
+    JSON payload — both omitted the actual rule body, forcing
+    the operator to open the modal and hunt for the dead-code
+    entry."""
+    from models.frontend import FrontendConfig
+    from routers.frontend import (
+        _enforce_routing_rule_contradictions,
+        _rule_to_signature,
+    )
+
+    stale = "be-x if acl1 !acl1"
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        use_backend_rules=[stale],
+    )
+    warnings = _enforce_routing_rule_contradictions(
+        fe, grandfathered_signatures={_rule_to_signature(stale)},
+    )
+    assert len(warnings) == 1
+    # Verbatim rule body appears in the warning.
+    assert stale in warnings[0]
+    # The wording no longer uses the operator-unfriendly
+    # "Grandfathered" lead-in; the message states what the
+    # branch actually knows ("not modified by this edit").
+    assert "not modified by this edit" in warnings[0].lower()
+
+
+def test_bulgu83_dict_redirect_warning_includes_signature_snippet():
+    """Dict-shaped redirect rules with a contradictory `condition`
+    must still surface the offending signature in the warning
+    (truncated to 160 chars to bound the toast length). Pin the
+    truncation contract so a future refactor doesn't accidentally
+    grow the warning into a multi-KB blob."""
+    from models.frontend import FrontendConfig
+    from routers.frontend import (
+        _enforce_routing_rule_contradictions,
+        _rule_to_signature,
+    )
+
+    # Note: the redirect condition itself carries the contradiction;
+    # the dict wrapper is what the wizard emits.
+    bad_dict = {
+        "type": "scheme",
+        "scheme": "https",
+        "code": 301,
+        "condition": "if acl1 !acl1",
+    }
+    fe = FrontendConfig(
+        name="fe1",
+        bind_port=80,
+        mode="http",
+        redirect_rules=[bad_dict],
+    )
+    warnings = _enforce_routing_rule_contradictions(
+        fe, grandfathered_signatures={_rule_to_signature(bad_dict)},
+    )
+    assert len(warnings) == 1
+    # Truncated signature (up to 160 chars) must appear in body.
+    assert "acl1" in warnings[0]
+    assert "!acl1" in warnings[0]
+    assert "redirect_rules" in warnings[0]
+
+
+def test_bulgu83_static_marker_in_fe_warning_toast():
+    """Front-end pin — the FrontendManagement.js toast no longer
+    calls these rules "legacy" and now lists each offending rule
+    body. Static-source check so a refactor that re-introduces
+    the misleading wording or drops the rule snippets is caught.
+
+    Skipped automatically when the test runs inside the backend
+    Dockerfile build context (which only copies `backend/` and
+    therefore has no `frontend/` tree next to it). This mirrors
+    the guard already used by every other front-end static-source
+    pin in this file (e.g. lines 583-592, 795-810, 833-845, etc.)
+    — Bulgu #83's pin was missing it, which broke the corporate
+    CI's `RUN python -m pytest` step in the backend Docker build
+    immediately after Round-23 went live.
+    """
+    fm_path = (
+        _BACKEND_DIR.parent
+        / "frontend"
+        / "src"
+        / "components"
+        / "FrontendManagement.js"
+    )
+    if not fm_path.exists():
+        pytest.skip(
+            f"frontend not present at {fm_path}; running in backend-only "
+            "container is expected — skip JS source pin"
+        )
+    fm_src = fm_path.read_text()
+    assert "Bulgu #83 (round-23 audit)" in fm_src
+    # No more `legacy routing/redirect rule(s)` wording.
+    assert "legacy routing/redirect rule(s) with a self-contradictory" not in fm_src
+    # The new toast wires the offending rule snippets into the
+    # message body via `ruleSnippets`.
+    assert "ruleSnippets" in fm_src
+    # And it re-surfaces server-emitted warnings as a safety net.
+    assert "serverWarnings" in fm_src
+
+
+# ----------------------------------------------------------------------------
+# Bulgu #84 + #85 (round-23 audit — comprehensive Site Wizard live-test pass)
+# ----------------------------------------------------------------------------
+
+
+def test_bulgu84_acme_diagnostics_uses_last_seen_column():
+    """Bulgu #84 (round-23 audit) — duplicate pin in this file so the
+    full bulgu-pin suite has a one-stop reference; the original sits
+    in `test_acme_diagnostics.py::test_bulgu84_check_agents_uses_
+    last_seen_column` and asserts the AsyncMock call shape.
+
+    This duplicate is a STATIC-SOURCE check that's cheaper to grep:
+    we open the service file and verify the wrong column name is
+    not back in the SQL, plus the Bulgu marker is present so a
+    future refactor that drops the comment also fails.
+    """
+    svc_path = _BACKEND_DIR / "services" / "acme_diagnostics.py"
+    src = svc_path.read_text()
+    assert "Bulgu #84 (round-23 audit)" in src, (
+        "Bulgu #84 marker missing from acme_diagnostics.py — the "
+        "fix comment was removed but the underlying SQL change "
+        "may have been reverted too. Re-verify check_agents."
+    )
+    # Strip comments before scanning for the wrong column — the fix
+    # comment legitimately mentions ``a.last_heartbeat`` as the
+    # pre-fix symptom. We only care about NON-comment occurrences.
+    non_comment_lines = "\n".join(
+        ln for ln in src.splitlines() if not ln.lstrip().startswith("#")
+    )
+    # The canonical SELECT inside check_agents must use a.last_seen.
+    assert "SELECT a.id, a.hostname, a.status, a.last_seen" in non_comment_lines, (
+        "acme_diagnostics.check_agents must SELECT a.last_seen "
+        "(the canonical agents-table column). Pre-fix this used "
+        "a non-existent a.last_heartbeat and every preflight-acme "
+        "call 500'd in production."
+    )
+    assert "a.last_heartbeat" not in non_comment_lines, (
+        "acme_diagnostics still references a.last_heartbeat in non-"
+        "comment code — this column does NOT exist on the agents "
+        "table. Use a.last_seen."
+    )
+
+
+def test_bulgu87_wizard_create_endpoint_rate_limited():
+    """Bulgu #87 (round-23 audit) — `POST /api/sites` must be
+    rate-limited, matching the existing 5/min caps on the
+    /preview and /preflight-acme sibling endpoints.
+
+    Pre-fix the wizard's actual create endpoint had NO rate cap:
+      - /api/sites/preflight-acme  → rate-limited (site_acme_preflight)
+      - /api/sites/preview         → rate-limited (site_previewed)
+      - /api/sites (POST, create)  → UNLIMITED ← gap
+
+    Round-3 live testing fired 10 wizard creates inside one
+    second against demo-cluster1; the response codes were
+    200/409/409/...409/200/409... — no 429 ever fired. With
+    Bulgu #86 fixed (version_name collision), the UNIQUE-error
+    side-effect that ACCIDENTALLY rate-limited rapid-fire creates
+    disappears too, so an authenticated user/script can now
+    create entities until the cluster apply queue clogs.
+
+    The fix calls `_enforce_rate_limit(conn, user_id,
+    "wizard_create_site")` early in create_site (after auth +
+    cluster-access checks). The action name must match what the
+    router itself logs (`wizard_create_site` — see
+    `_log_user_activity(... action="wizard_create_site", ...)`
+    in the same router); the activity_logger middleware
+    intentionally skips this endpoint to avoid double-logging,
+    so the rate-limit must reference the router-emitted action.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #87 (round-23 audit)" in sw_src, (
+        "Bulgu #87 marker missing — the wizard create rate-limit "
+        "may have been reverted."
+    )
+    # The create_site function body must call _enforce_rate_limit
+    # with the canonical create action name.
+    import re as _re
+    m = _re.search(
+        r"async def create_site\([^)]*\)[^{]*:(.*?)(?=\nasync def |\n@router\.|\Z)",
+        sw_src,
+        _re.DOTALL,
+    )
+    assert m, "could not locate create_site function body in site_wizard.py"
+    fn_body = m.group(1)
+    assert (
+        '_enforce_rate_limit(conn, user_id, "wizard_create_site")' in fn_body
+        or "_enforce_rate_limit(conn, user_id, 'wizard_create_site')" in fn_body
+    ), (
+        "create_site must call _enforce_rate_limit(... "
+        '"wizard_create_site") so rapid-fire POST /api/sites '
+        "calls hit a 429 cap; pre-fix the endpoint was unlimited."
+    )
+
+
+def test_bulgu86_wizard_version_name_has_unique_suffix():
+    """Bulgu #86 (round-23 audit) — pin the version-name generation so
+    two wizard POST /api/sites calls submitted within the SAME epoch
+    second cannot collide on `config_versions(cluster_id, version_name)`.
+
+    Pre-fix `version_name = f"bulk-site-create-{int(time.time())}"`
+    had seconds resolution. Round-3 live testing reproduced the
+    failure with a back-to-back wizard automation: the second call
+    returned 409 with the misleading
+
+        "A wizard entity with this name already exists on the
+         cluster (UNIQUE constraint). Pick a different name."
+
+    fall-through detail, even though the operator-supplied
+    backend / frontend / SSL names were genuinely unique — the
+    real collision was on the auto-generated version name. The
+    operator who tried to rename the wizard inputs would still
+    hit the same error and have no way to make progress until the
+    epoch second ticked over.
+
+    Fix: append a 6-hex-char UUID suffix
+    (`bulk-site-create-{ts}-{uuid.uuid4().hex[:6]}`) so the
+    suffix space is 16M and birthday-collision-proof at any
+    realistic per-second request rate. The `bulk-site-create-`
+    prefix is preserved so reject_pending_changes /
+    restore-from-prior-version paths (cluster.py:4179 prefix scan)
+    keep working.
+
+    Static-source pin so any future refactor that strips the
+    suffix fails this test.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #86 (round-23 audit)" in sw_src, (
+        "Bulgu #86 marker missing — the version_name uniqueness "
+        "suffix may have been reverted."
+    )
+    # The fixed form references uuid.uuid4().hex[:6] (or similar
+    # length-bounded random suffix) appended to the ts. A pure
+    # `f"bulk-site-create-{ts}"` line (without any suffix
+    # interpolation after ts) is the regressed form.
+    import re as _re
+    # Find the wizard's version_name assignment. We allow flexible
+    # spacing / quoting around the f-string but require the suffix
+    # interpolation token immediately after `{ts}-`.
+    has_suffix = bool(_re.search(
+        r'version_name\s*=\s*f"bulk-site-create-\{ts\}-\{[^}]+\}"',
+        sw_src,
+    ))
+    has_regressed = bool(_re.search(
+        r'version_name\s*=\s*f"bulk-site-create-\{ts\}"\s*$',
+        sw_src,
+        _re.MULTILINE,
+    ))
+    assert has_suffix, (
+        "wizard's version_name must append a unique suffix "
+        "(e.g. uuid.uuid4().hex[:6]) after `{ts}` so back-to-back "
+        "wizard creates within the same epoch second cannot "
+        "collide on config_versions UNIQUE."
+    )
+    assert not has_regressed, (
+        "wizard's version_name regressed to seconds-only resolution "
+        "— this re-introduces Bulgu #86 (UNIQUE collision on rapid "
+        "back-to-back creates)."
+    )
+
+
+def test_bulgu85_unique_violation_handler_extracts_constraint_name():
+    """Bulgu #85 (round-23 audit) — the wizard's UniqueViolationError
+    handler must surface the offending constraint name in the operator-
+    visible 409 detail so debugging a "wizard entity with this name
+    already exists" toast doesn't require shell access to the server
+    logs.
+
+    Pre-fix the handler had THREE branches:
+      * backends_name_cluster_id_key  → "backend with this name"
+      * frontends_name_cluster_id_key → "frontend with this name"
+      * else                          → generic "wizard entity"
+    Any other constraint (SSL cert UNIQUE, backend_servers UNIQUE,
+    config_versions UNIQUE) fell through to the generic message,
+    leaving the operator with NO actionable hint and requiring an
+    on-call engineer to ssh into the API pod and tail `logger.info`
+    output to find ``constraint=<name>`` in the asyncpg traceback.
+
+    The fix:
+      1. Echoes the entity NAME (body.backend.name, body.frontend.name,
+         body.ssl.name) so the operator can match toast to wizard input.
+      2. Adds dedicated branches for `ssl_certificates_*` and
+         `backend_servers_*` constraints with actionable hints.
+      3. In the fall-through ELSE branch, echoes the constraint name
+         from `uve.constraint_name` (or substring scan as fallback)
+         so an unknown constraint still gives the engineer a stable
+         schema identifier to grep.
+    """
+    sw_src = (_BACKEND_DIR / "routers" / "site_wizard.py").read_text()
+    assert "Bulgu #85 (round-23 audit)" in sw_src, (
+        "Bulgu #85 marker missing from site_wizard.py UniqueViolation "
+        "handler — the constraint-echoing fix may have been reverted."
+    )
+    # Entity-name echoing in branch detail bodies.
+    assert "body.backend.name" in sw_src and (
+        "A backend named '{body.backend.name}'" in sw_src
+        or "A backend named '\"{body.backend.name}\"" in sw_src
+        or "A backend named '" in sw_src and "body.backend.name" in sw_src
+    )
+    assert "A frontend named '{body.frontend.name}'" in sw_src or (
+        "A frontend named '" in sw_src and "body.frontend.name" in sw_src
+    )
+    # SSL cert and backend_servers branches present.
+    assert "ssl_certificates" in sw_src
+    assert "backend_servers" in sw_src
+    # Fall-through echoes constraint name.
+    assert "constraint=" in sw_src or "constraint_name" in sw_src
+    # The asyncpg attribute is preferred over str(uve) scanning.
+    assert 'getattr(uve, "constraint_name"' in sw_src, (
+        "Fix should prefer asyncpg's structured constraint_name "
+        "attribute over substring scanning str(uve), which is the "
+        "human-formatted DETAIL line and may vary across server "
+        "versions / locales."
+    )
+
+
+# ----- Bulgu #88 / #89 (round-24 audit) ----------------------------------
+#
+# `POST /api/sites/preview` skipped the SSL-cert cluster-RBAC gate that
+# `POST /api/sites` (create) enforces via `select_existing_cert()`. The
+# `select_existing_cert()` helper encodes "global cert OR junction-bound
+# to <cluster_id>"; pre-fix the preview accepted ssl_certificate_id values
+# bound to OTHER clusters and returned the full rendered `would_create`
+# envelope — a tenant-boundary information leak for the cert id and the
+# rendered HAProxy config snippet. The fix mirrors the create-time check
+# in the preview path: same predicate, same 400 status, same error text.
+
+def test_bulgu88_preview_enforces_ssl_cert_cluster_rbac():
+    """Preview must reject `ssl.ssl_certificate_id` referencing a cert
+    that is bound to a DIFFERENT cluster — same RBAC predicate the
+    create endpoint enforces. Static source check: the preview body
+    invokes `select_existing_cert` with `body.ssl.ssl_certificate_id`
+    and raises 400 with a "not found / inactive / not bound" hint."""
+    import os
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+
+    # Locate the preview function.
+    assert "async def preview_create(" in sw_src
+    preview_start = sw_src.index("async def preview_create(")
+    # Bound the slice at the next top-level `async def` so we don't
+    # match the create_site copy (which is at line ~1900 and would
+    # cause this test to trivially pass even if preview was unfixed).
+    next_def = sw_src.index("\nasync def ", preview_start + 1)
+    preview_body = sw_src[preview_start:next_def]
+
+    # The preview must call select_existing_cert.
+    assert "select_existing_cert(" in preview_body, (
+        "Preview path must mirror create_site's cluster-RBAC gate by "
+        "invoking select_existing_cert(conn, ssl_certificate_id, "
+        "cluster_id). Pre-fix preview returned 200 with the full "
+        "rendered would_create envelope when the cert belonged to a "
+        "different cluster — a tenant-boundary information leak."
+    )
+
+    # The check must be wired to the body.ssl.ssl_certificate_id field
+    # (not some unrelated cert reference).
+    assert "body.ssl.ssl_certificate_id" in preview_body, (
+        "Preview RBAC check must reference body.ssl.ssl_certificate_id"
+    )
+
+    # Must raise 400 (consistent with create-time message).
+    assert "status_code=400" in preview_body
+    assert "not found / inactive / not bound" in preview_body or (
+        "not found / inactive" in preview_body
+        and "not bound" in preview_body
+    ), (
+        "Preview should raise 400 with the same 'not found / inactive "
+        "/ not bound to this cluster' hint create_site emits, so wizard "
+        "clients can match the message uniformly."
+    )
+
+
+def test_bulgu89_preview_enforces_per_server_ca_bundle_cluster_rbac():
+    """Preview must also enforce per-server `ssl_certificate_id` (CA
+    bundle) cluster-RBAC. Pre-fix only `create_site` checked this — a
+    direct API caller could submit a preview with
+    `servers[i].ssl_certificate_id=<cert from another cluster>` and
+    receive the full rendered backend block back, including the
+    `ca-file` directive bound to a cert id the caller's cluster has
+    no junction row for."""
+    import os
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+    preview_start = sw_src.index("async def preview_create(")
+    next_def = sw_src.index("\nasync def ", preview_start + 1)
+    preview_body = sw_src[preview_start:next_def]
+
+    # Must iterate body.servers and check per-server ssl_certificate_id.
+    assert "body.servers" in preview_body, (
+        "Preview must iterate body.servers to validate each "
+        "server.ssl_certificate_id reference"
+    )
+    assert "ssl_certificate_id" in preview_body
+    # The per-server message must be field-qualified so operators see
+    # WHICH server triggered the gate.
+    assert "servers[" in preview_body and "ssl_certificate_id=" in preview_body, (
+        "Per-server preview error must include the array index "
+        "(servers[i].ssl_certificate_id=<id>) so operators can find "
+        "the offending row in their wizard payload."
+    )
+
+
+# ----- Bulgu #90 (round-24 audit) ----------------------------------------
+#
+# `backend.cookie_name`, `backend.cookie_options`, and `server.cookie_value`
+# only rejected newline characters pre-fix. HAProxy's directive parser
+# tokenises by whitespace and treats `;` as an inline-comment, so a value
+# like `SESS'; DROP TABLE backends; --` rendered as
+# `cookie SESS'; DROP TABLE backends; -- insert indirect nocache` which
+# HAProxy parsed as `cookie SESS'` + comment — silently truncating the
+# operator's persistence options and producing a malformed Set-Cookie
+# header that browsers may drop. Tighten to the conservative
+# alphanumeric+`_.-` set (cookie_name / cookie_value) and the keyword/
+# `=`-bearing set (cookie_options).
+
+def test_bulgu90_cookie_name_rejects_haproxy_parser_hostile_chars():
+    """cookie_name must reject `;` (HAProxy inline comment), whitespace
+    (token delimiter), and other punctuation that breaks the rendered
+    `cookie <name>` directive."""
+    from models.site_wizard import BackendStep
+    import pydantic
+    # Semicolon (HAProxy comment) — used to be silently accepted.
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name="SESS'; DROP TABLE backends; --")
+    # Space (token delimiter) — splits the cookie directive.
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name="SESS ID")
+    # Quote (legal in RFC 6265 but produces ugly Set-Cookie headers
+    # and confuses log scrapers).
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(name="be1", cookie_name='SESS"id')
+    # Common-case valid input continues to work.
+    BackendStep(name="be1", cookie_name="SESS_ID-1.app")
+
+
+def test_bulgu90_cookie_options_rejects_parser_hostile_chars():
+    """cookie_options must reject `;` and quoting metacharacters but
+    still accept legitimate `attr SameSite=Lax` style values."""
+    from models.site_wizard import BackendStep
+    import pydantic
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(
+            name="be1", cookie_name="SESS",
+            cookie_options="insert indirect; rm -rf /",
+        )
+    with pytest.raises(pydantic.ValidationError):
+        BackendStep(
+            name="be1", cookie_name="SESS",
+            cookie_options='insert "indirect"',
+        )
+    # Valid HAProxy cookie options continue to parse.
+    BackendStep(
+        name="be1", cookie_name="SESS",
+        cookie_options="insert indirect nocache attr SameSite=Lax",
+    )
+
+
+def test_bulgu90_server_cookie_value_rejects_rfc6265_disallowed_chars():
+    """server.cookie_value must reject `;`, whitespace, and other
+    chars RFC 6265 disallows in cookie-values."""
+    from models.site_wizard import ServerStep
+    import pydantic
+    with pytest.raises(pydantic.ValidationError):
+        ServerStep(
+            server_name="s1", server_address="10.0.0.1",
+            server_port=8080, cookie_value="srv1; secure",
+        )
+    with pytest.raises(pydantic.ValidationError):
+        ServerStep(
+            server_name="s1", server_address="10.0.0.1",
+            server_port=8080, cookie_value="srv 1",
+        )
+    # Valid cookie values continue to work.
+    ServerStep(
+        server_name="s1", server_address="10.0.0.1",
+        server_port=8080, cookie_value="srv-1.app_2",
+    )
+
+
+# ----- Bulgu #93 (round-24 audit) ----------------------------------------
+#
+# `GET /api/sites/suggest` produced backend/frontend name suggestions
+# using `c.isalnum()` over the first domain label. Python's `isalnum()`
+# is Unicode-aware and returns True for non-ASCII letters (ü, é, ñ, …),
+# so an operator typing `bücher.example.com` received
+# `backend_name='be-bücher'`. The wizard CREATE path then rejected the
+# very name the SUGGEST endpoint returned, because the entity-name
+# regex (`^[a-zA-Z][a-zA-Z0-9_-]{0,63}$`) and the domain validator are
+# both ASCII-only. The fix converts non-ASCII labels through IDN/
+# punycode FIRST and only then applies ASCII-only sanitisation, so the
+# resulting name is one the operator can submit unchanged.
+
+def test_bulgu93_suggest_idn_unicode_produces_ascii_safe_slug():
+    """`suggest` must produce slugs that the backend/frontend name
+    regex `^[a-zA-Z][a-zA-Z0-9_-]{0,63}$` accepts. Static source check:
+    the suggest function uses an ASCII-only predicate
+    (`c.isascii() and c.isalnum()` etc.) and routes IDN labels through
+    `encode('idna')` to preserve the operator's intent in punycode."""
+    import os, re
+    sw_path = os.path.join(
+        os.path.dirname(__file__), "..", "routers", "site_wizard.py",
+    )
+    with open(sw_path, "r") as fh:
+        sw_src = fh.read()
+
+    # Locate the suggest_defaults function body.
+    assert "async def suggest_defaults(" in sw_src
+    start = sw_src.index("async def suggest_defaults(")
+    end = sw_src.index("\n@router.", start + 1)
+    suggest_body = sw_src[start:end]
+
+    # The fix must use IDN/punycode encoding when the input is non-ASCII.
+    assert 'encode("idna")' in suggest_body or "encode('idna')" in suggest_body, (
+        "suggest_defaults must route non-ASCII labels through IDN/"
+        "punycode so the rendered slug matches the ASCII-only entity-"
+        "name regex the create endpoint enforces."
+    )
+
+    # The sanitiser must use an ASCII-only predicate, not Unicode-aware
+    # `isalnum()` alone (which is the pre-fix bug).
+    assert "isascii()" in suggest_body, (
+        "Sanitisation step must explicitly bound to ASCII (via "
+        "`c.isascii()`) so Unicode alphabetics get mapped to '-' rather "
+        "than retained verbatim."
+    )
+
+    # The first occurrence of the Unicode-only `c.isalnum()` (without an
+    # adjacent isascii() guard) must no longer exist in the slug-build
+    # path. We sanity-check by counting the matches of the bare
+    # `c.isalnum()` token within the function body — pre-fix there was
+    # exactly one such occurrence in the slug builder.
+    bare = re.findall(r"\bc\.isalnum\(\)\b", suggest_body)
+    assert all(
+        # Each occurrence must be paired with an `isascii()` guard on
+        # the same line — confirm by inspecting the line context.
+        any("isascii()" in ln for ln in suggest_body.splitlines() if "c.isalnum()" in ln)
+        for _ in bare
+    ), (
+        "Every `c.isalnum()` predicate in the slug builder must be "
+        "AND'd with `c.isascii()` so non-ASCII letters cannot leak "
+        "into the suggested entity names."
+    )
+
+
+def test_bulgu93_suggest_function_behavioural_smoke():
+    """Behavioural check: the same logic, invoked directly, should
+    produce an ASCII-only slug for a Unicode input."""
+    # Replicate the post-fix slug builder inline so the test does not
+    # require an event loop / FastAPI fixture. The point is to confirm
+    # the algorithmic shape matches what the source check above
+    # enforces.
+    def _build_slug(domain: str) -> str:
+        first_label = (
+            domain.replace("*.", "").split(".")[0]
+            if domain.replace("*.", "")
+            else ""
+        )
+        ascii_label = first_label
+        if first_label and not first_label.isascii():
+            try:
+                ascii_label = first_label.encode("idna").decode("ascii")
+            except (UnicodeError, UnicodeDecodeError):
+                ascii_label = "".join(
+                    c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+                    else "-"
+                    for c in first_label
+                )
+        slug = (ascii_label or "newhost")[:32].lower()
+        slug = "".join(
+            c if c.isascii() and (c.isalnum() or c in ("-", "_"))
+            else "-"
+            for c in slug
+        )
+        if slug.startswith("_"):
+            slug = "h-" + slug.lstrip("_")
+        if not slug or not slug[0].isalpha():
+            slug = "h-" + slug
+        return slug
+
+    # Unicode IDN — should round-trip through punycode.
+    s = _build_slug("bücher.example.com")
+    # punycode of 'bücher' is 'xn--bcher-kva'
+    assert s == "xn--bcher-kva", f"expected xn--bcher-kva got {s!r}"
+    # The slug must satisfy the entity-name regex.
+    import re
+    assert re.fullmatch(r"^[a-zA-Z][a-zA-Z0-9_-]{0,63}$", s), s
+
+    # Plain ASCII domain still produces the obvious slug.
+    assert _build_slug("example.com") == "example"
+
+    # Uppercase normalises to lowercase.
+    assert _build_slug("EXAMPLE.COM") == "example"
+
+    # Trailing dot (FQDN absolute) stripped.
+    assert _build_slug("example.com.") == "example"
+
+    # Empty / dots-only input falls back to 'newhost'.
+    assert _build_slug("") == "newhost"
+    assert _build_slug("...") == "newhost"

backend/tests/test_keepalive_fault_state.py

@@ -0,0 +1,21 @@
+"""v1.7.6 — guard: the agent must surface keepalived FAULT state.
+
+A VIP whose interface has no usable IPv4 (or whose track-script fails) puts keepalived into
+FAULT — the virtual IP is NOT held. Previously get_keepalive_state only grepped (MASTER|BACKUP),
+so a FAULT'd VIP reported as BACKUP — misleading (looks healthy-ish). It must report FAULT so the
+UI shows it red. Both get_keepalive_state copies (installer + SKIP_TO_DAEMON daemon) must include
+FAULT. Pure source guard."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def test_keepalive_state_detects_fault_in_both_copies():
+    with open(os.path.join(ROOT, "utils", "agent_scripts", "linux_install.sh"), encoding="utf-8") as f:
+        s = f.read()
+    # FAULT added to the state grep in both copies (installer + daemon), both detection methods.
+    assert s.count("MASTER|BACKUP|FAULT") >= 2
+    # and the misleading MASTER|BACKUP-only grep is gone.
+    assert 'grep -oE "(MASTER|BACKUP)"' not in s

backend/tests/test_keepalived_config.py

@@ -0,0 +1,129 @@
+"""Issue #27 (v1.7.0) — unit tests for the keepalived config generator + secret crypto.
+
+Pure-function tests; no DB. Validates the rendered keepalived.conf for MASTER/BACKUP,
+unicast peers, the failover weight arithmetic, the script-security requirements, the
+ownership marker, and Fernet round-trip.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from services import keepalived_config as kc  # noqa: E402
+
+
+VIP = {
+    "id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+    "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True,
+}
+MEMBERS = [
+    {"role": "MASTER", "priority": 150, "network_interface": "eth0", "agent_id": 1, "ip_address": "10.0.0.11"},
+    {"role": "BACKUP", "priority": 100, "network_interface": "eth0", "agent_id": 2, "ip_address": "10.0.0.12"},
+]
+
+
+def _render(this_idx, auth="s3cr3t"):
+    this_agent = MEMBERS[this_idx]
+    peers = [m["ip_address"] for m in MEMBERS if m["agent_id"] != this_agent["agent_id"]]
+    return kc.render_keepalived_conf(vip=VIP, members=MEMBERS, this_agent=this_agent,
+                                     peer_ips=peers, auth_pass_plain=auth)
+
+
+class TestRender:
+    def test_master_state_priority_iface_vrid(self):
+        conf = _render(0)
+        assert "state MASTER" in conf
+        assert "priority 150" in conf
+        assert "interface eth0" in conf
+        assert "virtual_router_id 51" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+    def test_backup_state(self):
+        conf = _render(1)
+        assert "state BACKUP" in conf
+        assert "priority 100" in conf
+
+    def test_unicast_peers(self):
+        # MASTER's config lists the BACKUP as its unicast peer (and its own src ip).
+        conf = _render(0)
+        assert "unicast_src_ip 10.0.0.11" in conf
+        assert "unicast_peer" in conf
+        assert "10.0.0.12" in conf
+
+    def test_script_security_block(self):
+        conf = _render(0)
+        assert "enable_script_security" in conf
+        assert "script_user root" in conf
+
+    def test_ownership_marker(self):
+        assert kc.OWNERSHIP_MARKER in _render(0)
+
+    def test_weight_makes_failed_master_lose(self):
+        # On HAProxy failure the master's effective priority must drop below the backup.
+        conf = _render(0)
+        weight_line = [l for l in conf.splitlines() if l.strip().startswith("weight ")][0]
+        weight = int(weight_line.strip().split()[1])
+        assert 150 + weight < 100, "failed master must fall below every backup"
+        assert "track_script" in conf and "chk_haproxy" in conf
+
+    def test_track_disabled_omits_script(self):
+        vip = {**VIP, "track_haproxy": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain=None)
+        assert "vrrp_script" not in conf
+        assert "track_script" not in conf
+
+    def test_no_auth_when_secret_absent(self):
+        conf = _render(0, auth=None)
+        assert "auth_pass" not in conf
+
+    def test_multicast_omits_unicast(self):
+        vip = {**VIP, "use_unicast": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain="x")
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+
+    def test_single_node_omits_unicast_block(self):
+        # Single-node VIP (no peers): even with use_unicast=True we must NOT emit a bare
+        # `unicast_src_ip`/`unicast_peer` — keepalived treats a unicast keyword with no peers
+        # as deprecated, warns, and falls back to multicast (and `keepalived -t` flags it).
+        # Omitting the block yields a clean multicast config that holds the VIP solo.
+        only = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                 "agent_id": 1, "ip_address": "10.0.0.11"}]
+        conf = kc.render_keepalived_conf(vip=VIP, members=only, this_agent=only[0],
+                                         peer_ips=[], auth_pass_plain=None)
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+        assert "state MASTER" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+
+class TestCheckScript:
+    def test_default_process_name(self):
+        s = kc.build_haproxy_check_script()
+        assert "pidof haproxy" in s
+        assert kc.OWNERSHIP_MARKER in s
+
+    def test_process_name_from_bin_path(self):
+        s = kc.build_haproxy_check_script(bin_path="/opt/hap/sbin/haproxy-ent")
+        assert "pidof haproxy-ent" in s
+
+    def test_malicious_bin_path_falls_back(self):
+        s = kc.build_haproxy_check_script(bin_path="/x/haproxy; rm -rf /")
+        assert "rm -rf" not in s
+        assert "pidof haproxy" in s
+
+
+class TestSecretCrypto:
+    def test_roundtrip(self):
+        kc.reset_fernet_for_tests()
+        token = kc.encrypt_vrrp_secret("s3cr3t")
+        assert token != "s3cr3t"
+        assert kc.decrypt_vrrp_secret(token) == "s3cr3t"
+
+    def test_decrypt_garbage_returns_none(self):
+        kc.reset_fernet_for_tests()
+        assert kc.decrypt_vrrp_secret("not-a-fernet-token") is None

backend/tests/test_mfa_backwards_compat.py

@@ -0,0 +1,150 @@
+"""Backwards-compatibility regression tests for the MFA rollout (Issue #18).
+
+These tests don't hit a real database — they exercise the authoritative
+contract surfaces (login response shape, auth_middleware behaviour) using
+mocks where needed so the suite stays fast and deterministic.
+"""
+from __future__ import annotations
+
+import os
+import sys
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+class TestAuthMiddlewareUnchanged:
+    """auth_middleware MUST NOT look for MFA claims — Madde 2 of the plan."""
+
+    def test_decoder_imports_without_mfa_dependencies(self):
+        import auth_middleware
+        # The middleware's verification function exists and is callable.
+        assert callable(getattr(auth_middleware, "get_current_user_from_token", None))
+
+    def test_middleware_source_has_no_mfa_claim_check(self):
+        """The middleware source must not reference ``mfa`` claims directly."""
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "auth_middleware.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        # Allow incidental occurrences (e.g. comments); but never a claim lookup.
+        assert "payload.get('mfa'" not in source
+        assert 'payload.get("mfa"' not in source
+        assert "claims['mfa'" not in source
+        assert 'claims["mfa"' not in source
+
+
+class TestMfaModelsCoexistWithUserModels:
+    def test_models_user_module_unchanged_pydantic_shape(self):
+        from models import user as user_mod
+        # Ensure that User / UserUpdate / LoginRequest still load and still
+        # don't expose mfa-related fields (kept in models.mfa).
+        for cls in (user_mod.User, user_mod.UserUpdate, user_mod.LoginRequest):
+            fields = set(cls.model_fields.keys())
+            assert not {"mfa_enabled", "mfa_required", "mfa_token"} & fields, (
+                f"{cls.__name__} unexpectedly exposes MFA field; should stay byte-identical."
+            )
+
+    def test_models_mfa_module_exposes_expected_models(self):
+        from models import mfa as mfa_mod
+        for name in (
+            "MfaVerifyRequest",
+            "MfaEnrollStartResponse",
+            "MfaEnrollConfirmRequest",
+            "MfaEnrollConfirmResponse",
+            "MfaDisableRequest",
+            "MfaRegenerateBackupRequest",
+            "MfaRegenerateBackupResponse",
+            "MfaAdminResetRequest",
+            "MfaAdminResetAllRequest",
+            "MfaStatusResponse",
+        ):
+            assert hasattr(mfa_mod, name), f"Missing model: {name}"
+
+
+class TestRouterIncluded:
+    def test_main_includes_mfa_router(self):
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "main.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        assert "from routers.mfa import router as mfa_router" in source
+        assert "app.include_router(mfa_router)" in source
+
+
+class TestLoginResponseShapeForNonMfaUser:
+    """When MFA columns are missing or mfa_enabled=FALSE, /login returns the
+    pre-MFA response shape — no ``mfa_required`` / ``mfa_token`` keys leak through.
+    """
+
+    def test_login_without_mfa_returns_legacy_shape(self):
+        from fastapi.testclient import TestClient
+        from main import app
+
+        client = TestClient(app)
+
+        async def _fetch_mfa_state_none(conn, user_id):
+            return None
+
+        async def _no_log(*args, **kwargs):
+            return None
+
+        fake_user = {
+            "id": 1,
+            "username": "admin",
+            "email": "admin@example.com",
+            "password_hash": "$2b$12$placeholder",
+            "is_active": True,
+            "role": "admin",
+            "created_at": None,
+            "updated_at": None,
+            "last_login_at": None,
+        }
+
+        mock_conn = MagicMock()
+        mock_conn.fetchrow = AsyncMock(return_value=fake_user)
+        mock_conn.fetch = AsyncMock(return_value=[])
+        mock_conn.execute = AsyncMock(return_value=None)
+
+        async def _get_conn():
+            return mock_conn
+
+        async def _close(conn):
+            return None
+
+        with patch(
+            "routers.auth.get_database_connection", _get_conn
+        ), patch("routers.auth.close_database_connection", _close), patch(
+            "routers.auth._fetch_mfa_state", _fetch_mfa_state_none
+        ), patch("routers.auth.log_user_activity", _no_log), patch(
+            "bcrypt.checkpw", return_value=True
+        ):
+            resp = client.post(
+                "/api/auth/login",
+                json={"username": "admin", "password": "anything"},
+            )
+
+        assert resp.status_code == 200, resp.text
+        body = resp.json()
+        assert "access_token" in body
+        assert "token_type" in body
+        assert "expires_in" in body
+        assert "user" in body
+        assert "roles" in body
+        assert "permissions" in body
+        # CRITICAL — pre-MFA contract must not be polluted with MFA fields.
+        assert "mfa_required" not in body
+        assert "mfa_token" not in body
+        assert "methods" not in body

backend/tests/test_mfa_rate_limit_key.py

@@ -0,0 +1,200 @@
+"""Tests for middleware.mfa_rate_limit_key — user-aware + ingress-aware key."""
+import importlib
+from datetime import datetime, timedelta
+from typing import Dict, Optional
+
+import pytest
+from fastapi import Request
+from jose import jwt
+
+
+def _make_request(
+    headers: Optional[Dict[str, str]] = None,
+    peer: str = "127.0.0.1",
+) -> Request:
+    """Tiny ASGI scope shim — enough for the key_func surface."""
+    raw_headers = []
+    if headers:
+        raw_headers = [
+            (k.encode("latin-1"), v.encode("latin-1")) for k, v in headers.items()
+        ]
+    scope = {
+        "type": "http",
+        "headers": raw_headers,
+        "client": (peer, 12345),
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    return Request(scope)
+
+
+@pytest.fixture
+def reload_key(monkeypatch):
+    """Reload the key module so MFA_TRUSTED_PROXY_CIDRS is re-parsed."""
+
+    def _reload(**env):
+        for var in ("MFA_TRUSTED_PROXY_CIDRS",):
+            monkeypatch.delenv(var, raising=False)
+        for k, v in env.items():
+            monkeypatch.setenv(k, v)
+        from middleware import mfa_rate_limit_key as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+# ---------------------------------------------------------------------------
+# User-aware key extraction
+# ---------------------------------------------------------------------------
+
+
+def _mint_jwt(user_id, claim: str = "user_id") -> str:
+    """Mint a test JWT. Note: RFC 7519 says ``sub`` is a StringOrURI,
+    and python-jose validates that type when decoding, so callers that use
+    ``claim='sub'`` must pass a string user_id (matches production behavior
+    where auth_middleware also accepts string ``sub``)."""
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        claim: user_id,
+        "exp": datetime.utcnow() + timedelta(minutes=10),
+    }
+    return jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+
+def test_user_aware_via_user_id_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt(42, claim="user_id")
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:42"
+
+
+def test_user_aware_via_sub_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt("7", claim="sub")  # JWT spec: sub is a string
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:7"
+
+
+def test_no_auth_header_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_missing_bearer_prefix_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(headers={"authorization": "abc.def.ghi"}, peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_bearer_null_or_undefined_falls_back_to_ip(reload_key):
+    m = reload_key()
+    for bogus in ("null", "undefined", "", "   "):
+        req = _make_request(
+            headers={"authorization": f"Bearer {bogus}"}, peer="198.51.100.9"
+        )
+        assert m.mfa_rate_limit_key(req) == "ip:198.51.100.9"
+
+
+def test_tampered_jwt_falls_back_to_ip(reload_key):
+    """A token with a forged signature must NOT be honored — fallback to IP."""
+    m = reload_key()
+    bad = "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjogMTIzfQ.NOT_A_VALID_SIGNATURE"
+    req = _make_request(headers={"authorization": f"Bearer {bad}"}, peer="10.1.2.3")
+    assert m.mfa_rate_limit_key(req) == "ip:10.1.2.3"
+
+
+def test_expired_jwt_falls_back_to_ip(reload_key):
+    m = reload_key()
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        "user_id": 9,
+        "exp": datetime.utcnow() - timedelta(minutes=5),
+    }
+    expired = jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+    req = _make_request(headers={"authorization": f"Bearer {expired}"}, peer="10.0.0.7")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.7"
+
+
+# ---------------------------------------------------------------------------
+# Trusted-proxy X-Forwarded-For handling
+# ---------------------------------------------------------------------------
+
+
+def test_untrusted_peer_xff_is_ignored(reload_key):
+    """X-Forwarded-For from an untrusted client cannot move buckets."""
+    m = reload_key()  # no trusted CIDRs
+    req = _make_request(
+        headers={"x-forwarded-for": "1.2.3.4"},
+        peer="203.0.113.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_trusted_peer_xff_is_honored(reload_key):
+    """Peer in trusted CIDR → first XFF hop becomes the bucket."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(
+        headers={"x-forwarded-for": "203.0.113.42, 10.0.0.99"},
+        peer="10.0.0.99",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.42"
+
+
+def test_trusted_cidr_multiple_ranges(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8, 172.16.0.0/12")
+    req = _make_request(
+        headers={"x-forwarded-for": "198.51.100.4"},
+        peer="172.16.5.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:198.51.100.4"
+
+
+def test_trusted_peer_no_xff_falls_back_to_peer(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(peer="10.0.0.99")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.99"
+
+
+def test_invalid_cidr_in_env_is_logged_and_ignored(reload_key, caplog):
+    import logging
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limit_key"):
+        m = reload_key(MFA_TRUSTED_PROXY_CIDRS="not-a-cidr, 10.0.0.0/8")
+    assert any("ignoring invalid CIDR" in r.message for r in caplog.records)
+    # The valid one is still effective.
+    req = _make_request(
+        headers={"x-forwarded-for": "9.9.9.9"},
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:9.9.9.9"
+
+
+def test_user_bucket_wins_over_xff(reload_key):
+    """Auth always wins, even from a trusted proxy."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    token = _mint_jwt(99)  # integer user_id claim
+    req = _make_request(
+        headers={
+            "authorization": f"Bearer {token}",
+            "x-forwarded-for": "1.1.1.1",
+        },
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "user:99"
+
+
+def test_no_client_in_scope_does_not_crash(reload_key):
+    m = reload_key()
+    scope = {
+        "type": "http",
+        "headers": [],
+        "client": None,
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    req = Request(scope)
+    # Whatever it returns, it must be deterministic and not raise.
+    out = m.mfa_rate_limit_key(req)
+    assert out.startswith("ip:")

backend/tests/test_mfa_rate_limits.py

@@ -0,0 +1,93 @@
+"""Tests for middleware.mfa_rate_limits — env-driven MFA rate-limit config."""
+import importlib
+import logging
+
+import pytest
+
+
+@pytest.fixture
+def reload_module(monkeypatch):
+    """Helper: reload the module after env mutation so dataclass defaults
+    pick up the new values."""
+
+    def _reload(**env):
+        for key in list(globals().get('_OVERRIDDEN_ENVS', set())):
+            monkeypatch.delenv(key, raising=False)
+        for key, value in env.items():
+            monkeypatch.setenv(key, value)
+        from middleware import mfa_rate_limits as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+def test_defaults_when_no_env(reload_module, monkeypatch):
+    """No env var set → secure defaults applied (user-aware key assumption)."""
+    for key in (
+        "MFA_RATE_LIMIT_ENROLL_START",
+        "MFA_RATE_LIMIT_ENROLL_CONFIRM",
+        "MFA_RATE_LIMIT_DISABLE",
+        "MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES",
+        "MFA_RATE_LIMIT_ADMIN_RESET",
+        "MFA_RATE_LIMIT_ADMIN_RESET_ALL",
+    ):
+        monkeypatch.delenv(key, raising=False)
+    m = reload_module()
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert m.MFA_LIMITS.enroll_confirm == "10/minute"
+    assert m.MFA_LIMITS.disable == "10/minute"
+    assert m.MFA_LIMITS.regenerate_backup_codes == "5/hour"
+    assert m.MFA_LIMITS.admin_reset == "60/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "1/day"
+
+
+def test_env_override_per_endpoint(reload_module):
+    m = reload_module(
+        MFA_RATE_LIMIT_ENROLL_START="100/hour",
+        MFA_RATE_LIMIT_ADMIN_RESET_ALL="3/day",
+    )
+    assert m.MFA_LIMITS.enroll_start == "100/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "3/day"
+    # Untouched values still default.
+    assert m.MFA_LIMITS.disable == "10/minute"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "totally-bogus",
+        "5/lightyear",
+        "abc/minute",
+        "5",
+        "/minute",
+        "5//minute",
+        "",
+    ],
+)
+def test_invalid_format_falls_back_to_default(reload_module, caplog, bad):
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limits"):
+        m = reload_module(MFA_RATE_LIMIT_ENROLL_START=bad)
+    # Falls back to the secure default for enroll_start.
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert any("not a valid slowapi limit string" in r.message for r in caplog.records)
+
+
+def test_whitespace_around_value_is_tolerated(reload_module):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE="  30/minute  ")
+    assert m.MFA_LIMITS.disable == "30/minute"
+
+
+@pytest.mark.parametrize(
+    "valid",
+    ["1/second", "100/minute", "1000/hour", "10/day"],
+)
+def test_all_valid_periods_accepted(reload_module, valid):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE=valid)
+    assert m.MFA_LIMITS.disable == valid
+
+
+def test_dataclass_is_frozen(reload_module):
+    """Frozen dataclass guards against accidental mutation after import."""
+    m = reload_module()
+    with pytest.raises((AttributeError, Exception)):
+        m.MFA_LIMITS.disable = "999/second"  # type: ignore[misc]

backend/tests/test_mfa_service.py

@@ -0,0 +1,223 @@
+"""Unit tests for the MFA service layer (Issue #18, v1.6.0).
+
+These tests cover the pure-Python side of MFA — no DB, no FastAPI.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+import re
+import sys
+import time
+
+import pytest
+
+# Repo path setup (mirrors other tests in this folder).
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import pyotp  # noqa: E402
+
+from services import mfa_service  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# TOTP
+# ---------------------------------------------------------------------------
+
+
+class TestTotpSecret:
+    def test_secret_is_base32(self):
+        secret = mfa_service.generate_totp_secret()
+        # pyotp.random_base32() returns 32-character base32 strings.
+        assert len(secret) == 32
+        assert re.fullmatch(r"[A-Z2-7]+", secret), "secret must be valid base32"
+
+    def test_secrets_are_unique(self):
+        secrets = {mfa_service.generate_totp_secret() for _ in range(50)}
+        assert len(secrets) == 50
+
+
+class TestVerifyTotp:
+    def setup_method(self):
+        self.secret = mfa_service.generate_totp_secret()
+        self.totp = pyotp.TOTP(self.secret, digits=6, interval=30, digest="sha1")
+
+    def test_happy_path(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        assert step == int(time.time()) // 30
+
+    def test_invalid_code_format_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "abc", None)
+        assert ok is False and step is None
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "12345", None)
+        assert ok is False and step is None
+
+    def test_tolerance_minus_30s(self):
+        now = int(time.time())
+        previous_step = (now // 30) - 1
+        prev_code = self.totp.at(previous_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, prev_code, None)
+        assert ok is True
+        assert step == previous_step
+
+    def test_tolerance_plus_30s(self):
+        now = int(time.time())
+        next_step = (now // 30) + 1
+        next_code = self.totp.at(next_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, next_code, None)
+        assert ok is True
+        assert step == next_step
+
+    def test_replay_rejected(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        # Submit again with the previously-consumed step: must be rejected.
+        ok2, step2 = mfa_service.verify_totp_with_replay_guard(self.secret, code, step)
+        assert ok2 is False
+        assert step2 is None
+
+    def test_wrong_code_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "000000", None)
+        assert ok is False
+        assert step is None
+
+
+# ---------------------------------------------------------------------------
+# Fernet + key resolution
+# ---------------------------------------------------------------------------
+
+
+class TestFernet:
+    def setup_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def teardown_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def test_encrypt_decrypt_roundtrip_with_env_key(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        key = Fernet.generate_key().decode()
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", key)
+        mfa_service.reset_fernet_for_tests()
+
+        secret = "JBSWY3DPEHPK3PXP" * 2
+        token = mfa_service.encrypt_secret(secret)
+        assert token and token != secret
+        recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+
+    def test_decrypt_invalid_token_returns_none(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", Fernet.generate_key().decode())
+        mfa_service.reset_fernet_for_tests()
+
+        assert mfa_service.decrypt_secret("not-a-valid-fernet-token") is None
+
+    def test_hkdf_fallback_when_env_unset(self, monkeypatch, caplog):
+        monkeypatch.delenv("MFA_ENCRYPTION_KEY", raising=False)
+        mfa_service.reset_fernet_for_tests()
+
+        with caplog.at_level("WARNING"):
+            secret = "JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP"
+            token = mfa_service.encrypt_secret(secret)
+            recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+        assert any("MFA_ENCRYPTION_KEY" in r.message for r in caplog.records), (
+            "expected a WARN log when falling back to SECRET_KEY derivation"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+class TestBackupCodes:
+    def test_generate_count_and_format(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(codes) == 10
+        # 31-char alphabet: A-H J K M N P-Z 2-9 (excludes I, L, O, 0, 1).
+        for code in codes:
+            assert re.fullmatch(r"[A-HJKM-NP-Z2-9]{4}-[A-HJKM-NP-Z2-9]{4}", code), code
+
+    def test_alphabet_excludes_confusing_characters(self):
+        # Generate enough codes to virtually guarantee any forbidden char would surface.
+        for _ in range(20):
+            codes = mfa_service.generate_backup_codes()
+            for code in codes:
+                for ch in code.replace("-", ""):
+                    assert ch not in "0O1IL", f"forbidden char {ch!r} in {code!r}"
+
+    def test_codes_are_unique(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(set(codes)) == len(codes)
+
+    def test_normalize_strips_case_dash_space(self):
+        assert mfa_service.normalize_backup_code("abcd-efgh") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code(" ab cd-ef gh ") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code("") == ""
+        assert mfa_service.normalize_backup_code(None) == ""  # type: ignore[arg-type]
+
+    def test_hash_and_check_async(self):
+        async def _run():
+            plain = mfa_service.generate_backup_codes()[:1]
+            hashes = await mfa_service.hash_backup_codes(plain)
+            assert len(hashes) == 1
+            assert await mfa_service.check_backup_code(plain[0], hashes[0]) is True
+            assert await mfa_service.check_backup_code("WRONG-CODE", hashes[0]) is False
+            # Case + dash normalization
+            assert await mfa_service.check_backup_code(plain[0].lower(), hashes[0]) is True
+            assert (
+                await mfa_service.check_backup_code(plain[0].replace("-", ""), hashes[0])
+                is True
+            )
+
+        asyncio.run(_run())
+
+
+# ---------------------------------------------------------------------------
+# otpauth URI
+# ---------------------------------------------------------------------------
+
+
+class TestOtpAuthUri:
+    def test_uri_shape(self):
+        uri = mfa_service.build_otpauth_uri("alice@example.com", "JBSWY3DPEHPK3PXP")
+        assert uri.startswith("otpauth://totp/")
+        assert "secret=JBSWY3DPEHPK3PXP" in uri
+        assert "issuer=" in uri
+        assert "algorithm=SHA1" in uri
+        assert "digits=6" in uri
+        assert "period=30" in uri
+
+    def test_account_label_env_override(self, monkeypatch):
+        monkeypatch.setenv("MFA_ACCOUNT_LABEL_DOMAIN", "ops.example.com")
+        label = mfa_service.build_account_label("alice", hostname_hint="ignored.com")
+        assert label == "alice@ops.example.com"
+
+    def test_account_label_hostname_hint(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint="api.local")
+        assert label == "alice@api.local"
+
+    def test_account_label_fallback(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint=None)
+        assert label == "alice@haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Challenge token
+# ---------------------------------------------------------------------------
+
+
+class TestChallengeToken:
+    def test_length_and_uniqueness(self):
+        tokens = {mfa_service.generate_challenge_token() for _ in range(50)}
+        assert len(tokens) == 50
+        for t in tokens:
+            assert len(t) == 64
+            assert re.fullmatch(r"[0-9a-f]{64}", t)

backend/tests/test_vip_delete_approval.py

@@ -0,0 +1,69 @@
+"""Issue #27 safety (v1.7.2) — source-level guards for APPROVAL-GATED VIP deletion.
+
+The critical invariant: an agent must NEVER tear a VIP down without an explicit human approval.
+We enforce that by keeping the VIP is_active=TRUE (so the agent keeps serving it) when a delete
+is merely *requested*; only an APPROVE (apply) flips is_active=FALSE. These guards lock in that
+wiring so a future edit can't silently make delete immediate again. Pure (no DB)."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_delete_stages_for_approval_keeps_vip_running():
+    s = _read("routers/vip.py")
+    # A running (applied) VIP's delete sets pending_delete=TRUE and stages a delete version —
+    # it must NOT flip is_active=FALSE in delete_vip (that only happens on approval in apply_vip).
+    assert "pending_delete=TRUE" in s
+    assert '_stage_vip_version(conn, vip_id, "delete"' in s
+    # The staged-delete branch (running VIP) must not contain an is_active=FALSE soft-delete;
+    # only the "never applied" branch may remove immediately (guarded by applied_snapshot IS NULL).
+    assert 'v["applied_snapshot"] is None' in s
+
+
+def test_apply_performs_delete_only_on_approval():
+    s = _read("routers/vip.py")
+    # apply_vip short-circuits on pending_delete and only THEN flips is_active=FALSE.
+    assert 'if v["pending_delete"]:' in s
+    assert "is_active=FALSE, pending_delete=FALSE" in s
+
+
+def test_reject_cancels_delete_as_noop():
+    s = _read("routers/vip.py")
+    # reject_vip clears the staged delete + purge intent; the VIP keeps running (is_active never
+    # touched here) — the "nothing happened" path.
+    assert "pending_delete=FALSE, purge_on_teardown=FALSE" in s
+    assert "Deletion rejected" in s
+
+
+def test_agent_delivery_teardown_only_on_inactive():
+    # The agent is told to tear down ONLY when is_active=FALSE; a pending-delete VIP is still
+    # is_active=TRUE, so the delivery returns 'available' and the agent keeps serving it.
+    s = _read("routers/agent.py")
+    assert "if not row['is_active']:" in s
+    assert '"status": "teardown"' in s
+
+
+def test_migration_adds_pending_delete_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "pending_delete BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7
+
+
+def test_list_visibility_backward_compat_gates_on_applied():
+    # A VIP soft-deleted under the OLD immediate-delete (pre-1.7.2: is_active=FALSE,
+    # last_config_status='PENDING') must NOT reappear in the list as DELETING — the
+    # teardown-tracking clause is gated on last_config_status='APPLIED' (new-flow approved
+    # deletes only). Locks the backward-compat fix.
+    s = _read("routers/vip.py")
+    assert "v.last_config_status = 'APPLIED' AND EXISTS" in s

backend/tests/test_vip_diff_minimal.py

@@ -0,0 +1,34 @@
+"""v1.7.5 — guard for the HA/VIP "View Change" diff.
+
+Editing a VIP (e.g. priority + virtual IP) must show a REAL line diff (only the changed lines)
+against the previous applied vip-* config — not the whole keepalived.conf marked as "added", and
+without the doubled "+ +" prefix (the line must be stored WITHOUT a +/- prefix; the UI adds it
+from `type`, exactly like the standard haproxy diff). Pure source guard (the behavioural diff
+test needs a DB); locks the wiring so it can't regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _cluster_src() -> str:
+    with open(os.path.join(ROOT, "routers", "cluster.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_vip_diff_uses_real_difflib_against_previous_applied():
+    s = _cluster_src()
+    # The previous APPLIED vip-* config for this VIP is fetched as the diff baseline.
+    assert "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1" in s
+    # And the diff is a real unified_diff of old vs new (not "everything added").
+    assert "difflib.unified_diff(old_content.split('\\n'), new_content.split('\\n')" in s
+
+
+def test_vip_diff_stores_lines_without_prefix():
+    s = _cluster_src()
+    # The old vip bug prepended "+ {line}", doubling the UI prefix ("+ +"). The vip diff now
+    # stores the stripped diff line (dl[1:]) — the UI adds the +/- from `type`. `dl` is unique
+    # to the vip branch (the standard haproxy diff uses `line`), so this targets the vip fix
+    # only and does not touch the (separate, out-of-scope) SSL diff branch.
+    assert '"line": dl[1:], "line_number": line_number' in s

backend/tests/test_vip_models.py

@@ -0,0 +1,91 @@
+"""Issue #27 (v1.7.0) — unit tests for VIP request-model validation (pure, no DB)."""
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from models.vip import VIPCreate, VIPMemberIn  # noqa: E402
+
+
+def _members(master_prio=150, backup_prio=100, n_backup=1):
+    members = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": master_prio}]
+    for i in range(n_backup):
+        members.append({"agent_id": 2 + i, "network_interface": "eth0", "role": "BACKUP", "priority": backup_prio})
+    return members
+
+
+def _create(**over):
+    base = dict(name="web-vip", pool_id=1, virtual_ip="10.0.0.100", members=_members())
+    base.update(over)
+    return VIPCreate(**base)
+
+
+class TestField:
+    def test_ok(self):
+        v = _create()
+        assert v.virtual_ip == "10.0.0.100"
+        assert v.use_unicast is True  # cloud-safe default
+
+    def test_ipv6_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="fd00::1")
+
+    def test_bad_ip_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="not-an-ip")
+
+    def test_auth_pass_max_8(self):
+        _create(auth_pass="12345678")
+        with pytest.raises(ValueError):
+            _create(auth_pass="123456789")
+
+    def test_interface_forbidden_char(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0; rm -rf /", role="BACKUP", priority=100)
+
+    def test_priority_range(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0", role="BACKUP", priority=255)
+
+    def test_role_normalized(self):
+        m = VIPMemberIn(agent_id=1, network_interface="eth0", role="master", priority=150)
+        assert m.role == "MASTER"
+
+    def test_vrid_range(self):
+        with pytest.raises(ValueError):
+            _create(virtual_router_id=300)
+
+
+class TestMembers:
+    def test_single_node_allowed(self):
+        # A single-node VIP (one MASTER, no BACKUP) is valid: a keepalived-managed floating
+        # IP without failover (Issue #27 follow-up — relaxed from >=2 members to >=1, e.g. a
+        # one-box cluster that wants a stable VIP, or before a 2nd node is added for real HA).
+        v = VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5",
+                      members=[{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150}])
+        assert len(v.members) == 1 and v.members[0].role == "MASTER"
+
+    def test_needs_at_least_one(self):
+        # An empty membership is still rejected — a VIP must have at least one node.
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=[])
+
+    def test_exactly_one_master(self):
+        bad = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 2, "network_interface": "eth0", "role": "MASTER", "priority": 140}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=bad)
+
+    def test_master_must_be_highest(self):
+        with pytest.raises(ValueError):
+            _create(members=_members(master_prio=100, backup_prio=120))
+
+    def test_no_duplicate_agent(self):
+        dup = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 1, "network_interface": "eth1", "role": "BACKUP", "priority": 100}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=dup)

backend/tests/test_vip_purge.py

@@ -0,0 +1,51 @@
+"""Issue #27 follow-up (v1.7.2) — source-level guards for the opt-in keepalived package
+uninstall + the "we installed it" marker. Pure (no DB), mirroring the project's other
+source-assertion tests: they lock in the *safe defaults* so a future edit can't silently
+turn routine VIP deletion into a package purge, or purge a package we didn't install."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_agent_script_purge_is_optin_and_marker_guarded():
+    s = _read("utils/agent_scripts/linux_install.sh")
+    # The install marker is written only when WE install keepalived, and read by the purge
+    # guard — present in BOTH function copies (installer-mode + SKIP_TO_DAEMON).
+    assert s.count(".hom_installed") >= 4
+    # Multi-distro purge cascade exists, but ONLY inside the opt-in branch.
+    assert "apt-get purge -y -qq keepalived" in s
+    assert "apk del keepalived" in s
+    # Orphan self-heal (not_configured) must NEVER purge — graceful teardown only, both copies.
+    assert s.count('_kp_teardown "false"') >= 2
+    # Purge is honored only on an explicit teardown, parsed from the delivery response.
+    assert s.count(".purge // false") >= 2
+
+
+def test_agent_delivery_signals_purge_on_teardown():
+    s = _read("routers/agent.py")
+    assert "v.purge_on_teardown" in s
+    assert '"purge"' in s  # teardown response carries the opt-in flag
+
+
+def test_delete_endpoint_accepts_purge_package_default_off():
+    s = _read("routers/vip.py")
+    # Query param defaults to False (safe), and it sets the persisted teardown flag.
+    assert "purge_package: bool = False" in s
+    assert "purge_on_teardown=$2" in s
+
+
+def test_migration_adds_purge_column_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7

backend/tests/test_waf_stick_table_autoinject.py

@@ -0,0 +1,34 @@
+"""v1.7.4 — guard for the stick-table auto-inject in the HAProxy config renderer.
+
+A frontend that uses a stick counter (`track-sc<N>` or an `sc_*_rate(...)` fetch) but declares
+no `stick-table` makes HAProxy fatally reject the WHOLE cluster config with "table '<frontend>'
+used but not configured". This happens with rate-limit directives baked into a frontend's stored
+request_headers/options by an older version or a config import. The renderer now injects a default
+stick-table in that case. Pure source guard (the behavioural renderer test needs a DB and is
+disabled); this locks the wiring so it can't silently regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src() -> str:
+    with open(os.path.join(ROOT, "services", "haproxy_config.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_renderer_detects_stick_counter_usage():
+    s = _src()
+    # Detection: any track-sc<N> write OR an sc_*_rate(...) fetch flips _sc_counter_used.
+    assert "_sc_counter_used" in s
+    assert '"track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped)' in s
+
+
+def test_renderer_injects_stick_table_when_missing():
+    s = _src()
+    # Inject ONLY when a counter is used AND no stick-table was emitted (purely additive).
+    assert "if _sc_counter_used and not _stick_table_emitted:" in s
+    assert 'stick-table type ip size 100k expire 30s store http_req_rate(10s)' in s
+    # The inject must register the table so it is not added twice.
+    assert "_stick_table_emitted = True" in s

backend/utils/agent_scripts/linux_install.sh

@@ -153,7 +153,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -470,7 +470,12 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "haproxy-agent.service"; do
+# issue #31: match ONLY the installed agent (binary path + service), never the bare string
+# "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command line
+# or a sudo/PAM ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself ("Killed", install aborts). The systemd service is also stopped below; the
+# "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "haproxy-agent.service"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -635,7 +640,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -1055,7 +1060,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1289,7 +1299,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -1309,7 +1319,7 @@ get_keepalive_state() {
     # Method 1: journalctl (most reliable on RHEL/CentOS/Ubuntu with systemd)
     if command -v journalctl &>/dev/null; then
         state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-            | grep -oE "(MASTER|BACKUP)" | tail -1)
+            | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
     fi
 
     # Method 2: Fallback to log files (for non-systemd or restricted journalctl)
@@ -1317,22 +1327,27 @@ get_keepalive_state() {
         for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
             if [[ -r "$logfile" ]]; then
                 state=$(tail -200 "$logfile" 2>/dev/null \
-                    | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                    | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                 [[ -n "$state" ]] && break
             fi
         done
     fi
 
-    # Method 3: Check VIP presence on network interfaces (confirms MASTER)
+    # Method 3: VIP presence on local interfaces — the most portable signal, independent
+    # of logging/journald (works on any distro / init system / keepalived install type).
+    # MASTER iff ANY configured VIP is actually held locally; keepalived up but holding no
+    # VIP => BACKUP. Exact whole-line IP match (grep -Fxq) so 10.0.0.1 can't falsely match
+    # 10.0.0.10/100, and ALL configured VIPs are checked (not just the first).
     if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-        local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+        local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-            | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
+            | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
-        if [[ -n "$conf_vip" ]]; then
+        if [[ -n "$conf_vips" ]]; then
-            if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
+            local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
-                state="MASTER"
+            state="BACKUP"
-            else
+            local _cvip
-                state="BACKUP"
+            for _cvip in $conf_vips; do
-            fi
+                if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+            done
         fi
     fi
 
@@ -1352,7 +1367,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -1665,6 +1685,163 @@ check_ssl_updates() {
 }
 
 # Reload HAProxy service (Linux specific)
+# Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence. OPT-IN and inert by default:
+# a node with no APPLIED VIP gets status=not_configured and (lacking our ownership
+# marker) does NOTHING. Never clobbers a hand-managed keepalived. Uses $AGENT_TOKEN
+# (kept in sync with the live token in both daemon loops).
+fetch_and_deploy_keepalived_config() {
+    local marker="# Managed by HAProxy OpenManager"
+    local resp status http_code we_own="false" conf chk
+
+    # Timeouts so a hung management server can never stall the daemon loop.
+    resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+        "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    http_code="${resp##*$'\n'}"   # last line = HTTP status
+    resp="${resp%$'\n'*}"          # everything before = JSON body
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+    # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+    # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+    [[ "$http_code" == "404" ]] && status="teardown"
+    # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+    conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+    [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+    chk="$(dirname "$conf")/check_haproxy.sh"
+    if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+    _kp_report() {  # $1=state  $2=vip_id(or empty)  $3=hash  $4=message
+        local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+        curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+            >/dev/null 2>&1 || true
+    }
+    _kp_teardown() {
+        # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+        # keepalived and remove OUR config (the VIP is released) but KEEP the package — the safe
+        # enterprise default. Purge only when opted in AND we are the ones who installed it
+        # (the .hom_installed marker); never remove a package the admin pre-installed.
+        local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+        log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+        systemctl stop keepalived >/dev/null 2>&1
+        systemctl disable keepalived >/dev/null 2>&1
+        rm -f "$conf" "$chk"
+        if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+            log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+            fi
+            rm -f "$marker_inst"
+            if command -v keepalived >/dev/null 2>&1; then
+                _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+            else
+                _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+            fi
+        elif [[ "$purge" == "true" ]]; then
+            log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+            _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+        else
+            _kp_report "disabled" "" "" "torn down"
+        fi
+    }
+
+    case "$status" in
+        available) : ;;  # fall through to converge
+        teardown)
+            # Explicit server-side delete. Honor the operator's opt-in package purge (.purge);
+            # marker-guarded inside _kp_teardown, and a node we don't own (no marker conf) is a no-op.
+            local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+            [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"
+            return 0 ;;
+        not_configured)
+            [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only, never purge
+            return 0 ;;
+        *) return 0 ;;   # unknown / auth error → no-op
+    esac
+
+    # --- status == available: converge ---
+    local vip_id new_conf check_script install_if new_hash
+    vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+    new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+    new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+    check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+    install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+    [[ -z "$new_conf" ]] && return 0
+
+    # Ownership guard: never overwrite a keepalived.conf we don't own.
+    if [[ -f "$conf" && "$we_own" != "true" ]]; then
+        log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+        _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+        return 0
+    fi
+
+    # Hybrid install: install keepalived only if missing.
+    if ! command -v keepalived >/dev/null 2>&1; then
+        if [[ "$install_if" == "true" ]]; then
+            log "INFO" "KEEPALIVED: installing package..."
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+            fi
+        fi
+        if ! command -v keepalived >/dev/null 2>&1; then
+            log "ERROR" "KEEPALIVED: not installed/available"
+            _kp_report "error" "$vip_id" "" "keepalived not installed"
+            return 0
+        fi
+        # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+        # delete removes only OUR install, never an admin's pre-existing keepalived package.
+        mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+    fi
+
+    # Idempotency: skip write+reload when the on-disk content already matches.
+    if [[ -f "$conf" ]]; then
+        local cur_hash would_hash
+        cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+        if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            return 0
+        fi
+    fi
+
+    mkdir -p "$(dirname "$conf")"
+    if [[ -n "$check_script" ]]; then
+        printf '%s' "$check_script" > "$chk"
+        chown root:root "$chk" 2>/dev/null
+        chmod 0755 "$chk"   # root-owned, not world-writable — required by enable_script_security
+    fi
+    # Validate on a TEMP file and swap in only on success: a bad render must never land on
+    # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+    local tmp_conf="${conf}.hom.tmp"
+    printf '%s' "$new_conf" > "$tmp_conf"
+    chmod 0644 "$tmp_conf"
+    if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+        rm -f "$tmp_conf"
+        log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+        _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+        return 0
+    fi
+    mv -f "$tmp_conf" "$conf"
+    chown root:root "$conf" 2>/dev/null
+    chmod 0644 "$conf"
+    systemctl enable keepalived >/dev/null 2>&1
+    if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+        log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+        _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+    else
+        log "ERROR" "KEEPALIVED: reload/restart failed"
+        _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+    fi
+    return 0
+}
+
 reload_haproxy_service() {
     log "INFO" "Performing zero-downtime HAProxy configuration reload..."
     
@@ -2356,8 +2533,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence at the SSL cadence (inert for non-VIP nodes)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2789,28 +2970,31 @@ SYSTEM_INFO_EOF
 
         if command -v journalctl &>/dev/null; then
             state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-                | grep -oE "(MASTER|BACKUP)" | tail -1)
+                | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
         fi
 
         if [[ -z "$state" ]]; then
             for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
                 if [[ -r "$logfile" ]]; then
                     state=$(tail -200 "$logfile" 2>/dev/null \
-                        | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                        | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                     [[ -n "$state" ]] && break
                 fi
             done
         fi
 
+        # Method 3: VIP presence on local interfaces — portable, logging-independent.
+        # MASTER iff ANY configured VIP is held locally (exact whole-line match, all VIPs).
         if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-            local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+            local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-                | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
+                | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
-            if [[ -n "$conf_vip" ]]; then
+            if [[ -n "$conf_vips" ]]; then
-                if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
+                local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
-                    state="MASTER"
+                state="BACKUP"
-                else
+                local _cvip
-                    state="BACKUP"
+                for _cvip in $conf_vips; do
-                fi
+                    if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+                done
             fi
         fi
 
@@ -2849,7 +3033,15 @@ SYSTEM_INFO_EOF
         local keepalive_info=$(get_keepalive_state)
         local keepalive_state=$(echo "$keepalive_info" | cut -d'|' -f1)
         local keepalive_ip=$(echo "$keepalive_info" | cut -d'|' -f2)
-        
+
+        # Network interfaces (DAEMON) — reported so the HA/VIP create form can offer this
+        # node's real NICs. /sys/class/net is universal across distros (no iproute2 needed);
+        # exclude loopback here, and the UI further hides docker/veth/bridge interfaces. Empty
+        # -> "[]" (never "[\"\"]") so a node with no NICs doesn't surface a blank option.
+        local net_ifaces ni_json
+        net_ifaces=$(ls /sys/class/net 2>/dev/null | grep -vE '^lo$' | tr '\n' ',' | sed 's/,$//')
+        if [ -n "$net_ifaces" ]; then ni_json="[\"${net_ifaces//,/\",\"}\"]"; else ni_json="[]"; fi
+
         # Prepare heartbeat payload with all data
         local heartbeat_payload="{
             \"name\": \"$AGENT_NAME\",
@@ -2859,6 +3051,8 @@ SYSTEM_INFO_EOF
             \"platform\": \"$platform\",
             \"architecture\": \"$(uname -m)\",
             \"version\": \"{{AGENT_VERSION}}\",
+            \"capabilities\": [\"haproxy_management\", \"ssl_deployment\", \"config_reload\", \"systemd_service\", \"keepalived_management\"],
+            \"network_interfaces\": $ni_json,
             \"haproxy_status\": \"$haproxy_status\",
             \"haproxy_version\": \"$haproxy_version\",
             \"cluster_id\": $CLUSTER_ID,
@@ -3041,9 +3235,159 @@ CONFIG_RESPONSE_EOF
     HAPROXY_BIN="${HAPROXY_BIN_PATH}"
     HAPROXY_CONFIG="${HAPROXY_CONFIG_PATH}"
     log "INFO" "DAEMON: Initialized HAProxy paths - bin: $HAPROXY_BIN, config: $HAPROXY_CONFIG"
-    
+
+    # Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence (DAEMON copy). Inert by
+    # default: a node with no APPLIED VIP gets not_configured and (lacking our marker)
+    # does nothing. Never clobbers a hand-managed keepalived. $AGENT_TOKEN is synced to
+    # the live token at the top of each loop iteration below.
+    fetch_and_deploy_keepalived_config() {
+        local marker="# Managed by HAProxy OpenManager"
+        local resp status http_code we_own="false" conf chk
+
+        # Timeouts so a hung management server can never stall the daemon loop.
+        resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+            "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        http_code="${resp##*$'\n'}"   # last line = HTTP status
+        resp="${resp%$'\n'*}"          # everything before = JSON body
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+        # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+        # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+        [[ "$http_code" == "404" ]] && status="teardown"
+        # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+        conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+        [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+        chk="$(dirname "$conf")/check_haproxy.sh"
+        if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+        _kp_report() {
+            local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+            curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+                >/dev/null 2>&1 || true
+        }
+        _kp_teardown() {
+            # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+            # keepalived and remove OUR config (VIP released) but KEEP the package. Purge only when
+            # opted in AND we installed it (.hom_installed marker) — never an admin's package.
+            local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+            log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+            systemctl stop keepalived >/dev/null 2>&1
+            systemctl disable keepalived >/dev/null 2>&1
+            rm -f "$conf" "$chk"
+            if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+                log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+                fi
+                rm -f "$marker_inst"
+                if command -v keepalived >/dev/null 2>&1; then
+                    _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+                else
+                    _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+                fi
+            elif [[ "$purge" == "true" ]]; then
+                log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+                _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+            else
+                _kp_report "disabled" "" "" "torn down"
+            fi
+        }
+
+        case "$status" in
+            available) : ;;
+            teardown)
+                local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+                [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"   # explicit delete: honor opt-in purge
+                return 0 ;;
+            not_configured)
+                [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only
+                return 0 ;;
+            *) return 0 ;;
+        esac
+
+        local vip_id new_conf check_script install_if new_hash
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+        new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+        check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+        install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+        [[ -z "$new_conf" ]] && return 0
+
+        if [[ -f "$conf" && "$we_own" != "true" ]]; then
+            log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+            _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+            return 0
+        fi
+
+        if ! command -v keepalived >/dev/null 2>&1; then
+            if [[ "$install_if" == "true" ]]; then
+                log "INFO" "KEEPALIVED: installing package..."
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+                fi
+            fi
+            if ! command -v keepalived >/dev/null 2>&1; then
+                log "ERROR" "KEEPALIVED: not installed/available"
+                _kp_report "error" "$vip_id" "" "keepalived not installed"
+                return 0
+            fi
+            # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+            # delete removes only OUR install, never an admin's pre-existing keepalived package.
+            mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+        fi
+
+        if [[ -f "$conf" ]]; then
+            local cur_hash would_hash
+            cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+            if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                return 0
+            fi
+        fi
+
+        mkdir -p "$(dirname "$conf")"
+        if [[ -n "$check_script" ]]; then
+            printf '%s' "$check_script" > "$chk"
+            chown root:root "$chk" 2>/dev/null
+            chmod 0755 "$chk"
+        fi
+        # Validate on a TEMP file and swap in only on success: a bad render must never land on
+        # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+        local tmp_conf="${conf}.hom.tmp"
+        printf '%s' "$new_conf" > "$tmp_conf"
+        chmod 0644 "$tmp_conf"
+        if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+            rm -f "$tmp_conf"
+            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+            return 0
+        fi
+        mv -f "$tmp_conf" "$conf"
+        chown root:root "$conf" 2>/dev/null
+        chmod 0644 "$conf"
+        systemctl enable keepalived >/dev/null 2>&1
+        if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+            log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+            _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+        else
+            log "ERROR" "KEEPALIVED: reload/restart failed"
+            _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+        fi
+        return 0
+    }
+
     log "DEBUG" "DAEMON: Starting agent monitoring loop for $AGENT_NAME"
-    
+
     # Enhanced daemon loop with upgrade capability - EXACT MacOS COPY
     while true; do
         sleep 30
@@ -3208,6 +3552,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP (Keepalived) convergence at the SSL cadence (~2.5 min);
+        # inert for non-VIP nodes (status=not_configured + no ownership marker).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/agent_scripts/macos_install.sh

@@ -318,7 +318,11 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "com.haproxy.agent"; do
+# issue #31: match ONLY the installed agent (binary path + LaunchDaemon label), never the bare
+# string "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command
+# line or a sudo ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself. The "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "com.haproxy.agent"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -522,7 +526,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -867,7 +871,7 @@ SSL_SYNC_TIMESTAMP_FILE="/tmp/haproxy-agent-ssl-sync-${AGENT_NAME}"
 get_cluster_paths() {
     log "DEBUG" "Fetching cluster paths from management API"
     local cluster_response=$("$CURL_BIN" -k -s -X GET "$MANAGEMENT_URL/api/clusters" \
-        -H "Authorization: Bearer $AGENT_TOKEN")
+        -H "X-API-Key: $AGENT_TOKEN")
     
     if [[ $? -eq 0 ]]; then
         local _cfg _bin _sock
@@ -920,7 +924,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1164,13 +1173,39 @@ get_keepalive_state() {
     return 0
 }
 
+# Issue #27 — keepalived/VRRP is Linux-only. macOS agents cannot run keepalived, so they
+# can never be VIP members (the backend also withholds the keepalived_management capability,
+# so the UI warns at assign time). This is a safe no-op that, if a VIP is somehow mis-assigned
+# to a macOS node, reports a clear "unsupported on macOS" status instead of silently never
+# converging. It never installs/writes anything.
+fetch_and_deploy_keepalived_config() {
+    local resp status vip_id
+    resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    if [[ "$status" == "available" ]]; then
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+        curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+    fi
+    return 0
+}
+
 # Send heartbeat
 send_heartbeat() {
     local haproxy_status=$(get_haproxy_status)
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -2197,8 +2232,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence (no-op on macOS; reports unsupported if assigned)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2629,7 +2668,25 @@ SYSTEM_INFO_EOF
         echo "|"
         return 0
     }
-    
+
+    # Issue #27 — keepalived/VRRP is Linux-only (DAEMON version). Safe no-op that reports a
+    # clear "unsupported on macOS" status if a VIP is ever mis-assigned to a macOS node.
+    fetch_and_deploy_keepalived_config() {
+        local resp status vip_id
+        resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        if [[ "$status" == "available" ]]; then
+            vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+            log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+            curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+        fi
+        return 0
+    }
+
     # Send heartbeat (DAEMON version - includes all stats)
     send_heartbeat() {
         local haproxy_status=$(get_haproxy_status)
@@ -3021,6 +3078,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP convergence at the SSL cadence (no-op on macOS; reports
+        # unsupported if a VIP is mis-assigned to this node).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/dns_credentials.py

@@ -0,0 +1,88 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): at-rest encryption for per-account DNS provider credentials.
+
+Mirrors the established Fernet + HKDF(SECRET_KEY) pattern used for the VRRP secret
+(backend/services/keepalived_config.py) and TOTP secrets (backend/services/mfa_service.py):
+prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY env var (enables key rotation), else derive a
+stable key from SECRET_KEY via HKDF with a versioned info string.
+
+DNS provider credentials are a small dict (e.g. {"api_token": "..."}). They are JSON-serialized,
+encrypted to a Fernet token string for storage, and only ever decrypted in-process when a DNS-01
+order needs to talk to the provider. Plaintext credentials are NEVER logged or returned by the API.
+
+NOTE on key rotation: if SECRET_KEY rotates and DNS_PROVIDER_ENCRYPTION_KEY is not set, previously
+stored credentials become undecryptable (decrypt returns None). Callers MUST treat a None result as
+"credentials unavailable — re-enter in Settings" and surface a clear error, never a silent hang.
+"""
+from __future__ import annotations
+
+import base64
+import json
+import logging
+import os
+from typing import Dict, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so credentials survive restarts)."""
+    explicit = os.getenv("DNS_PROVIDER_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("DNS_PROVIDER_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    logger.warning(
+        "DNS_PROVIDER_ENCRYPTION_KEY not set; deriving the DNS-credentials encryption key from "
+        "SECRET_KEY. Set DNS_PROVIDER_ENCRYPTION_KEY to a Fernet key to enable key rotation."
+    )
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"dns-provider-creds-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_dns_credentials(credentials: Dict[str, str]) -> str:
+    """JSON-serialize and Fernet-encrypt a credentials dict to a storable token string."""
+    payload = json.dumps(credentials, separators=(",", ":")).encode("utf-8")
+    return _get_fernet().encrypt(payload).decode("utf-8")
+
+
+def decrypt_dns_credentials(token: str) -> Optional[Dict[str, str]]:
+    """Decrypt a stored token back to the credentials dict. Returns None if the token can't be
+    decrypted (e.g. key rotated) — callers must surface a clear 're-enter credentials' error."""
+    try:
+        plain = _get_fernet().decrypt(token.encode("utf-8")).decode("utf-8")
+        data = json.loads(plain)
+        if not isinstance(data, dict):
+            logger.error("Decrypted DNS credentials are not a JSON object")
+            return None
+        return data
+    except InvalidToken:
+        logger.warning("Failed to decrypt DNS provider credentials (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting DNS provider credentials: %s", exc)
+        return None

docker-compose.yml

@@ -51,6 +51,9 @@ services:
       - LOG_LEVEL=INFO
       - PUBLIC_URL=http://localhost:8080
       - MANAGEMENT_BASE_URL=http://localhost:8080
+      # Empty when unset on the host: the image CMD then falls back to
+      # WEB_CONCURRENCY (uvicorn's native env) and finally to 1.
+      - UVICORN_WORKERS=${UVICORN_WORKERS:-}
     volumes:
       - haproxy_configs:/etc/haproxy
     expose:
@@ -71,15 +74,17 @@ services:
 
   # Frontend React App - pulls from Docker Hub by default
   # To build locally instead, uncomment the build section and run: docker compose build frontend
+  #
+  # NOTE: REACT_APP_* env vars are BUILD-time only for Create-React-App. The
+  # runtime container (serve -s build) does NOT consume them. The frontend
+  # uses same-origin (window.location) for /api/* and is routed by the nginx
+  # service below to the backend container. No env vars are required here.
   frontend:
     image: taylanbakircioglu/haproxy-openmanager-frontend:latest
     # build:
     #   context: ./frontend
     #   dockerfile: Dockerfile
     container_name: haproxy-openmanager-frontend
-    environment:
-      - REACT_APP_API_URL=
-      - NODE_ENV=production
     expose:
       - "3000"
     depends_on:
@@ -94,7 +99,9 @@ services:
 
   # Nginx Reverse Proxy
   nginx:
-    image: nginx:alpine
+    # Pinned to a patched release for the nginx "poolslip" advisory
+    # (mainline <=1.31.0 affected; fixed in mainline 1.31.1+ / stable 1.30.2+).
+    image: nginx:1.31.1-alpine
     container_name: haproxy-openmanager-nginx
     volumes:
       - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro

docs/STANDALONE_INSTALL.md

@@ -163,6 +163,11 @@ serve -s build -l 3000
 
 Option B — copy to nginx (recommended, see next section).
 
+> **Important: use the nginx URL on port 8080 as your entry point.** The `serve` option above (port 3000) hosts
+> only the static UI; there is no `/api` backend behind it, so the login page renders but cannot actually log you
+> in. nginx (next section) serves the UI *and* proxies `/api` to the backend on a single port, so do your login and
+> everyday use at `http://<server-ip>:8080`.
+
 ### Create a systemd service for the frontend (if using `serve`)
 
 ```bash
@@ -268,10 +273,10 @@ curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8080/
 # Login
 curl -s -X POST http://127.0.0.1:8080/api/auth/login \
     -H "Content-Type: application/json" \
-    -d '{"username": "admin", "password": "admin"}' | python3 -m json.tool
+    -d '{"username": "admin", "password": "admin123"}' | python3 -m json.tool
 ```
 
-> **Default credentials**: `admin` / `admin` — change the password immediately after first login.
+> **Default credentials**: `admin` / `admin123` — change the password immediately after first login.
 
 ## 9. Firewall
 
@@ -290,8 +295,8 @@ sudo ufw allow 8080/tcp
 | Service | Port | URL |
 |---------|------|-----|
 | Backend API | 8000 | `http://127.0.0.1:8000/api/health` |
-| Frontend | 3000 | `http://127.0.0.1:3000` |
+| Frontend (static only) | 3000 | `http://127.0.0.1:3000` (UI only, no API; not the login URL) |
-| Nginx (unified) | 8080 | `http://your-server-ip:8080` |
+| Nginx (unified, entry point) | 8080 | `http://your-server-ip:8080` (use this) |
 | PostgreSQL | 5432 | local |
 | Redis | 6379 | local |
 

frontend/.dockerignore

@@ -0,0 +1,34 @@
+# Build artifacts (regenerated inside the multi-stage builder)
+build
+node_modules
+coverage
+
+# Local dev-only overrides — MUST be excluded so the host's
+# `.env.local` (e.g. REACT_APP_API_URL=http://localhost:8000) does NOT
+# bleed into a production bundle via build-time inline-replace.
+.env
+.env.local
+.env.*.local
+.env.development
+.env.development.local
+.env.test
+.env.test.local
+
+# VCS / IDE / OS noise
+.git
+.gitignore
+.vscode
+.idea
+.DS_Store
+*.log
+
+# Tests
+**/__tests__
+**/*.test.js
+**/*.test.jsx
+**/*.test.ts
+**/*.test.tsx
+
+# Misc
+README.md
+.eslintcache

frontend/package-lock.json

@@ -1,20 +1,22 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.5.0",
+  "version": "1.7.8",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "haproxy-openmanager-frontend",
-      "version": "1.5.0",
+      "version": "1.7.8",
+      "license": "AGPL-3.0-or-later",
       "dependencies": {
         "@ant-design/icons": "^5.0.0",
         "@monaco-editor/react": "^4.6.0",
         "ace-builds": "^1.23.4",
         "antd": "^5.2.0",
-        "axios": "^1.3.0",
+        "axios": "^1.16.0",
         "moment": "^2.29.0",
         "monaco-editor": "^0.36.0",
+        "qrcode.react": "^4.0.0",
         "react": "^18.2.0",
         "react-ace": "^10.1.0",
         "react-dom": "^18.2.0",
@@ -1500,9 +1502,9 @@
       }
     },
     "node_modules/@babel/plugin-transform-modules-systemjs": {
-      "version": "7.29.0",
+      "version": "7.29.4",
-      "resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.0.tgz",
+      "resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.4.tgz",
-      "integrity": "sha512-PrujnVFbOdUpw4UHiVwKvKRLMMic8+eC0CuNlxjsyZUiBjhFdPsewdXCkveh2KqBA9/waD0W1b4hXSOBQJezpQ==",
+      "integrity": "sha512-N7QmZ0xRZfjHOfZeQLJjwgX2zS9pdGHSVl/cjSGlo4dXMqvurfxXDMKY4RqEKzPozV78VMcd0lxyG13mlbKc4w==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -4652,27 +4654,6 @@
         "url": "https://github.com/sponsors/gregberge"
       }
     },
-    "node_modules/@testing-library/dom": {
-      "version": "10.4.1",
-      "resolved": "https://registry.npmjs.org/@testing-library/dom/-/dom-10.4.1.tgz",
-      "integrity": "sha512-o4PXJQidqJl82ckFaXUeoAW+XysPLauYI43Abki5hABd853iMhitooc6znOnczgbTYmEP6U6/y1ZyKAIsvMKGg==",
-      "dev": true,
-      "license": "MIT",
-      "peer": true,
-      "dependencies": {
-        "@babel/code-frame": "^7.10.4",
-        "@babel/runtime": "^7.12.5",
-        "@types/aria-query": "^5.0.1",
-        "aria-query": "5.3.0",
-        "dom-accessibility-api": "^0.5.9",
-        "lz-string": "^1.5.0",
-        "picocolors": "1.1.1",
-        "pretty-format": "^27.0.2"
-      },
-      "engines": {
-        "node": ">=18"
-      }
-    },
     "node_modules/@testing-library/jest-dom": {
       "version": "5.17.0",
       "resolved": "https://registry.npmjs.org/@testing-library/jest-dom/-/jest-dom-5.17.0.tgz",
@@ -6543,12 +6524,12 @@
       }
     },
     "node_modules/axios": {
-      "version": "1.14.0",
+      "version": "1.16.0",
-      "resolved": "https://registry.npmjs.org/axios/-/axios-1.14.0.tgz",
+      "resolved": "https://registry.npmjs.org/axios/-/axios-1.16.0.tgz",
-      "integrity": "sha512-3Y8yrqLSwjuzpXuZ0oIYZ/XGgLwUIBU3uLvbcpb0pidD9ctpShJd43KSlEEkVQg6DS0G9NKyzOvBfUtDKEyHvQ==",
+      "integrity": "sha512-6hp5CwvTPlN2A31g5dxnwAX0orzM7pmCRDLnZSX772mv8WDqICwFjowHuPs04Mc8deIld1+ejhtaMn5vp6b+1w==",
       "license": "MIT",
       "dependencies": {
-        "follow-redirects": "^1.15.11",
+        "follow-redirects": "^1.16.0",
         "form-data": "^4.0.5",
         "proxy-from-env": "^2.1.0"
       }
@@ -6922,9 +6903,9 @@
       "license": "MIT"
     },
     "node_modules/body-parser": {
-      "version": "1.20.4",
+      "version": "1.20.5",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.5.tgz",
-      "integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
+      "integrity": "sha512-3grm+/2tUOvu2cjJkvsIxrv/wVpfXQW4PsQHYm7yk4vfpu7Ekl6nEsYBoJUL6qDwZUx8wUhQ8tR2qz+ad9c9OA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -6936,7 +6917,7 @@
         "http-errors": "~2.0.1",
         "iconv-lite": "~0.4.24",
         "on-finished": "~2.4.1",
-        "qs": "~6.14.0",
+        "qs": "~6.15.1",
         "raw-body": "~2.5.3",
         "type-is": "~1.6.18",
         "unpipe": "~1.0.0"
@@ -10021,15 +10002,15 @@
       }
     },
     "node_modules/express": {
-      "version": "4.22.1",
+      "version": "4.22.2",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.22.1.tgz",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
-      "integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
+      "integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "accepts": "~1.3.8",
         "array-flatten": "1.1.1",
-        "body-parser": "~1.20.3",
+        "body-parser": "~1.20.5",
         "content-disposition": "~0.5.4",
         "content-type": "~1.0.4",
         "cookie": "~0.7.1",
@@ -10048,7 +10029,7 @@
         "parseurl": "~1.3.3",
         "path-to-regexp": "~0.1.12",
         "proxy-addr": "~2.0.7",
-        "qs": "~6.14.0",
+        "qs": "~6.15.1",
         "range-parser": "~1.2.1",
         "safe-buffer": "5.2.1",
         "send": "~0.19.0",
@@ -10145,9 +10126,9 @@
       "license": "MIT"
     },
     "node_modules/fast-uri": {
-      "version": "3.1.0",
+      "version": "3.1.2",
-      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz",
+      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz",
-      "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==",
+      "integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==",
       "dev": true,
       "funding": [
         {
@@ -10412,9 +10393,9 @@
       "license": "ISC"
     },
     "node_modules/follow-redirects": {
-      "version": "1.15.11",
+      "version": "1.16.0",
-      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.11.tgz",
+      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz",
-      "integrity": "sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ==",
+      "integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==",
       "funding": [
         {
           "type": "individual",
@@ -18237,10 +18218,19 @@
         "teleport": ">=0.2.0"
       }
     },
+    "node_modules/qrcode.react": {
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/qrcode.react/-/qrcode.react-4.2.0.tgz",
+      "integrity": "sha512-QpgqWi8rD9DsS9EP3z7BT+5lY5SFhsqGjpgW5DY/i3mK4M9DTBNz3ErMi8BWYEfI3L0d8GIbGmcdFAS1uIRGjA==",
+      "license": "ISC",
+      "peerDependencies": {
+        "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
+      }
+    },
     "node_modules/qs": {
-      "version": "6.14.2",
+      "version": "6.15.2",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.2.tgz",
-      "integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
+      "integrity": "sha512-Rzq0KEyX/w/tEybncDgdkZrJgVUsUMk3xjh3t5bv3S1HTAtg+uOYt72+ZfwiQwKdysThkTBdL/rTi6HDmX9Ddw==",
       "dev": true,
       "license": "BSD-3-Clause",
       "dependencies": {
@@ -20357,9 +20347,9 @@
       }
     },
     "node_modules/shell-quote": {
-      "version": "1.8.3",
+      "version": "1.8.4",
-      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz",
+      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.4.tgz",
-      "integrity": "sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw==",
+      "integrity": "sha512-VsC6n6vz1ihYYyZZwX7YZSF5l5x36ca17OC+a69h94YqB7X6XLwf+5MOgynYir2SLFUbl8gIYvBo8K8RoNQ6bQ==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -21325,24 +21315,6 @@
         }
       }
     },
-    "node_modules/tailwindcss/node_modules/yaml": {
-      "version": "2.8.3",
-      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.3.tgz",
-      "integrity": "sha512-AvbaCLOO2Otw/lW5bmh9d/WEdcDFdQp2Z2ZUH3pX9U2ihyUY0nvLv7J6TrWowklRGPYbB/IuIMfYgxaCPg5Bpg==",
-      "dev": true,
-      "license": "ISC",
-      "optional": true,
-      "peer": true,
-      "bin": {
-        "yaml": "bin.mjs"
-      },
-      "engines": {
-        "node": ">= 14.6"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/eemeli"
-      }
-    },
     "node_modules/tapable": {
       "version": "2.3.2",
       "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.2.tgz",
@@ -22207,7 +22179,7 @@
       }
     },
     "node_modules/wbuf": {
-      "version": "1.7.3",
+      "version": "1.7.8",
       "resolved": "https://registry.npmjs.org/wbuf/-/wbuf-1.7.3.tgz",
       "integrity": "sha512-O84QOnr0icsbFGLS0O3bI5FswxzRr8/gHwWkDlQFskhSPryQXvrTMxjxGP4+iWYoauLoBvfDpkrOauZ+0iZpDA==",
       "dev": true,

frontend/package.json

@@ -1,12 +1,13 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.5.0",
+  "version": "1.8.6",
   "description": "HAProxy Load Balancer Management UI",
+  "license": "AGPL-3.0-or-later",
   "dependencies": {
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-router-dom": "^6.8.0",
-    "axios": "^1.3.0",
+    "axios": "^1.16.0",
     "antd": "^5.2.0",
     "@ant-design/icons": "^5.0.0",
     "recharts": "^2.5.0",
@@ -16,7 +17,8 @@
     "@monaco-editor/react": "^4.6.0",
     "react-ace": "^10.1.0",
     "ace-builds": "^1.23.4",
-    "react-window": "^1.8.10"
+    "react-window": "^1.8.10",
+    "qrcode.react": "^4.0.0"
   },
   "devDependencies": {
     "@types/react": "^18.0.0",

frontend/src/App.js

@@ -5,6 +5,7 @@ import {
   DashboardOutlined,
   SettingOutlined,
   CloudServerOutlined,
+  DeploymentUnitOutlined,
   FileTextOutlined,
   GlobalOutlined,
   SafetyCertificateOutlined,
@@ -42,6 +43,7 @@ import PoolManagement from './components/PoolManagement';
 import Login from './components/Login';
 import ClusterSelector from './components/ClusterSelector';
 import ClusterManagement from './components/ClusterManagement';
+import VIPManagement from './components/VIPManagement';
 import Configuration from './components/Configuration';
 import APIDocumentation from './components/APIDocumentation';
 import IPInventory from './components/IPInventory';
@@ -156,6 +158,11 @@ const { Text } = Typography;
       icon: <CloudServerOutlined />,
       label: <Link to="/clusters">Clusters</Link>,
     },
+    {
+      key: '/ha-vip',
+      icon: <DeploymentUnitOutlined />,
+      label: <Link to="/ha-vip">HA / VIP</Link>,
+    },
     {
       key: '/pools',
       icon: <GlobalOutlined />,
@@ -466,6 +473,7 @@ function AppContent() {
               <Route path="/pools" element={<PoolManagement />} />
               <Route path="/settings" element={<Settings />} />
               <Route path="/clusters" element={<ClusterManagement />} />
+              <Route path="/ha-vip" element={<VIPManagement />} />
               <Route path="/ip-inventory" element={<IPInventory />} />
             </Routes>
           </Content>

frontend/src/components/AgentManagement.js

@@ -2378,7 +2378,7 @@ const AgentManagement = () => {
                         const element = document.createElement('a');
                         const file = new Blob([installScript], { type: 'text/plain' });
                         element.href = URL.createObjectURL(file);
-                        element.download = `install-haproxy-agent-${selectedPlatform}.sh`;
+                        element.download = `install-agent-${selectedPlatform}.sh`;
                         document.body.appendChild(element);
                         element.click();
                         document.body.removeChild(element);
@@ -2516,7 +2516,7 @@ const AgentManagement = () => {
                                     const element = document.createElement('a');
                                     const file = new Blob([uninstallScript], { type: 'text/plain' });
                                     element.href = URL.createObjectURL(file);
-                                    element.download = `uninstall-haproxy-agent-${selectedPlatform}.sh`;
+                                    element.download = `uninstall-agent-${selectedPlatform}.sh`;
                                     document.body.appendChild(element);
                                     element.click();
                                     document.body.removeChild(element);
@@ -3147,7 +3147,7 @@ const AgentManagement = () => {
                           const element = document.createElement('a');
                           const file = new Blob([deleteUninstallScript], { type: 'text/plain' });
                           element.href = URL.createObjectURL(file);
-                          element.download = `uninstall-haproxy-agent-${agentToDelete.platform || 'linux'}.sh`;
+                          element.download = `uninstall-agent-${agentToDelete.platform || 'linux'}.sh`;
                           document.body.appendChild(element);
                           element.click();
                           document.body.removeChild(element);

frontend/src/components/ApplyManagement.js

@@ -37,6 +37,7 @@ const ApplyManagement = () => {
     backends: [],
     waf_rules: [],
     ssl_certificates: [],
+    vips: [],
     total_count: 0
   });
   const [configVersions, setConfigVersions] = useState([]);
@@ -53,6 +54,8 @@ const ApplyManagement = () => {
   // Validation Error Modal state
   const [validationErrorModalVisible, setValidationErrorModalVisible] = useState(false);
   const [selectedValidationError, setSelectedValidationError] = useState(null);
+  // HA/VIP (Issue #27): VIP changes appear in the standard right-panel "Pending Versions"
+  // (vip-* config_versions) and use the standard "View Change" diff — no bespoke VIP modal.
 
   // Initial load on component mount
   useEffect(() => {
@@ -67,7 +70,7 @@ const ApplyManagement = () => {
   useEffect(() => {
     if (selectedCluster) {
       // CRITICAL: Clear all state immediately when cluster changes to prevent cross-cluster contamination
-      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], total_count: 0 });
+      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], vips: [], total_count: 0 });
       setConfigVersions([]);
       setAgentSync(null);
       setEntitySyncStates({});
@@ -77,7 +80,7 @@ const ApplyManagement = () => {
       fetchConfigVersions();
       fetchAgentSync();
     } else {
-      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], total_count: 0 });
+      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], vips: [], total_count: 0 });
       setConfigVersions([]);
       setAgentSync(null);
       setEntitySyncStates({});
@@ -125,26 +128,32 @@ const ApplyManagement = () => {
         'Pragma': 'no-cache'
       };
       
-      const [frontendsRes, backendsRes, wafRes, sslRes] = await Promise.all([
+      const [frontendsRes, backendsRes, wafRes, sslRes, vipsRes] = await Promise.all([
-        axios.get('/api/frontends', { 
+        axios.get('/api/frontends', {
           params: { cluster_id: selectedCluster.id, include_inactive: true },
           headers: cacheHeaders
         }).catch(() => ({ data: { frontends: [] } })),
-        
+
-        axios.get('/api/backends', { 
+        axios.get('/api/backends', {
           params: { cluster_id: selectedCluster.id, include_inactive: true },
           headers: cacheHeaders
         }).catch(() => ({ data: { backends: [] } })),
-        
+
-        axios.get('/api/waf/rules', { 
+        axios.get('/api/waf/rules', {
           params: { cluster_id: selectedCluster.id },
           headers: cacheHeaders
         }).catch(() => ({ data: { rules: [] } })),
-        
+
-        axios.get('/api/ssl/certificates', { 
+        axios.get('/api/ssl/certificates', {
           params: { cluster_id: selectedCluster.id },
           headers: cacheHeaders
-        }).catch(() => ({ data: [] }))
+        }).catch(() => ({ data: [] })),
+
+        // HA/VIP (Issue #27): pool-scoped, fetched cluster-scoped so it lists like other entities
+        axios.get('/api/vip', {
+          params: { cluster_id: selectedCluster.id },
+          headers: cacheHeaders
+        }).catch(() => ({ data: { vips: [] } }))
       ]);
 
       // DEBUG: Log raw backend data
@@ -180,14 +189,17 @@ const ApplyManagement = () => {
       
       const waf_rules = (wafRes.data.rules || []).filter(w => w.has_pending_config);
       const ssl_certificates = (sslRes.data.ssl_certificates || sslRes.data || []).filter(s => s.has_pending_config);
+      // VIP "pending" is its last_config_status (no has_pending_config flag).
+      const vips = (vipsRes.data.vips || []).filter(v => v.last_config_status === 'PENDING');
 
-      const total_count = frontends.length + backends.length + waf_rules.length + ssl_certificates.length;
+      const total_count = frontends.length + backends.length + waf_rules.length + ssl_certificates.length + vips.length;
 
       setPendingChanges({
         frontends,
         backends,
         waf_rules,
         ssl_certificates,
+        vips,
         total_count
       });
 
@@ -356,7 +368,7 @@ const ApplyManagement = () => {
       title: 'Apply All Configuration Changes',
       content: (
         <div>
-          <p>You are about to apply <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to apply <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -370,6 +382,15 @@ const ApplyManagement = () => {
             {pendingChanges.ssl_certificates.length > 0 && (
               <li><strong>{pendingChanges.ssl_certificates.length}</strong> SSL certificate changes</li>
             )}
+            {(pendingChanges.vips || []).length > 0 && (
+              <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
+            )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All changes will be applied together and sent to agents"
@@ -387,6 +408,74 @@ const ApplyManagement = () => {
     });
   };
 
+  // HA/VIP convergence tracking (Issue #27 follow-up). VIP teardown/deploy is asynchronous —
+  // member agents converge keepalived on their next poll. Mirror the HAProxy agent-sync widget
+  // so the progress popup keeps showing "Syncing HA/VIP... X/Y" until the nodes report the VIP
+  // ACTIVE, instead of flashing green while the HA/VIP page still shows SYNCING. Fire-and-forget
+  // recursive poll exactly like checkAgentSync (applyLoading is released immediately; this runs
+  // in the background and updates the floating widget). Bounded so an offline node can't poll
+  // forever — it then completes with an informational "still converging" note.
+  const trackVipConvergence = (clusterId, vipIds, startedAt) => {
+    const token = localStorage.getItem('token');
+    const total = vipIds.length;
+    const poll = async () => {
+      let vips = [];
+      try {
+        const r = await axios.get(`/api/vip?cluster_id=${clusterId}`, { headers: { Authorization: `Bearer ${token}` } });
+        vips = r.data.vips || [];
+      } catch (e) { /* transient — keep polling */ }
+      // A change has converged when the node reports the VIP ACTIVE (create/edit) OR the VIP is
+      // fully torn down (delete approval): deploy_status DELETED, or it has dropped off the list
+      // entirely once every member acked the teardown.
+      const isConverged = (vid) => {
+        const v = vips.find(x => x.id === vid);
+        if (!v) return true;                       // gone from the list → torn down / deleted
+        return v.deploy_status === 'ACTIVE' || v.deploy_status === 'DELETED';
+      };
+      const synced = vipIds.filter(isConverged).length;   // VIP-level (drives completion)
+      const errored = vips.filter(v => vipIds.includes(v.id)
+        && (v.deploy_status === 'ERROR' || v.deploy_status === 'ATTENTION')).length;
+      // Per-NODE progress: sum member acks across the tracked VIPs still present, so a multi-node
+      // VIP shows "1/2 node(s)" like the HA/VIP table — not just "1 change". Gone (torn-down) VIPs
+      // are already counted converged via isConverged.
+      let nodeTotal = 0, nodeSynced = 0;
+      for (const vid of vipIds) {
+        const v = vips.find(x => x.id === vid);
+        if (v) { nodeTotal += (v.deploy_total || 0); nodeSynced += (v.deploy_synced || 0); }
+      }
+      const label = nodeTotal > 0 ? `${nodeSynced}/${nodeTotal} node(s)` : `${synced}/${total} change(s)`;
+      const prog = Math.min(95, 60 + Math.round(35 * (nodeTotal > 0 ? nodeSynced / nodeTotal : synced / Math.max(1, total))));
+      updateEntityCounts(nodeTotal > 0 ? nodeSynced : synced, nodeTotal > 0 ? nodeTotal : total, 0, 0, 0);
+
+      if (synced === total) {
+        const msg = `${total} HA/VIP change(s) fully converged on all member node(s).`;
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.success(msg); }, 1500);
+        return;
+      }
+      if (errored > 0) {
+        const msg = 'HA/VIP applied, but a member node reported an issue — open Diagnostics on the HA / VIP page.';
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.warning(msg); }, 1800);
+        return;
+      }
+      if (Date.now() - startedAt > 300000) { // ~5 min cap (e.g. an offline member node)
+        const msg = `HA/VIP applied — ${label} converged; the rest are still converging (or a node's agent is offline). Track live status on the HA / VIP page.`;
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.info(msg); }, 2000);
+        return;
+      }
+      const step = `Syncing HA/VIP... ${label} converged (keepalived deploy can take a couple of minutes)`;
+      setSyncProgress({ visible: true, step, progress: prog });
+      updateProgress(step, prog, { 'Synced HA/VIP': label });
+      setTimeout(poll, 5000);
+    };
+    setTimeout(poll, 2000);
+  };
+
   const executeApplyAll = async () => {
     setApplyLoading(true);
     
@@ -429,13 +518,38 @@ const ApplyManagement = () => {
       
       // Detect if this is a restore operation (safe display-only check)
       const pendingVersions = configVersions.filter(v => v.status === 'PENDING');
-      const isRestoreOperation = totalEntities === 0 && pendingVersions.some(v => v.version_name.startsWith('restore-'));
+      // HA/VIP (Issue #27): vip-* versions are VIP-owned and excluded from the HAProxy apply
-      
+      // (cluster.py). Ignore them when deciding whether this apply has any HAProxy/cluster
+      // work to track — otherwise a VIP-only apply (which now stages a vip-* version) would
+      // look like it has a pending version and the agent-sync tracker would hang on 0/0.
+      const nonVipPendingVersions = pendingVersions.filter(v => !(v.version_name || '').startsWith('vip-'));
+      const isRestoreOperation = totalEntities === 0 && nonVipPendingVersions.some(v => v.version_name.startsWith('restore-'));
+      // HA/VIP-only apply: no HAProxy entity or config-version goes through the cluster-sync
+      // pipeline, so complete promptly after the isolated VIP apply (member nodes converge
+      // async on their next agent poll) instead of looping on "Entities: 0/0".
+      const vipCount = (pendingChanges.vips || []).length;
+      const isVipOnly = totalEntities === 0 && !isRestoreOperation && nonVipPendingVersions.length === 0 && vipCount > 0;
+      // Config-version-only apply (e.g. cluster ACME enable/disable): no entity rows, not a restore,
+      // but there ARE non-vip config versions to push. Without this branch it falls to the "else" and
+      // shows a misleading "Entities: 0/0" while still syncing agents.
+      const isConfigVersionOnly = totalEntities === 0 && !isRestoreOperation && vipCount === 0 && nonVipPendingVersions.length > 0;
+
       if (isRestoreOperation) {
         // Restore operation: Show "Configuration" instead of "Entities"
         setSyncProgress({ visible: true, step: `Applying configuration restore... Configuration: 0/1, Agents: ⏳`, progress: 20 });
         startProgress('apply', `Applying configuration restore... Configuration: 0/1, Agents: ⏳`);
         updateEntityCounts(0, 1, 0, totalAgents, disabledAgents); // Show 1 configuration item
+      } else if (isVipOnly) {
+        // HA/VIP-only: avoid the "Entities: 0/0" / agent-sync widget entirely.
+        setSyncProgress({ visible: true, step: `Applying ${vipCount} HA/VIP change(s)...`, progress: 20 });
+        startProgress('apply', `Applying ${vipCount} HA/VIP change(s)...`);
+        updateEntityCounts(0, vipCount, 0, 0, 0);
+      } else if (isConfigVersionOnly) {
+        // Config-version-only (ACME toggle, etc.): show "Configuration" instead of "Entities: 0/0".
+        const cfgCount = nonVipPendingVersions.length;
+        setSyncProgress({ visible: true, step: `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`, progress: 20 });
+        startProgress('apply', `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`);
+        updateEntityCounts(0, cfgCount, 0, totalAgents, disabledAgents);
       } else {
         // Normal operation: Show "Entities" as usual
         setSyncProgress({ visible: true, step: `Applying configuration changes... Entities: 0/${totalEntities}, Agents: ⏳`, progress: 20 });
@@ -445,16 +559,61 @@ const ApplyManagement = () => {
     
     try {
       const token = localStorage.getItem('token');
-      
+
-      const response = await axios.post(
+      // HA/VIP (Issue #27): apply pending VIPs first (isolated endpoint; safe no-op for
-        `/api/clusters/${selectedCluster.id}/apply-changes`,
+      // HAProxy config). Done here so VIPs apply even when there are no HAProxy changes.
-        {},
+      const pendingVips = pendingChanges.vips || [];
-        { headers: { Authorization: `Bearer ${token}` } }
+      for (const vip of pendingVips) {
-      );
+        try {
-      
+          await axios.post(`/api/vip/${vip.id}/apply`, {}, { headers: { Authorization: `Bearer ${token}` } });
+        } catch (vipErr) {
+          console.error(`[APPLY VIP ${vip.id}] failed:`, vipErr);
+          message.warning(`VIP "${vip.name}" apply failed: ${extractApiError(vipErr, 'error')}`);
+        }
+      }
+
+      // Apply HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (cluster ACME enable/disable, restore, bulk-import). Gating only on haproxyPending used to
+      // skip the call for config-version-only states, leaving those versions stuck PENDING.
+      const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
+        + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
+        ? await axios.post(
+            `/api/clusters/${selectedCluster.id}/apply-changes`,
+            {},
+            { headers: { Authorization: `Bearer ${token}` } }
+          )
+        : { data: { message: `Applied ${pendingVips.length} HA/VIP change(s)`, applied_count: pendingVips.length } };
+
       // CRITICAL DEBUG: Log apply response
       console.log('[APPLY ALL RESPONSE]:', response.data);
-      
+
+      // HA/VIP-only apply (Issue #27): nothing goes through the HAProxy config-version /
+      // cluster-sync pipeline, so the agent-sync tracker below would loop forever on
+      // "Entities: 0/0". The isolated VIP apply already staged each node's snapshot; member
+      // agents install/configure keepalived and converge on their next poll — live status
+      // shows on the HA / VIP page (PENDING → SYNCING → ACTIVE). Complete now (no flash).
+      if (haproxyPending === 0 && nonVipPendingVersions.length === 0) {
+        await fetchPendingChanges();
+        await fetchConfigVersions();
+        if (pendingVips.length > 0) {
+          // Keep the popup in a "Syncing HA/VIP... X/Y" state (like every other entity) until the
+          // member nodes report the VIP ACTIVE — instead of flashing green while the HA/VIP page
+          // still shows SYNCING. The poll runs in the background (finally{} releases applyLoading).
+          const step = `Applied — syncing HA/VIP... 0/${pendingVips.length} node(s) converged`;
+          updateEntityCounts(0, pendingVips.length, 0, 0, 0);
+          setSyncProgress({ visible: true, step, progress: 60 });
+          updateProgress(step, 60);
+          trackVipConvergence(selectedCluster.id, pendingVips.map(v => v.id), Date.now());
+        } else {
+          const vmsg = 'Changes applied.';
+          setSyncProgress({ visible: true, step: vmsg, progress: 100 });
+          completeProgress(vmsg);
+          setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.success(vmsg); }, 1500);
+        }
+        return; // finally{} resets applyLoading; the VIP poll runs in the background (like checkAgentSync)
+      }
+
       setSyncProgress({ visible: true, step: `Configuration applied, syncing agents... Entities: ${totalEntities}/${totalEntities}, Agents: 0/${totalAgents}`, progress: 60 });
       updateProgress(`Configuration applied, syncing agents... Entities: ${totalEntities}/${totalEntities}, Agents: 0/${totalAgents}`, 60);
       updateEntityCounts(totalEntities, totalEntities, 0, totalAgents, disabledAgents);
@@ -664,7 +823,7 @@ const ApplyManagement = () => {
       title: 'Reject All Configuration Changes',
       content: (
         <div>
-          <p>You are about to reject <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to reject <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -678,6 +837,15 @@ const ApplyManagement = () => {
             {pendingChanges.ssl_certificates.length > 0 && (
               <li><strong>{pendingChanges.ssl_certificates.length}</strong> SSL certificate changes</li>
             )}
+            {(pendingChanges.vips || []).length > 0 && (
+              <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
+            )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All pending changes will be permanently discarded"
@@ -718,12 +886,34 @@ const ApplyManagement = () => {
     
     try {
       const token = localStorage.getItem('token');
-      
+
-      const response = await axios.delete(
+      // HA/VIP (Issue #27): reject pending VIPs (isolated endpoint; restores last applied state).
-        `/api/clusters/${selectedCluster.id}/pending-changes`,
+      const pendingVips = pendingChanges.vips || [];
-        { headers: { Authorization: `Bearer ${token}` } }
+      for (const vip of pendingVips) {
+        try {
+          await axios.post(`/api/vip/${vip.id}/reject`, {}, { headers: { Authorization: `Bearer ${token}` } });
+        } catch (vipErr) {
+          console.error(`[REJECT VIP ${vip.id}] failed:`, vipErr);
+          message.warning(`VIP "${vip.name}" reject failed: ${extractApiError(vipErr, 'error')}`);
+        }
+      }
+
+      // Reject HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (e.g. cluster ACME enable/disable, restore, bulk-import) — the backend DELETE rejects all
+      // non-vip PENDING versions and rolls back their snapshots. Gating only on haproxyPending used
+      // to skip the call for config-version-only states, returning the misleading "Rejected 0 HA/VIP".
+      const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
+        + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
+      const nonVipPendingVersions = configVersions.filter(
+        v => v.status === 'PENDING' && !(v.version_name || '').startsWith('vip-')
       );
-      
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
+        ? await axios.delete(
+            `/api/clusters/${selectedCluster.id}/pending-changes`,
+            { headers: { Authorization: `Bearer ${token}` } }
+          )
+        : { data: { message: `Rejected ${pendingVips.length} HA/VIP change(s)` } };
+
       // CRITICAL DEBUG: Log reject response
       console.log('[REJECT ALL RESPONSE]:', response.data);
       
@@ -797,6 +987,23 @@ const ApplyManagement = () => {
   const appliedVersions = configVersions.filter(v => v.status === 'APPLIED');
   const rejectedVersions = configVersions.filter(v => v.status === 'REJECTED');
   const effectiveTotal = pendingChanges.total_count > 0 ? pendingChanges.total_count : pendingVersions.length;
+  // Issue #35: cluster ACME enable/disable produce `cluster-<id>-acme-<enable|disable>-<ts>` config
+  // versions that have NO entity-level pending flag, so they were neither categorized nor counted.
+  const ACME_VERSION_RE = /^cluster-\d+-acme-(enable|disable)-/;
+  const ENTITY_VERSION_PREFIXES = ['frontend-', 'backend-', 'server-', 'ssl-', 'waf-'];
+  const acmeVersions = pendingVersions.filter(v => ACME_VERSION_RE.test(v.version_name || ''));
+  // "Other" config versions for the confirm modal = non-vip, non-acme, non-entity-backed (i.e.
+  // restore-*/bulk-import-*/other cluster-level) — entity-backed versions are already counted via
+  // total_count, and vips are listed separately, so excluding them avoids double-counting.
+  const otherConfigVersions = pendingVersions.filter(v => {
+    const n = v.version_name || '';
+    if (n.startsWith('vip-') || ACME_VERSION_RE.test(n)) return false;
+    return !ENTITY_VERSION_PREFIXES.some(p => n.startsWith(p));
+  });
+  // Confirm-modal header count: entities+VIPs (total_count) + ACME + other config versions, so the
+  // header equals the sum of the listed <li> items in every state. The button-enable gate keeps
+  // using effectiveTotal (unchanged) so entity-only button/Alert behavior is byte-identical.
+  const modalChangeCount = (pendingChanges.total_count || 0) + acmeVersions.length + otherConfigVersions.length;
 
   const renderPendingItem = (item, type, icon) => {
     // For PENDING items, don't show sync status since they haven't been applied yet
@@ -1133,12 +1340,67 @@ const ApplyManagement = () => {
                     </div>
                   )}
 
-                  {pendingChanges.frontends.length === 0 && pendingChanges.backends.length === 0 && pendingChanges.waf_rules.length === 0 && pendingChanges.ssl_certificates.length === 0 && pendingVersions.length > 0 && (
+                  {/* HA/VIP Changes (Issue #27) */}
+                  {(pendingChanges.vips || []).length > 0 && (
+                    <div style={{ marginBottom: 16 }}>
+                      <Title level={5}>
+                        <CloudServerOutlined style={{ marginRight: 8, color: '#13c2c2' }} />
+                        HA / VIP Changes ({pendingChanges.vips.length})
+                      </Title>
+                      {pendingChanges.vips.map(item => (
+                        <div key={`vip-${item.id}`} style={{ display: 'flex', alignItems: 'center', gap: 8, padding: '8px 12px', marginBottom: 6, border: item.pending_delete ? '1px solid #ffccc7' : '1px solid #f0f0f0', borderRadius: 6, background: item.pending_delete ? '#fff1f0' : undefined }}>
+                          <CloudServerOutlined style={{ color: item.pending_delete ? '#cf1322' : '#13c2c2' }} />
+                          <span style={{ fontWeight: 500 }}>{item.name}</span>
+                          <Tag>{item.virtual_ip}/{item.prefix_length}</Tag>
+                          {item.pool_name && <Tag color="blue">{item.pool_name}</Tag>}
+                          {item.pending_delete
+                            ? <Tag color="red">DELETION — approve to tear down, reject to keep</Tag>
+                            : <Tag color="orange">PENDING</Tag>}
+                        </div>
+                      ))}
+                    </div>
+                  )}
+
+                  {/* Issue #35: ACME Challenge Routing (cluster-<id>-acme-*) versions have no entity
+                      flag. Render them in their own section REGARDLESS of whether entity sections are
+                      present, so a co-pending ACME toggle is never hidden in the left panel. */}
+                  {acmeVersions.length > 0 && (
+                    <div style={{ marginTop: 8, marginBottom: 8 }}>
+                      <Title level={5}>
+                        <SafetyCertificateOutlined style={{ marginRight: 8, color: '#1890ff' }} />
+                        ACME Challenge Routing ({acmeVersions.length})
+                      </Title>
+                      {acmeVersions.map(v => {
+                        const isEnable = /^cluster-\d+-acme-enable-/.test(v.version_name);
+                        return (
+                          <div key={v.id} style={{
+                            padding: 10, border: '1px dashed #1890ff', borderRadius: 6, marginBottom: 8,
+                            display: 'flex', alignItems: 'center', justifyContent: 'space-between', backgroundColor: '#f0f8ff'
+                          }}>
+                            <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
+                            <span>
+                              <Tag color={isEnable ? 'green' : 'default'}>{isEnable ? 'ENABLE' : 'DISABLE'}</Tag>
+                              <Tag color="orange">PENDING</Tag>
+                            </span>
+                          </div>
+                        );
+                      })}
+                      <div style={{ fontSize: 12, color: token.colorTextSecondary, marginTop: 4 }}>
+                        ACME challenge routing change. Apply to push the updated HAProxy config to the agents.
+                      </div>
+                    </div>
+                  )}
+
+                  {pendingChanges.frontends.length === 0 && pendingChanges.backends.length === 0 && pendingChanges.waf_rules.length === 0 && pendingChanges.ssl_certificates.length === 0 && (pendingChanges.vips || []).length === 0 && pendingVersions.length > 0 && (
                     <div style={{ marginTop: 8 }}>
                       {(() => {
                         const restoreVersions = pendingVersions.filter(v => v.version_name.startsWith('restore-'));
                         const bulkImportVersions = pendingVersions.filter(v => v.version_name.startsWith('bulk-import-'));
-                        const otherVersions = pendingVersions.filter(v => !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-'));
+                        // Exclude restore-/bulk-import- (own sections), vip-* (VIP section), and acme-*
+                        // (the dedicated ACME section above) so they aren't duplicated in "Other".
+                        const otherVersions = pendingVersions.filter(v =>
+                          !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-')
+                          && !v.version_name.startsWith('vip-') && !ACME_VERSION_RE.test(v.version_name));
                         
                         return (
                           <>
@@ -1401,6 +1663,10 @@ const ApplyManagement = () => {
                                     confusing error post-click. */}
                                 {(() => {
                                   const vn = version?.version_name || '';
+                                  // vip-* versions ARE undoable (Issue #27): the VIP router
+                                  // re-stages the rejected change as PENDING (reactivating a
+                                  // rejected create, or re-applying a rejected edit), so they
+                                  // use the normal Undo button below — not this disabled case.
                                   const destructive = (
                                     vn.startsWith('bulk-site-create-')
                                     || vn.startsWith('bulk-import-')

frontend/src/components/BackendServers.js

@@ -172,11 +172,13 @@ const BackendServers = () => {
     
     setLoading(true);
     try {
-      const params = { cluster_id: selectedCluster.id };
+      // Issue #24: request inactive entities so DISABLED (toggled-OFF) servers
-      
+      // are returned and can be reactivated from the UI (mirrors ApplyManagement).
+      const params = { cluster_id: selectedCluster.id, include_inactive: true };
+
       // CRITICAL FIX: Add cache busting to prevent stale data from appearing
       // Browser/axios may cache GET requests, causing deleted backends to reappear
-      const response = await axios.get('/api/backends', { 
+      const response = await axios.get('/api/backends', {
         params,
         headers: {
           'Cache-Control': 'no-cache, no-store, must-revalidate',
@@ -184,7 +186,21 @@ const BackendServers = () => {
           'Expires': '0'
         }
       });
-      const fetchedBackends = response.data.backends || [];
+      // Issue #24: we now request include_inactive=true so DISABLED servers come
+      // back. That ALSO returns soft-deleted (is_active=false) BACKENDS, which the
+      // pre-change default (include_inactive=false -> WHERE is_active=TRUE) hid.
+      // Restore that filter here so soft-deleted backends don't reappear in the
+      // normal view (preserve commit f34a6ee), while still keeping inactive
+      // SERVERS inside active backends. Then hide ONLY soft-deleted-pending
+      // servers (last_config_status==='DELETION'); DISABLED servers (toggled
+      // OFF — status PENDING/APPLIED) stay visible so operators can re-enable
+      // them. Single chokepoint covers counts, expanded rows, and All Servers tab.
+      const fetchedBackends = (response.data.backends || [])
+        .filter(b => b.is_active !== false)
+        .map(b => ({
+          ...b,
+          servers: (b.servers || []).filter(s => s.last_config_status !== 'DELETION'),
+        }));
       setBackends(fetchedBackends);
       // CRITICAL FIX: Apply status filters after fetching to maintain filter state
       // This prevents backends from disappearing when updated (e.g., APPLIED → PENDING)
@@ -1289,6 +1305,7 @@ const BackendServers = () => {
           />
           <strong>{text}</strong>
           {record.backup_server && <Tag color="orange">Backup</Tag>}
+          {!record.is_active && <Tag color="red">Inactive</Tag>}
         </Space>
       ),
     },

frontend/src/components/ClusterManagement.js

@@ -156,6 +156,7 @@ const ClusterManagement = () => {
       stats_socket_path: cluster.stats_socket_path || '/run/haproxy/admin.sock',
       haproxy_config_path: cluster.haproxy_config_path || '/etc/haproxy/haproxy.cfg',
       haproxy_bin_path: cluster.haproxy_bin_path,
+      keepalived_config_path: cluster.keepalived_config_path || '/etc/keepalived/keepalived.conf',
       agent_pool_id: cluster.pool_id || undefined,
       haproxy_user: cluster.haproxy_user || '',
       haproxy_group: cluster.haproxy_group || '',
@@ -197,6 +198,7 @@ const ClusterManagement = () => {
         stats_socket_path: values.stats_socket_path,
         haproxy_config_path: values.haproxy_config_path,
         haproxy_bin_path: values.haproxy_bin_path,
+        keepalived_config_path: values.keepalived_config_path,
         pool_id: values.agent_pool_id || null,
         haproxy_user: values.haproxy_user || null,
         haproxy_group: values.haproxy_group || null,
@@ -748,6 +750,14 @@ const ClusterManagement = () => {
             <Input placeholder="/usr/sbin/haproxy" />
           </Form.Item>
 
+          <Form.Item
+            label="Keepalived Config Path"
+            name="keepalived_config_path"
+            tooltip="Where the agent writes keepalived.conf for this cluster's HA/VIPs. The default (/etc/keepalived/keepalived.conf) is what the keepalived service loads on every distro — leave it unless you run a non-standard install AND have configured the keepalived service/unit to load this exact path."
+          >
+            <Input placeholder="/etc/keepalived/keepalived.conf" />
+          </Form.Item>
+
           <Form.Item
             label="ACME Challenge Routing"
             name="acme_enabled"

frontend/src/components/FrontendManagement.js

@@ -1015,12 +1015,56 @@ const FrontendManagement = () => {
       return;
     }
     if (grandfatheredContradictions.length > 0) {
+      // Bulgu #83 (round-23 audit) — surface the actual offending rule
+      // string(s) instead of just a count. Pre-fix the warning said
+      // "1 legacy rule has X !X" and the operator had to hunt
+      // through the ACL Builder cards to figure out which rule the
+      // gate was complaining about. The unchanged-rule path is the
+      // common case (operator changes port / maxconn on a frontend
+      // that already had a self-contradictory routing rule from a
+      // prior session), so making the rule discoverable from the
+      // toast keeps "Edit and Save" → "fix the dead rule" workflows
+      // single-screen. Also stop calling these rules "legacy" —
+      // the operator may have written them seconds earlier; the
+      // only thing this branch knows is that they weren't modified
+      // by the current edit.
+      const renderGrandfatheredRule = (r) => {
+        if (typeof r === 'string') return r;
+        if (r && typeof r === 'object') {
+          try { return JSON.stringify(r); } catch (_e) { return '[rule]'; }
+        }
+        return '[rule]';
+      };
+      const ruleSnippets = grandfatheredContradictions
+        .slice(0, 5)
+        .map(renderGrandfatheredRule)
+        .map((s) => (s.length > 160 ? `${s.slice(0, 157)}...` : s));
+      const extra = grandfatheredContradictions.length > ruleSnippets.length
+        ? ` (+${grandfatheredContradictions.length - ruleSnippets.length} more)`
+        : '';
       message.warning(
-        `This frontend has ${grandfatheredContradictions.length} legacy ` +
+        <div>
-        `routing/redirect rule(s) with a self-contradictory \`X !X\` ` +
+          <div>
-        `condition. The rule(s) never fire — fix them at your convenience. ` +
+            <strong>
-        `Your current edit will still be saved.`,
+              {grandfatheredContradictions.length} routing/redirect rule(s)
-        6,
+              you didn't modify in this edit contain a self-contradictory
+              `X !X` condition (e.g. `if acl1 !acl1`).
+            </strong>
+          </div>
+          <div style={{ marginTop: 4, fontSize: '12px' }}>
+            HAProxy accepts the syntax but `X AND NOT X` is always false,
+            so the rule never fires and traffic silently falls through to
+            `default_backend`. Your current edit will still be saved; fix
+            the rule(s) at your convenience.
+          </div>
+          <div style={{ marginTop: 6, fontSize: '12px', fontFamily: 'monospace' }}>
+            {ruleSnippets.map((s, i) => (
+              <div key={i}>• {s}</div>
+            ))}
+            {extra && <div>{extra}</div>}
+          </div>
+        </div>,
+        10,
       );
     }
 
@@ -1081,6 +1125,36 @@ const FrontendManagement = () => {
         } else {
           message.success('Frontend updated successfully');
         }
+
+        // Bulgu #83 (round-23 audit) — surface server-emitted
+        // grandfathered-rule warnings (e.g. `X !X` contradictions
+        // in routing/redirect rules that the operator did not
+        // touch this edit). The FE client-side gate ALSO catches
+        // these and fires its own toast above the modal close;
+        // we re-surface the server view here as a safety net in
+        // case the client gate missed an edge shape (different
+        // dict serialization, etc.). Server warnings already
+        // include the verbatim rule text, so the operator sees
+        // exactly which entry to fix.
+        const serverWarnings = Array.isArray(response.data?.warnings)
+          ? response.data.warnings
+          : [];
+        if (serverWarnings.length > 0 && grandfatheredContradictions.length === 0) {
+          message.warning(
+            <div>
+              <div><strong>Frontend saved, but the server flagged {serverWarnings.length} rule warning(s):</strong></div>
+              <div style={{ marginTop: 6, fontSize: '12px', fontFamily: 'monospace' }}>
+                {serverWarnings.slice(0, 5).map((w, i) => (
+                  <div key={i}>• {w.length > 240 ? `${w.slice(0, 237)}...` : w}</div>
+                ))}
+                {serverWarnings.length > 5 && (
+                  <div>(+{serverWarnings.length - 5} more)</div>
+                )}
+              </div>
+            </div>,
+            10,
+          );
+        }
       } else {
         response = await axios.post('/api/frontends', requestData);
         

frontend/src/components/Login.js

@@ -1,20 +1,20 @@
-import React, { useState } from 'react';
+import React, { useEffect, useRef, useState } from 'react';
-import { 
+import {
-  Card, 
+  Card,
-  Form, 
+  Form,
-  Input, 
+  Input,
-  Button, 
+  Button,
-  message, 
+  message,
-  Typography, 
+  Typography,
-  Row, 
+  Row,
   Col,
   Alert,
-  Spin
 } from 'antd';
-import { 
+import {
-  UserOutlined, 
+  UserOutlined,
-  LockOutlined, 
+  LockOutlined,
-  ClusterOutlined 
+  ClusterOutlined,
+  SafetyCertificateOutlined,
 } from '@ant-design/icons';
 import axios from 'axios';
 import { useAuth } from '../contexts/AuthContext';
@@ -23,77 +23,260 @@ import './Login.css';
 
 const { Title, Text } = Typography;
 
+const PHASE_CREDENTIALS = 'credentials';
+const PHASE_MFA = 'mfa';
+const PHASE_SUBMITTING = 'submitting';
+
 const Login = () => {
-  const [form] = Form.useForm();
+  const [credentialsForm] = Form.useForm();
+  const [mfaForm] = Form.useForm();
+  const [phase, setPhase] = useState(PHASE_CREDENTIALS);
   const [loading, setLoading] = useState(false);
   const [error, setError] = useState('');
   const { login } = useAuth();
 
-  const handleSubmit = async (values) => {
+  // MFA-specific state — RAM only, never persisted.
+  const mfaTokenRef = useRef(null);
+  const [mfaExpiresAt, setMfaExpiresAt] = useState(null);
+  const [mfaCountdown, setMfaCountdown] = useState(0);
+
+  useEffect(() => {
+    if (phase !== PHASE_MFA || !mfaExpiresAt) return undefined;
+    const id = setInterval(() => {
+      const remaining = Math.max(0, Math.floor((mfaExpiresAt - Date.now()) / 1000));
+      setMfaCountdown(remaining);
+      if (remaining <= 0) {
+        clearInterval(id);
+        resetToCredentials('MFA session expired. Please log in again.');
+      }
+    }, 1000);
+    return () => clearInterval(id);
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, [phase, mfaExpiresAt]);
+
+  const resetToCredentials = (errMessage) => {
+    mfaTokenRef.current = null;
+    setMfaExpiresAt(null);
+    setMfaCountdown(0);
+    mfaForm.resetFields();
+    setPhase(PHASE_CREDENTIALS);
+    if (errMessage) setError(errMessage);
+  };
+
+  const completeAuth = (authData) => {
+    // Write storage + axios header ONLY after a full, MFA-cleared response.
+    localStorage.setItem('token', authData.access_token);
+    localStorage.setItem('authToken', authData.access_token);
+    localStorage.setItem('userData', JSON.stringify(authData.user));
+    localStorage.setItem('userRoles', JSON.stringify([]));
+    localStorage.setItem('userPermissions', JSON.stringify({}));
+    const expiryDate = new Date();
+    expiryDate.setSeconds(expiryDate.getSeconds() + authData.expires_in);
+    localStorage.setItem('tokenExpiry', expiryDate.toISOString());
+    const loginSuccess = login(authData);
+    if (loginSuccess) {
+      message.success(`Welcome back, ${authData.user.username}!`);
+    } else {
+      throw new Error('Failed to update authentication state');
+    }
+  };
+
+  const handleCredentialsSubmit = async (values) => {
     setLoading(true);
+    setPhase(PHASE_SUBMITTING);
     setError('');
-    
     try {
       const response = await axios.post('/api/auth/login', {
         username: values.username,
-        password: values.password
+        password: values.password,
       });
 
-      // Store authentication data - API returns access_token, not session_token!
+      if (response.data && response.data.mfa_required) {
-      localStorage.setItem('token', response.data.access_token);
+        // Phase 2 — TOTP / backup code challenge. Keep credentials secret-free.
-      localStorage.setItem('authToken', response.data.access_token);
+        mfaTokenRef.current = response.data.mfa_token;
-      localStorage.setItem('userData', JSON.stringify(response.data.user));
+        const ttlSeconds = response.data.expires_in || 300;
-      localStorage.setItem('userRoles', JSON.stringify([])); // API doesn't return roles directly
+        setMfaExpiresAt(Date.now() + ttlSeconds * 1000);
-      localStorage.setItem('userPermissions', JSON.stringify({})); // API doesn't return permissions directly
+        setMfaCountdown(ttlSeconds);
-      
+        setPhase(PHASE_MFA);
-      // Calculate expiry from expires_in (seconds)
+        return;
-      const expiryDate = new Date();
-      expiryDate.setSeconds(expiryDate.getSeconds() + response.data.expires_in);
-      localStorage.setItem('tokenExpiry', expiryDate.toISOString());
-
-      // Update authentication context
-      const loginSuccess = login(response.data);
-      
-      if (loginSuccess) {
-        message.success(`Welcome back, ${response.data.user.username}!`);
-        // The authentication context will automatically trigger a re-render
-        // and the user will be redirected to the main dashboard
-      } else {
-        throw new Error('Failed to update authentication state');
       }
-      
+
-    } catch (error) {
+      completeAuth(response.data);
-      const errorMessage = extractApiError(error, 'Login failed. Please try again.');
+    } catch (err) {
+      const errorMessage = extractApiError(err, 'Login failed. Please try again.');
       setError(errorMessage);
       message.error(errorMessage);
+      setPhase(PHASE_CREDENTIALS);
     } finally {
       setLoading(false);
     }
   };
 
+  const handleMfaSubmit = async (values) => {
+    if (!mfaTokenRef.current) {
+      resetToCredentials('MFA session lost. Please log in again.');
+      return;
+    }
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post(
+        '/api/auth/login/mfa-verify',
+        {
+          mfa_token: mfaTokenRef.current,
+          code: (values.code || '').trim(),
+        },
+        // Explicit opt-out: never attach a stale Authorization header here.
+        { headers: { Authorization: undefined } },
+      );
+      completeAuth(response.data);
+    } catch (err) {
+      const status = err && err.response && err.response.status;
+      const errorMessage = extractApiError(err, 'Verification failed.');
+      if (status === 410) {
+        resetToCredentials(errorMessage || 'MFA challenge invalidated. Please log in again.');
+      } else {
+        setError(errorMessage);
+        mfaForm.setFieldsValue({ code: '' });
+      }
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const renderCredentialsForm = () => (
+    <Form
+      form={credentialsForm}
+      name="login"
+      onFinish={handleCredentialsSubmit}
+      layout="vertical"
+      autoComplete="off"
+    >
+      <Form.Item
+        name="username"
+        rules={[
+          { required: true, message: 'Please enter your username!' },
+          { min: 3, message: 'Username must be at least 3 characters!' },
+        ]}
+      >
+        <Input prefix={<UserOutlined />} placeholder="Username" autoComplete="username" />
+      </Form.Item>
+
+      <Form.Item
+        name="password"
+        rules={[
+          { required: true, message: 'Please enter your password!' },
+          { min: 6, message: 'Password must be at least 6 characters!' },
+        ]}
+      >
+        <Input.Password
+          prefix={<LockOutlined />}
+          placeholder="Password"
+          autoComplete="current-password"
+        />
+      </Form.Item>
+
+      <Form.Item style={{ marginBottom: 0 }}>
+        <Button
+          type="primary"
+          htmlType="submit"
+          loading={loading}
+          block
+          className="login-button"
+        >
+          {loading ? 'Signing in...' : 'Sign In'}
+        </Button>
+      </Form.Item>
+    </Form>
+  );
+
+  const renderMfaForm = () => {
+    const minutes = Math.floor(mfaCountdown / 60);
+    const seconds = String(mfaCountdown % 60).padStart(2, '0');
+    return (
+      <Form form={mfaForm} name="mfa" onFinish={handleMfaSubmit} layout="vertical" autoComplete="off">
+        <Alert
+          message="Multi-Factor Authentication"
+          description={
+            <span>
+              Enter the 6-digit code from your authenticator app, or use a backup code
+              (format: <code>XXXX-YYYY</code>).
+              {mfaCountdown > 0 && (
+                <>
+                  {' '}Session expires in <strong>{minutes}:{seconds}</strong>.
+                </>
+              )}
+            </span>
+          }
+          type="info"
+          showIcon
+          style={{ marginBottom: 16 }}
+        />
+        <Form.Item
+          name="code"
+          rules={[
+            { required: true, message: 'Please enter your MFA code.' },
+            { min: 6, message: 'Code must be at least 6 characters.' },
+            { max: 10, message: 'Code is too long.' },
+          ]}
+        >
+          <Input
+            prefix={<SafetyCertificateOutlined />}
+            placeholder="123456 or XXXX-YYYY"
+            autoComplete="one-time-code"
+            inputMode="text"
+            maxLength={10}
+            autoFocus
+          />
+        </Form.Item>
+        <Form.Item style={{ marginBottom: 8 }}>
+          <Button
+            type="primary"
+            htmlType="submit"
+            loading={loading}
+            block
+            className="login-button"
+          >
+            {loading ? 'Verifying...' : 'Verify'}
+          </Button>
+        </Form.Item>
+        <Form.Item style={{ marginBottom: 0 }}>
+          <Button
+            type="default"
+            block
+            onClick={() => resetToCredentials('')}
+            disabled={loading}
+          >
+            Use a different account
+          </Button>
+        </Form.Item>
+      </Form>
+    );
+  };
+
   return (
     <div className="login-container">
-      <Row 
+      <Row
-        justify="center" 
+        justify="center"
-        align="middle" 
+        align="middle"
-        style={{ 
+        style={{
           minHeight: '100vh',
           minHeight: '100dvh',
           width: '100%',
-          margin: 0
+          margin: 0,
         }}
       >
-        <Col 
+        <Col
-          xs={24} 
+          xs={24}
-          sm={20} 
+          sm={20}
-          md={16} 
+          md={16}
-          lg={12} 
+          lg={12}
-          xl={10} 
+          xl={10}
           xxl={8}
-          style={{ 
+          style={{
             display: 'flex',
             justifyContent: 'center',
-            padding: '0 8px'
+            padding: '0 8px',
           }}
         >
           <Card className="login-card">
@@ -118,56 +301,13 @@ const Login = () => {
               />
             )}
 
-            <Form
+            {phase === PHASE_MFA ? renderMfaForm() : renderCredentialsForm()}
-              form={form}
-              name="login"
-              onFinish={handleSubmit}
-              layout="vertical"
-              autoComplete="off"
-            >
-              <Form.Item
-                name="username"
-                rules={[
-                  { required: true, message: 'Please enter your username!' },
-                  { min: 3, message: 'Username must be at least 3 characters!' }
-                ]}
-              >
-                <Input
-                  prefix={<UserOutlined />}
-                  placeholder="Username"
-                  autoComplete="username"
-                />
-              </Form.Item>
-
-              <Form.Item
-                name="password"
-                rules={[
-                  { required: true, message: 'Please enter your password!' },
-                  { min: 6, message: 'Password must be at least 6 characters!' }
-                ]}
-              >
-                <Input.Password
-                  prefix={<LockOutlined />}
-                  placeholder="Password"
-                  autoComplete="current-password"
-                />
-              </Form.Item>
-
-              <Form.Item style={{ marginBottom: 0 }}>
-                <Button
-                  type="primary"
-                  htmlType="submit"
-                  loading={loading}
-                  block
-                  className="login-button"
-                >
-                  {loading ? 'Signing in...' : 'Sign In'}
-                </Button>
-              </Form.Item>
-            </Form>
 
             <div className="login-footer">
-              <Text type="secondary" style={{ fontSize: 12, display: 'block', textAlign: 'center' }}>
+              <Text
+                type="secondary"
+                style={{ fontSize: 12, display: 'block', textAlign: 'center' }}
+              >
                 Centralized management for multiple HAProxy clusters
               </Text>
             </div>
@@ -178,4 +318,4 @@ const Login = () => {
   );
 };
 
-export default Login; 
+export default Login;

frontend/src/components/MFAEnrollModal.js

@@ -0,0 +1,306 @@
+import React, { useEffect, useRef, useState } from 'react';
+import {
+  Modal,
+  Steps,
+  Form,
+  Input,
+  Button,
+  Alert,
+  Typography,
+  Space,
+  Checkbox,
+  message,
+} from 'antd';
+import { QRCodeSVG } from 'qrcode.react';
+import axios from 'axios';
+import { extractApiError } from '../utils/apiError';
+
+const { Text, Paragraph } = Typography;
+
+const STEP_SETUP = 0;
+const STEP_VERIFY = 1;
+const STEP_BACKUP = 2;
+
+/**
+ * MFA enrollment wizard. Strictly modal-controlled: the modal cannot be
+ * dismissed via the X / mask in step 2/3 — backup codes are shown only once
+ * and the server-side pending row is opaque after enrollment confirms.
+ */
+const MFAEnrollModal = ({ open, onClose, onEnrolled }) => {
+  const [verifyForm] = Form.useForm();
+  const [step, setStep] = useState(STEP_SETUP);
+  const [loading, setLoading] = useState(false);
+  const [error, setError] = useState('');
+  const [otpauthUri, setOtpauthUri] = useState('');
+  const [secret, setSecret] = useState('');
+  const [backupCodes, setBackupCodes] = useState([]);
+  const [savedAcknowledged, setSavedAcknowledged] = useState(false);
+  const startedRef = useRef(false);
+
+  useEffect(() => {
+    if (!open) return undefined;
+    if (startedRef.current) return undefined;
+    startedRef.current = true;
+    startEnrollment();
+    return () => {
+      // No-op: cleanup happens via the explicit handleClose path.
+    };
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, [open]);
+
+  const resetState = () => {
+    setStep(STEP_SETUP);
+    setLoading(false);
+    setError('');
+    setOtpauthUri('');
+    setSecret('');
+    setBackupCodes([]);
+    setSavedAcknowledged(false);
+    startedRef.current = false;
+    verifyForm.resetFields();
+  };
+
+  const startEnrollment = async () => {
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post('/api/mfa/enroll/start', {});
+      setOtpauthUri(response.data.otpauth_uri);
+      setSecret(response.data.secret);
+    } catch (err) {
+      setError(extractApiError(err, 'Could not start MFA enrollment.'));
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const handleVerify = async (values) => {
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post('/api/mfa/enroll/confirm', {
+        code: (values.code || '').trim(),
+      });
+      setBackupCodes(response.data.backup_codes || []);
+      setStep(STEP_BACKUP);
+    } catch (err) {
+      setError(extractApiError(err, 'Verification failed.'));
+      verifyForm.setFieldsValue({ code: '' });
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const handleCopyAll = () => {
+    const text = backupCodes.join('\n');
+    if (navigator.clipboard && navigator.clipboard.writeText) {
+      navigator.clipboard.writeText(text).then(
+        () => message.success('Backup codes copied to clipboard'),
+        () => message.error('Could not copy. Please copy manually.'),
+      );
+    } else {
+      message.warning('Clipboard API unavailable. Please copy manually.');
+    }
+  };
+
+  const handleDownload = () => {
+    const blob = new Blob(
+      [
+        'HAProxy OpenManager — MFA backup codes\n',
+        'Generated: ' + new Date().toISOString() + '\n',
+        'Each code is single-use. Store them somewhere safe and offline.\n\n',
+        ...backupCodes.map((c) => c + '\n'),
+      ],
+      { type: 'text/plain;charset=utf-8' },
+    );
+    const url = URL.createObjectURL(blob);
+    const link = document.createElement('a');
+    link.href = url;
+    link.download = 'haproxy-openmanager-mfa-backup-codes.txt';
+    document.body.appendChild(link);
+    link.click();
+    document.body.removeChild(link);
+    URL.revokeObjectURL(url);
+  };
+
+  const handleClose = (force = false) => {
+    if (step === STEP_BACKUP && !savedAcknowledged && !force) return;
+    resetState();
+    if (step === STEP_BACKUP) {
+      if (typeof onEnrolled === 'function') onEnrolled();
+    } else if (typeof onClose === 'function') {
+      onClose();
+    }
+  };
+
+  const renderSetup = () => (
+    <Space direction="vertical" size="middle" style={{ width: '100%' }}>
+      <Paragraph>
+        Open your authenticator app (Google Authenticator, Authy, 1Password, Microsoft
+        Authenticator) and scan this QR code, or enter the secret manually.
+      </Paragraph>
+      <div style={{ display: 'flex', justifyContent: 'center' }}>
+        {otpauthUri ? (
+          <QRCodeSVG value={otpauthUri} size={220} level="M" includeMargin />
+        ) : (
+          <Text type="secondary">Generating…</Text>
+        )}
+      </div>
+      {secret && (
+        <Alert
+          message="Trouble scanning?"
+          description={
+            <Space direction="vertical" size={4}>
+              <Text>Enter this secret manually in your authenticator app:</Text>
+              <Text code copyable={{ text: secret }} style={{ fontSize: 16 }}>
+                {secret}
+              </Text>
+            </Space>
+          }
+          type="info"
+          showIcon
+        />
+      )}
+      <div style={{ textAlign: 'right' }}>
+        <Space>
+          <Button onClick={() => handleClose(true)}>Cancel</Button>
+          <Button
+            type="primary"
+            onClick={() => setStep(STEP_VERIFY)}
+            disabled={!otpauthUri}
+          >
+            I&apos;ve added the account
+          </Button>
+        </Space>
+      </div>
+    </Space>
+  );
+
+  const renderVerify = () => (
+    <Form form={verifyForm} layout="vertical" onFinish={handleVerify}>
+      <Alert
+        message="Verify your authenticator"
+        description="Enter the 6-digit code displayed by your authenticator app. You have 5 attempts before the enrollment is invalidated and you'll need to start over."
+        type="info"
+        showIcon
+        style={{ marginBottom: 16 }}
+      />
+      <Form.Item
+        name="code"
+        label="Authenticator code"
+        rules={[
+          { required: true, message: 'Please enter the 6-digit code.' },
+          { len: 6, message: 'Code must be exactly 6 digits.' },
+        ]}
+      >
+        <Input
+          placeholder="123456"
+          autoComplete="one-time-code"
+          inputMode="numeric"
+          maxLength={6}
+          autoFocus
+        />
+      </Form.Item>
+      <div style={{ textAlign: 'right' }}>
+        <Space>
+          <Button onClick={() => setStep(STEP_SETUP)} disabled={loading}>
+            Back
+          </Button>
+          <Button onClick={() => handleClose(true)} disabled={loading}>
+            Cancel
+          </Button>
+          <Button type="primary" htmlType="submit" loading={loading}>
+            Verify
+          </Button>
+        </Space>
+      </div>
+    </Form>
+  );
+
+  const renderBackup = () => (
+    <Space direction="vertical" size="middle" style={{ width: '100%' }}>
+      <Alert
+        message="Save your backup codes now"
+        description={
+          <>
+            Each code can be used <strong>once</strong> when you can&apos;t access your
+            authenticator. <strong>They won&apos;t be shown again.</strong> If you lose
+            them, ask an administrator to reset your MFA.
+          </>
+        }
+        type="warning"
+        showIcon
+      />
+      <div
+        style={{
+          display: 'grid',
+          gridTemplateColumns: '1fr 1fr',
+          gap: '8px 16px',
+          padding: '12px',
+          backgroundColor: 'var(--ant-color-fill-quaternary, #fafafa)',
+          borderRadius: 6,
+        }}
+      >
+        {backupCodes.map((code) => (
+          <Text key={code} code style={{ fontSize: 15, letterSpacing: 1 }}>
+            {code}
+          </Text>
+        ))}
+      </div>
+      <Space>
+        <Button onClick={handleCopyAll}>Copy all</Button>
+        <Button onClick={handleDownload}>Download .txt</Button>
+      </Space>
+      <Checkbox
+        checked={savedAcknowledged}
+        onChange={(e) => setSavedAcknowledged(e.target.checked)}
+      >
+        I have saved my backup codes somewhere safe.
+      </Checkbox>
+      <div style={{ textAlign: 'right' }}>
+        <Button
+          type="primary"
+          disabled={!savedAcknowledged}
+          onClick={() => handleClose(false)}
+        >
+          Close
+        </Button>
+      </div>
+    </Space>
+  );
+
+  return (
+    <Modal
+      open={open}
+      title="Enable Multi-Factor Authentication"
+      width={520}
+      footer={null}
+      closable={false}
+      maskClosable={false}
+      destroyOnClose
+      keyboard={false}
+    >
+      <Steps
+        size="small"
+        current={step}
+        items={[{ title: 'Set up' }, { title: 'Verify' }, { title: 'Backup codes' }]}
+        style={{ marginBottom: 24 }}
+      />
+      {error && (
+        <Alert
+          message={error}
+          type="error"
+          showIcon
+          closable
+          onClose={() => setError('')}
+          style={{ marginBottom: 16 }}
+        />
+      )}
+      {step === STEP_SETUP && renderSetup()}
+      {step === STEP_VERIFY && renderVerify()}
+      {step === STEP_BACKUP && renderBackup()}
+    </Modal>
+  );
+};
+
+export default MFAEnrollModal;

frontend/src/components/Settings.js

@@ -212,6 +212,7 @@ const Settings = () => {
                 eab_kid: '',
                 eab_hmac_key: '',
                 challenge_backend_url: '',
+                dns01_enabled: false,
               }}
             >
               <Form.Item name="provider" label="ACME Provider">
@@ -308,6 +309,36 @@ const Settings = () => {
                 }]}
               />
 
+              {/* Issue #35: DNS-01 challenge support (global kill-switch). Per-account DNS provider
+                  credentials are configured on each ACME account in ACME Automation. */}
+              <Collapse
+                ghost
+                style={{ marginTop: 16 }}
+                items={[{
+                  key: 'dns01',
+                  label: 'DNS-01 Challenge (Advanced)',
+                  children: (
+                    <>
+                      <Alert
+                        type="info"
+                        showIcon
+                        style={{ marginBottom: 16 }}
+                        message="DNS-01 validates certificates via a DNS TXT record instead of HTTP on port 80."
+                        description="Use it for internal/isolated clusters with no public inbound port 80, or for wildcard certificates. When enabled, choose DNS-01 and a DNS provider per ACME account in ACME Automation. Leaving this off keeps the default HTTP-01 behavior unchanged."
+                      />
+                      <Form.Item
+                        name="dns01_enabled"
+                        label="Enable DNS-01 Challenge"
+                        valuePropName="checked"
+                        tooltip="Master switch. While off, DNS-01 options are hidden and no DNS-01 orders can be created."
+                      >
+                        <Switch />
+                      </Form.Item>
+                    </>
+                  ),
+                }]}
+              />
+
               <div style={{ marginTop: 24, display: 'flex', gap: 12 }}>
                 <Button type="primary" htmlType="submit" loading={acmeSaving}>
                   Save ACME Settings

frontend/src/components/UserManagement.js

@@ -32,11 +32,15 @@ import {
   HistoryOutlined,
   UserAddOutlined,
   KeyOutlined,
-  DownloadOutlined
+  DownloadOutlined,
+  SafetyCertificateOutlined,
+  ReloadOutlined,
+  InfoCircleOutlined,
 } from '@ant-design/icons';
 import axios from 'axios';
 import { useAuth } from '../contexts/AuthContext';
 import { extractApiError } from '../utils/apiError';
+import MFAEnrollModal from './MFAEnrollModal';
 
 const { TabPane } = Tabs;
 const { Option } = Select;
@@ -142,6 +146,17 @@ const PERMISSION_TREE = [
       { title: 'View Cluster Config', key: 'clusters.config' }
     ]
   },
+  {
+    title: '🛰️ HA / VIP Management',
+    key: 'vip',
+    children: [
+      { title: 'View VIPs', key: 'vip.read' },
+      { title: 'Create VIP', key: 'vip.create' },
+      { title: 'Edit VIP', key: 'vip.update' },
+      { title: 'Delete VIP', key: 'vip.delete' },
+      { title: 'Apply / Deploy VIP', key: 'vip.apply' }
+    ]
+  },
   {
     title: '📝 Configuration',
     key: 'config',
@@ -225,8 +240,16 @@ const PERMISSION_TREE = [
 ];
 
 const UserManagement = () => {
-  const { isAdmin } = useAuth(); // Get admin status from auth context
+  const { isAdmin, user: currentUser } = useAuth(); // Get admin status from auth context
   const { token } = theme.useToken();
+
+  // MFA UI state (Issue #18, v1.6.0)
+  const [mfaEnrollOpen, setMfaEnrollOpen] = useState(false);
+  const [mfaDisableOpen, setMfaDisableOpen] = useState(false);
+  const [mfaDisableForm] = Form.useForm();
+  const [mfaActionLoading, setMfaActionLoading] = useState(false);
+  const [mfaResetTarget, setMfaResetTarget] = useState(null);
+  const [mfaResetForm] = Form.useForm();
   const [activeTab, setActiveTab] = useState('users');
   
   // Users state
@@ -525,6 +548,47 @@ const UserManagement = () => {
     }
   };
 
+  // ---- MFA handlers (Issue #18, v1.6.0) -----------------------------------
+
+  const handleMfaEnrolled = () => {
+    setMfaEnrollOpen(false);
+    message.success('MFA enabled for your account.');
+    fetchUsers();
+  };
+
+  const handleMfaDisableSubmit = async (values) => {
+    setMfaActionLoading(true);
+    try {
+      await axios.post('/api/mfa/disable', { code: (values.code || '').trim() });
+      message.success('MFA disabled for your account.');
+      setMfaDisableOpen(false);
+      mfaDisableForm.resetFields();
+      fetchUsers();
+    } catch (error) {
+      message.error(extractApiError(error, 'Failed to disable MFA'));
+    } finally {
+      setMfaActionLoading(false);
+    }
+  };
+
+  const handleMfaAdminResetSubmit = async (values) => {
+    if (!mfaResetTarget) return;
+    setMfaActionLoading(true);
+    try {
+      await axios.post(`/api/mfa/admin-reset/${mfaResetTarget.id}`, {
+        reason: (values.reason || '').trim(),
+      });
+      message.success(`MFA reset for user '${mfaResetTarget.username}'.`);
+      setMfaResetTarget(null);
+      mfaResetForm.resetFields();
+      fetchUsers();
+    } catch (error) {
+      message.error(extractApiError(error, 'Failed to reset MFA'));
+    } finally {
+      setMfaActionLoading(false);
+    }
+  };
+
   const handleChangePassword = (user) => {
     setSelectedUser(user);
     passwordForm.resetFields();
@@ -717,6 +781,15 @@ const UserManagement = () => {
         </Space>
       )
     },
+    {
+      title: 'MFA',
+      dataIndex: 'mfa_enabled',
+      key: 'mfa_enabled',
+      width: 90,
+      render: (enabled) => enabled
+        ? <Tag icon={<SafetyCertificateOutlined />} color="green">ON</Tag>
+        : <Tag color="default">OFF</Tag>
+    },
     {
       title: 'Last Login',
       dataIndex: 'last_login_at',
@@ -726,31 +799,94 @@ const UserManagement = () => {
     {
       title: 'Actions',
       key: 'actions',
-      render: (_, record) => (
+      render: (_, record) => {
-        <Space>
+        const isSelf = currentUser && record.id === currentUser.id;
-          {isAdmin() && (
+        return (
-            <>
+          <Space>
-              <Tooltip title="Edit User">
+            {isAdmin() && (
-                <Button 
+              <>
-                  icon={<EditOutlined />} 
+                <Tooltip title="Edit User">
-                  size="small" 
+                  <Button
-                  onClick={() => handleEditUser(record)}
+                    icon={<EditOutlined />}
+                    size="small"
+                    onClick={() => handleEditUser(record)}
+                  />
+                </Tooltip>
+                <Tooltip title="Assign Roles">
+                  <Button
+                    icon={<TeamOutlined />}
+                    size="small"
+                    onClick={() => handleAssignRoles(record)}
+                  />
+                </Tooltip>
+                <Tooltip title="Change Password">
+                  <Button
+                    icon={<KeyOutlined />}
+                    size="small"
+                    onClick={() => handleChangePassword(record)}
+                  />
+                </Tooltip>
+              </>
+            )}
+            {isSelf && !record.mfa_enabled && (
+              <Tooltip title="Enable MFA for your account">
+                <Button
+                  icon={<SafetyCertificateOutlined />}
+                  size="small"
+                  type="primary"
+                  ghost
+                  onClick={() => setMfaEnrollOpen(true)}
+                >
+                  Enable MFA
+                </Button>
+              </Tooltip>
+            )}
+            {isSelf && record.mfa_enabled && (
+              <Tooltip title="Disable MFA for your account">
+                <Button
+                  icon={<SafetyCertificateOutlined />}
+                  size="small"
+                  onClick={() => {
+                    mfaDisableForm.resetFields();
+                    setMfaDisableOpen(true);
+                  }}
+                >
+                  Disable MFA
+                </Button>
+              </Tooltip>
+            )}
+            {isAdmin() && !isSelf && record.mfa_enabled && (
+              <Tooltip title="Reset this user's MFA (admin only)">
+                <Button
+                  icon={<ReloadOutlined />}
+                  size="small"
+                  danger
+                  onClick={() => {
+                    mfaResetForm.resetFields();
+                    setMfaResetTarget(record);
+                  }}
                 />
               </Tooltip>
-              <Tooltip title="Assign Roles">
+            )}
-                <Button 
+            {isAdmin() && !isSelf && !record.mfa_enabled && (
-                  icon={<TeamOutlined />} 
+              <Tooltip
-                  size="small" 
+                title={
-                  onClick={() => handleAssignRoles(record)}
+                  <span>
-                />
+                    Only this user can enable their own MFA — the TOTP secret
-              </Tooltip>
+                    must be set up from their device. Ask them to sign in and
-              <Tooltip title="Change Password">
+                    click <b>Enable MFA</b> in their own row.
-                <Button 
+                  </span>
-                  icon={<KeyOutlined />} 
+                }
-                  size="small" 
+              >
-                  onClick={() => handleChangePassword(record)}
+                <Button
+                  icon={<InfoCircleOutlined />}
+                  size="small"
+                  type="text"
+                  style={{ color: '#8c8c8c' }}
                 />
               </Tooltip>
+            )}
+            {isAdmin() && (
               <Popconfirm
                 title="Are you sure you want to delete this user?"
                 onConfirm={() => handleDeleteUser(record)}
@@ -758,20 +894,18 @@ const UserManagement = () => {
                 cancelText="No"
               >
                 <Tooltip title="Delete User">
-                  <Button 
+                  <Button
-                    icon={<DeleteOutlined />} 
+                    icon={<DeleteOutlined />}
-                    danger 
+                    danger
                     size="small"
                   />
                 </Tooltip>
               </Popconfirm>
-            </>
+            )}
-          )}
+            {!isAdmin() && !isSelf && <Text type="secondary">View Only</Text>}
-          {!isAdmin() && (
+          </Space>
-            <Text type="secondary">View Only</Text>
+        );
-          )}
+      }
-        </Space>
-      )
     }
   ];
 
@@ -1512,6 +1646,93 @@ const UserManagement = () => {
           </Form.Item>
         </Form>
       </Modal>
+
+      {/* MFA enrollment wizard (self) */}
+      <MFAEnrollModal
+        open={mfaEnrollOpen}
+        onClose={() => setMfaEnrollOpen(false)}
+        onEnrolled={handleMfaEnrolled}
+      />
+
+      {/* MFA self-disable modal */}
+      <Modal
+        title="Disable Multi-Factor Authentication"
+        open={mfaDisableOpen}
+        onCancel={() => setMfaDisableOpen(false)}
+        footer={null}
+        destroyOnClose
+      >
+        <Form form={mfaDisableForm} layout="vertical" onFinish={handleMfaDisableSubmit}>
+          <p>
+            Enter a current code from your authenticator app, or one of your backup
+            codes (format: <code>XXXX-YYYY</code>), to disable MFA.
+          </p>
+          <Form.Item
+            name="code"
+            label="MFA code"
+            rules={[
+              { required: true, message: 'Please enter your MFA code.' },
+              { min: 6, message: 'Code is too short.' },
+              { max: 10, message: 'Code is too long.' },
+            ]}
+          >
+            <Input
+              placeholder="123456 or XXXX-YYYY"
+              autoComplete="one-time-code"
+              maxLength={10}
+              autoFocus
+            />
+          </Form.Item>
+          <Form.Item style={{ marginBottom: 0, textAlign: 'right' }}>
+            <Space>
+              <Button onClick={() => setMfaDisableOpen(false)} disabled={mfaActionLoading}>
+                Cancel
+              </Button>
+              <Button type="primary" danger htmlType="submit" loading={mfaActionLoading}>
+                Disable MFA
+              </Button>
+            </Space>
+          </Form.Item>
+        </Form>
+      </Modal>
+
+      {/* Admin: reset another user's MFA */}
+      <Modal
+        title={mfaResetTarget ? `Reset MFA for '${mfaResetTarget.username}'` : 'Reset MFA'}
+        open={!!mfaResetTarget}
+        onCancel={() => setMfaResetTarget(null)}
+        footer={null}
+        destroyOnClose
+      >
+        <Form form={mfaResetForm} layout="vertical" onFinish={handleMfaAdminResetSubmit}>
+          <p>
+            This will disable MFA and invalidate all backup codes for{' '}
+            <strong>{mfaResetTarget && mfaResetTarget.username}</strong>. The user must
+            re-enroll on their next login. The action is recorded in the audit log.
+          </p>
+          <Form.Item
+            name="reason"
+            label="Reason (required, will be audit-logged)"
+            rules={[
+              { required: true, message: 'Please provide a reason.' },
+              { min: 3, message: 'Reason is too short.' },
+              { max: 500, message: 'Reason is too long (max 500 characters).' },
+            ]}
+          >
+            <Input.TextArea rows={3} placeholder="e.g. Lost phone — verified identity via ticket #12345." />
+          </Form.Item>
+          <Form.Item style={{ marginBottom: 0, textAlign: 'right' }}>
+            <Space>
+              <Button onClick={() => setMfaResetTarget(null)} disabled={mfaActionLoading}>
+                Cancel
+              </Button>
+              <Button type="primary" danger htmlType="submit" loading={mfaActionLoading}>
+                Reset MFA
+              </Button>
+            </Space>
+          </Form.Item>
+        </Form>
+      </Modal>
     </div>
   );
 };

frontend/src/components/VIPManagement.js

@@ -0,0 +1,583 @@
+import React, { useState, useEffect, useCallback } from 'react';
+import {
+  Table, Button, Space, Modal, Form, Input, InputNumber, Select, Tag, message,
+  Switch, Typography, Card, Alert, Tooltip, Spin
+} from 'antd';
+import {
+  PlusOutlined, EditOutlined, DeleteOutlined, ReloadOutlined, WarningOutlined,
+  PlayCircleOutlined, SyncOutlined, CrownOutlined, ClockCircleOutlined, FileSearchOutlined,
+  InfoCircleOutlined
+} from '@ant-design/icons';
+import { useCluster } from '../contexts/ClusterContext';
+import { extractApiError } from '../utils/apiError';
+
+const { Option } = Select;
+const { Title, Text } = Typography;
+
+// Interfaces that are never sensible VIP carriers — hidden from the dropdown.
+const IFACE_HIDE = /^(lo|docker|veth|br-|cni|flannel|kube|virbr)/i;
+
+const authHeaders = () => ({
+  'Content-Type': 'application/json',
+  'Authorization': `Bearer ${localStorage.getItem('authToken') || ''}`,
+});
+
+// Convergence-aware status (issue #27 follow-up): like other entities, a VIP only reads
+// "live" once its member agents have actually deployed & acked — never the instant Apply
+// is clicked. Backend returns deploy_status; we fall back to last_config_status.
+const DEPLOY_STATUS = {
+  PENDING:   { color: 'orange',     label: 'PENDING',  tip: 'Staged change — review and Apply (or Reject) it from the Apply Management page.' },
+  PENDING_DELETE: { color: 'volcano', label: 'PENDING DELETE', tip: 'Deletion staged for approval — the VIP keeps running untouched until you APPROVE it in Apply Management. Reject to keep it. The node is not changed until approval.' },
+  DELETING:  { color: 'processing', label: 'DELETING', tip: 'Deletion approved — member node(s) are stopping keepalived and releasing the VIP. Disappears once every node has torn down.' },
+  DELETED:   { color: 'default',    label: 'DELETED',  tip: 'All member nodes have torn keepalived down.' },
+  SYNCING:   { color: 'processing', label: 'SYNCING',  tip: 'Applied — an online member node is installing/configuring keepalived and will acknowledge on its next poll (~2–3 min).' },
+  AWAITING:  { color: 'gold',       label: 'AWAITING AGENT', tip: 'Applied, but the member node(s) that still need it are OFFLINE, so nothing can deploy yet. Bring the node\'s agent online — it converges on its next poll. (Not a hang.)' },
+  ACTIVE:    { color: 'green',      label: 'ACTIVE',   tip: 'Applied and every member node has deployed keepalived and acknowledged the current config.' },
+  ERROR:     { color: 'red',        label: 'ERROR',    tip: 'A member node failed to deploy keepalived — see Members / Live state for the node, then check that agent.' },
+  ATTENTION: { color: 'gold',       label: 'ATTENTION',tip: 'A member already runs a hand-managed keepalived; the agent left it untouched (externally managed). Resolve it on that node or remove it from the VIP.' },
+  APPLIED:   { color: 'green',      label: 'APPLIED',  tip: 'Applied.' },
+};
+
+// agents.capabilities / network_interfaces come from the API as JSONB → a JSON string
+// (asyncpg has no jsonb codec). Mirror AgentManagement.js and JSON.parse when needed.
+const parseArr = (v) => {
+  if (Array.isArray(v)) return v;
+  if (typeof v === 'string') { try { const p = JSON.parse(v); return Array.isArray(p) ? p : []; } catch { return []; } }
+  return [];
+};
+
+// This component uses raw fetch(), but extractApiError expects an axios-shaped error
+// (err.response.data). Read the fetch Response body and reuse the envelope-aware extractor
+// so backend messages — e.g. the 409 "node already in VIP X" — actually reach the user.
+const fetchApiError = async (res, fallback) => {
+  try { const data = await res.json(); return extractApiError({ response: { data } }, fallback); }
+  catch { return fallback; }
+};
+
+const VIPManagement = () => {
+  const { clusters } = useCluster();
+  const [vips, setVips] = useState([]);
+  const [loading, setLoading] = useState(false);
+  const [modalVisible, setModalVisible] = useState(false);
+  const [editing, setEditing] = useState(null);
+  const [selectedPoolId, setSelectedPoolId] = useState(null);
+  // One row per agent in the selected pool — the user toggles which participate.
+  const [memberRows, setMemberRows] = useState([]);
+  const [form] = Form.useForm();
+  // Delete confirmation (with opt-in package uninstall) + diagnostics modal state.
+  const [deleteTarget, setDeleteTarget] = useState(null);
+  const [showL2Note, setShowL2Note] = useState(false);
+  const [diagVip, setDiagVip] = useState(null);
+  const [diagData, setDiagData] = useState(null);
+  const [diagLoading, setDiagLoading] = useState(false);
+
+  // Distinct pools derived from the cluster list (cluster -> pool_id).
+  const pools = React.useMemo(() => {
+    const seen = new Map();
+    (clusters || []).forEach((c) => {
+      if (c.pool_id && !seen.has(c.pool_id)) seen.set(c.pool_id, c.name || `pool ${c.pool_id}`);
+    });
+    return Array.from(seen, ([id, name]) => ({ id, name }));
+  }, [clusters]);
+
+  const fetchVips = useCallback(async () => {
+    setLoading(true);
+    try {
+      const res = await fetch('/api/vip', { headers: authHeaders() });
+      if (res.ok) {
+        const data = await res.json();
+        setVips(data.vips || []);
+      } else if (res.status === 403) {
+        message.warning('You do not have permission to view VIPs (vip.read).');
+        setVips([]);
+      }
+    } catch (e) {
+      console.error('fetchVips failed', e);
+    } finally {
+      setLoading(false);
+    }
+  }, []);
+
+  useEffect(() => {
+    fetchVips();
+    const t = setInterval(fetchVips, 30000); // live MASTER/BACKUP via existing detection pipeline
+    return () => clearInterval(t);
+  }, [fetchVips]);
+
+  // Build the participating-nodes table from the pool's EXISTING agents (installed via the
+  // standard Agent Management process). On edit, pre-select the VIP's current members.
+  const buildMemberRows = (agents, existing) => {
+    const ex = {};
+    (existing || []).forEach((m) => { ex[m.agent_id] = m; });
+    return (agents || [])
+      .filter((a) => !String(a.name).startsWith('token_'))
+      .map((a) => {
+        const interfaces = parseArr(a.network_interfaces).filter((n) => !IFACE_HIDE.test(n));
+        const e = ex[a.id];
+        return {
+          agent_id: a.id,
+          agent_name: a.name,
+          ip_address: a.ip_address,
+          capable: parseArr(a.capabilities).includes('keepalived_management'),
+          interfaces,
+          participate: !!e,
+          role: e ? e.role : 'BACKUP',
+          priority: e ? e.priority : 100,
+          network_interface: e ? e.network_interface : (interfaces[0] || ''),
+        };
+      });
+  };
+
+  const loadPoolMembers = useCallback(async (poolId, existing) => {
+    if (!poolId) { setMemberRows([]); return; }
+    try {
+      const res = await fetch(`/api/agents?pool_id=${poolId}`, { headers: authHeaders() });
+      const data = res.ok ? await res.json() : { agents: [] };
+      setMemberRows(buildMemberRows(data.agents || [], existing));
+    } catch (e) {
+      console.error('loadPoolMembers failed', e);
+      setMemberRows([]);
+    }
+  }, []);
+
+  const setRow = (agentId, patch) =>
+    setMemberRows((rows) => rows.map((r) => (r.agent_id === agentId ? { ...r, ...patch } : r)));
+
+  // Toggling a node into the VIP: if no other participating node is MASTER yet, make this
+  // one the MASTER. This makes the single-node case work without the operator having to flip
+  // the role by hand (a one-node VIP's only node IS the master), and gives a sensible default
+  // for multi-node (first picked = master, the rest backup). Editing keeps stored roles.
+  const toggleParticipate = (agentId, on) =>
+    setMemberRows((rows) => {
+      const otherMaster = rows.some((r) => r.agent_id !== agentId && r.participate && r.role === 'MASTER');
+      return rows.map((r) => {
+        if (r.agent_id !== agentId) return r;
+        if (on && !otherMaster) return { ...r, participate: true, role: 'MASTER', priority: 150 };
+        return { ...r, participate: on };
+      });
+    });
+
+  const openCreate = () => {
+    setEditing(null);
+    setSelectedPoolId(null);
+    setMemberRows([]);
+    form.resetFields();
+    form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true });
+    setModalVisible(true);
+  };
+
+  const openEdit = (vip) => {
+    setEditing(vip);
+    setSelectedPoolId(vip.pool_id);
+    loadPoolMembers(vip.pool_id, vip.members);
+    form.resetFields();
+    form.setFieldsValue({
+      name: vip.name, description: vip.description, pool_id: vip.pool_id,
+      virtual_ip: vip.virtual_ip, prefix_length: vip.prefix_length,
+      virtual_router_id: vip.virtual_router_id, advert_int: vip.advert_int,
+      use_unicast: vip.use_unicast, track_haproxy: vip.track_haproxy,
+    });
+    setModalVisible(true);
+  };
+
+  const submit = async () => {
+    let values;
+    try { values = await form.validateFields(); }
+    catch { return; }
+
+    const chosen = memberRows.filter((r) => r.participate);
+    if (chosen.length < 1) { message.error('Select at least 1 participating node.'); return; }
+    const masters = chosen.filter((r) => r.role === 'MASTER');
+    if (masters.length !== 1) { message.error('Exactly one participating node must be MASTER.'); return; }
+    if (chosen.some((r) => !r.network_interface)) { message.error('Pick a network interface for every participating node.'); return; }
+    const maxBackup = Math.max(...chosen.filter((r) => r.role === 'BACKUP').map((r) => r.priority));
+    if (masters[0].priority <= maxBackup) { message.error('The MASTER must have a higher priority than every BACKUP.'); return; }
+
+    const body = {
+      ...values,
+      members: chosen.map((r) => ({
+        agent_id: r.agent_id, network_interface: r.network_interface, role: r.role, priority: r.priority,
+      })),
+    };
+    if (!body.auth_pass) delete body.auth_pass; // omit to keep existing on edit
+    try {
+      const url = editing ? `/api/vip/${editing.id}` : '/api/vip';
+      const res = await fetch(url, { method: editing ? 'PUT' : 'POST', headers: authHeaders(), body: JSON.stringify(body) });
+      if (res.ok) {
+        message.success(editing
+          ? 'VIP updated (PENDING) — apply it from the Apply Management page'
+          : 'VIP created (PENDING) — apply it from the Apply Management page');
+        setModalVisible(false);
+        fetchVips();
+      } else {
+        message.error(await fetchApiError(res, 'Failed to save VIP'));
+      }
+    } catch (e) {
+      message.error('Failed to save VIP: ' + e.message);
+    }
+  };
+
+  const deleteVip = async (vip, purge) => {
+    try {
+      const res = await fetch(`/api/vip/${vip.id}${purge ? '?purge_package=true' : ''}`,
+        { method: 'DELETE', headers: authHeaders() });
+      if (res.ok) {
+        let body = {};
+        try { body = await res.json(); } catch (_) { /* ignore */ }
+        // Backend returns staged=true (approval required, VIP still running) or staged=false
+        // (never-applied VIP removed at once). Surface its exact message either way.
+        (body.staged ? message.info : message.success)(
+          body.message || 'Deletion requested.');
+        setDeleteTarget(null); fetchVips();
+      } else message.error(await fetchApiError(res, 'Delete failed'));
+    } catch (e) { message.error('Delete failed: ' + e.message); }
+  };
+
+  // Diagnostics: per-member deploy state/ack from GET /api/vip/{id}/status — the live view
+  // of what each node reported (installing/applied/error/externally-managed), most useful
+  // while a freshly-applied VIP is SYNCING (keepalived install can take ~30s).
+  const openDiagnostics = async (vip) => {
+    setDiagVip(vip); setDiagData(null); setDiagLoading(true);
+    try {
+      const res = await fetch(`/api/vip/${vip.id}/status`, { headers: authHeaders() });
+      if (res.ok) setDiagData(await res.json());
+      else message.error(await fetchApiError(res, 'Failed to load diagnostics'));
+    } catch (e) { message.error('Diagnostics failed: ' + e.message); }
+    finally { setDiagLoading(false); }
+  };
+
+  // Colorful, IP-Inventory/Agent-consistent state: live VRRP MASTER (green, crowned) /
+  // BACKUP (orange) / FAULT (red); when the agent hasn't reported a live state yet, show
+  // the configured role as a dashed outline tag (same color) so it's clearly "intended,
+  // not yet observed". A node whose agent is too old gets an "awaiting agent" flag.
+  const renderMembers = (_, vip) => (
+    <Space direction="vertical" size={4}>
+      {(vip.members || []).map((m) => {
+        const live = m.keepalive_state && m.keepalive_state !== 'NONE' ? m.keepalive_state : null;
+        const roleColor = m.role === 'MASTER' ? 'green' : 'orange';
+        const awaiting = !live && !m.keepalived_capable;
+        return (
+          <Space key={m.agent_id} size={6}>
+            <Text style={{ fontSize: 12 }}>{m.agent_name || `agent ${m.agent_id}`}</Text>
+            {live ? (
+              <Tooltip title={`Live VRRP state: ${live}`}>
+                <Tag
+                  color={live === 'MASTER' ? 'green' : live === 'BACKUP' ? 'orange' : 'red'}
+                  icon={live === 'MASTER' ? <CrownOutlined /> : undefined}
+                  style={{ marginInlineEnd: 0, fontWeight: 600 }}
+                >
+                  {live}
+                </Tag>
+              </Tooltip>
+            ) : (
+              <Tooltip title="Configured role — live VRRP state not observed yet (agent offline or still converging).">
+                <Tag color={roleColor} style={{ marginInlineEnd: 0, borderStyle: 'dashed', opacity: 0.85 }}>
+                  {m.role}
+                </Tag>
+              </Tooltip>
+            )}
+            {awaiting && (
+              <Tooltip title="This node's agent does not advertise keepalived_management — upgrade the agent.">
+                <Tag color="gold" icon={<WarningOutlined />} style={{ marginInlineEnd: 0 }}>awaiting agent</Tag>
+              </Tooltip>
+            )}
+          </Space>
+        );
+      })}
+    </Space>
+  );
+
+  const columns = [
+    { title: 'Name', dataIndex: 'name', key: 'name' },
+    { title: 'Virtual IP', key: 'vip', render: (_, v) => <Text code>{v.virtual_ip}/{v.prefix_length}</Text> },
+    { title: 'Pool', dataIndex: 'pool_name', key: 'pool' },
+    { title: 'VRID', dataIndex: 'virtual_router_id', key: 'vrid' },
+    { title: 'Members / Live state', key: 'members', render: renderMembers },
+    {
+      title: 'Status', key: 'status', render: (_, v) => {
+        const s = v.deploy_status || v.last_config_status;
+        const d = DEPLOY_STATUS[s] || { color: 'default', label: s, tip: '' };
+        const count = (s === 'SYNCING' || s === 'AWAITING' || s === 'ACTIVE' || s === 'DELETING' || s === 'DELETED') && v.deploy_total
+          ? ` (${v.deploy_synced}/${v.deploy_total})` : '';
+        return (
+          <Tooltip title={d.tip}>
+            <Tag color={d.color} icon={(s === 'SYNCING' || s === 'DELETING') ? <SyncOutlined spin /> : s === 'AWAITING' ? <ClockCircleOutlined /> : undefined}>
+              {(d.label || s)}{count}
+            </Tag>
+          </Tooltip>
+        );
+      },
+    },
+    {
+      title: 'Actions', key: 'actions', render: (_, v) => (
+        <Space>
+          {v.last_config_status === 'PENDING' && (
+            <Tooltip title="Apply pending configuration changes">
+              <Button type="primary" size="small" icon={<PlayCircleOutlined />}
+                onClick={() => { window.location.href = '/apply-management'; }}
+                style={{ backgroundColor: '#1890ff', borderColor: '#1890ff' }}>
+                Apply
+              </Button>
+            </Tooltip>
+          )}
+          <Tooltip title="Edit VIP (changes become PENDING; apply from Apply Management)">
+            <Button size="small" icon={<EditOutlined />} onClick={() => openEdit(v)} />
+          </Tooltip>
+          <Tooltip title="Diagnostics — per-node keepalived deploy status & logs">
+            <Button size="small" icon={<FileSearchOutlined />} onClick={() => openDiagnostics(v)} />
+          </Tooltip>
+          <Tooltip title="Delete VIP">
+            <Button size="small" danger icon={<DeleteOutlined />}
+              onClick={() => setDeleteTarget(v)} />
+          </Tooltip>
+        </Space>
+      ),
+    },
+  ];
+
+  // Member-selection table inside the modal — the pool's installed agents (nodes).
+  const memberColumns = [
+    {
+      title: 'Node (agent)', key: 'node', render: (_, r) => (
+        <span>
+          <Text strong>{r.agent_name}</Text>{' '}
+          <Text type="secondary" style={{ fontSize: 12 }}>{r.ip_address ? `(${r.ip_address})` : '(no IP yet)'}</Text>
+          {!r.capable && (
+            <Tooltip title="This agent doesn't advertise keepalived_management — upgrade it or this node won't deploy.">
+              {' '}<Tag color="gold" icon={<WarningOutlined />}>agent too old</Tag>
+            </Tooltip>
+          )}
+        </span>
+      ),
+    },
+    {
+      title: 'Participate', key: 'participate', width: 100, render: (_, r) => (
+        <Switch checked={r.participate} onChange={(c) => toggleParticipate(r.agent_id, c)} />
+      ),
+    },
+    {
+      title: 'Role', key: 'role', width: 130, render: (_, r) => (
+        <Select size="small" style={{ width: 110 }} value={r.role} disabled={!r.participate}
+          onChange={(val) => setRow(r.agent_id, { role: val, priority: val === 'MASTER' ? 150 : 100 })}>
+          <Option value="MASTER">MASTER</Option>
+          <Option value="BACKUP">BACKUP</Option>
+        </Select>
+      ),
+    },
+    {
+      title: 'Priority', key: 'priority', width: 110, render: (_, r) => (
+        <InputNumber size="small" min={1} max={254} value={r.priority} disabled={!r.participate}
+          onChange={(val) => setRow(r.agent_id, { priority: val })} />
+      ),
+    },
+    {
+      title: 'Interface', key: 'iface', width: 160, render: (_, r) => (
+        <Select size="small" style={{ width: 140 }} value={r.network_interface || undefined}
+          placeholder="interface" disabled={!r.participate} showSearch
+          onChange={(val) => setRow(r.agent_id, { network_interface: val })}
+          notFoundContent="no interfaces reported"
+          options={(r.interfaces || []).map((n) => ({ label: n, value: n }))}
+          {...((r.interfaces || []).length === 0 ? { mode: 'tags' } : {})} />
+      ),
+    },
+  ];
+
+  return (
+    <div>
+      <Card>
+        <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: 16 }}>
+          <Title level={2} style={{ margin: 0 }}>HA / VIP (Keepalived)</Title>
+          <Space>
+            <Button icon={<ReloadOutlined />} onClick={fetchVips}>Refresh</Button>
+            <Button type="primary" icon={<PlusOutlined />} onClick={openCreate}>Create VIP</Button>
+          </Space>
+        </div>
+        {/* The cloud caveat is rarely relevant for the on-prem target audience, so it's a
+            subtle, collapsed-by-default info note (not a prominent yellow warning). */}
+        <div style={{ marginBottom: 12 }}>
+          <Button type="link" size="small" icon={<InfoCircleOutlined />} style={{ paddingLeft: 0 }}
+            onClick={() => setShowL2Note((v) => !v)}>
+            Network requirements (on-prem / L2)
+          </Button>
+          {showL2Note && (
+            <Alert
+              type="info" showIcon style={{ marginTop: 4 }}
+              message="On-prem / L2 networks"
+              description="VRRP-based VIP failover targets bare-metal / VMware / on-prem L2 segments. On AWS/Azure/GCP, cloud fabrics don't honor VRRP/gratuitous-ARP, so VIPs won't move. Ensure VRRP (IP protocol 112) is permitted by host firewalls."
+            />
+          )}
+        </div>
+        <Table rowKey="id" columns={columns} dataSource={vips} loading={loading} pagination={{ pageSize: 10 }} />
+      </Card>
+
+      <Modal
+        title={editing ? `Edit VIP — ${editing.name}` : 'Create VIP'}
+        open={modalVisible}
+        onCancel={() => setModalVisible(false)}
+        onOk={submit}
+        okText={editing ? 'Save (PENDING)' : 'Create (PENDING)'}
+        width={880}
+        destroyOnClose
+      >
+        <Form form={form} layout="vertical">
+          <Form.Item name="name" label="Name" rules={[{ required: true }]}>
+            <Input placeholder="web-vip" disabled={!!editing} />
+          </Form.Item>
+          <Form.Item name="description" label="Description">
+            <Input placeholder="optional" />
+          </Form.Item>
+          {!editing && (
+            <Form.Item name="pool_id" label="Pool" rules={[{ required: true }]}
+              tooltip="The VIP's nodes are the HAProxy servers (agents) already enrolled in this pool.">
+              <Select placeholder="Select a pool" onChange={(pid) => { setSelectedPoolId(pid); loadPoolMembers(pid); }}>
+                {pools.map((p) => <Option key={p.id} value={p.id}>{p.name}</Option>)}
+              </Select>
+            </Form.Item>
+          )}
+          <Space size="large" style={{ display: 'flex' }}>
+            <Form.Item name="virtual_ip" label="Virtual IP (IPv4)" rules={[{ required: true }]}>
+              <Input placeholder="10.0.0.100" />
+            </Form.Item>
+            <Form.Item name="prefix_length" label="Prefix" rules={[{ required: true }]}>
+              <InputNumber min={1} max={32} />
+            </Form.Item>
+            <Form.Item name="virtual_router_id" label="VRID (blank = auto)">
+              <InputNumber min={1} max={255} placeholder="auto" />
+            </Form.Item>
+            <Form.Item name="advert_int" label="Advert int (s)">
+              <InputNumber min={1} max={255} />
+            </Form.Item>
+          </Space>
+          <Space size="large">
+            <Form.Item name="use_unicast" label="Unicast VRRP" valuePropName="checked" tooltip="Recommended; works where multicast is blocked.">
+              <Switch />
+            </Form.Item>
+            <Form.Item name="track_haproxy" label="Fail over when HAProxy drops" valuePropName="checked">
+              <Switch />
+            </Form.Item>
+            <Form.Item name="auth_pass" label="VRRP secret (≤8 chars)">
+              <Input.Password placeholder={editing ? '•••• (unchanged)' : 'optional'} maxLength={8} />
+            </Form.Item>
+          </Space>
+
+          <Text strong>Participating nodes</Text>
+          <div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
+            These are the HAProxy servers (agents) already enrolled in this pool — toggle which join the VIP.
+            Pick <b>exactly one MASTER</b> (highest priority); the rest are BACKUP. A <b>single node</b> is allowed
+            (a keepalived-managed VIP <i>without</i> failover) — add a second node for real HA. Add new servers from
+            the standard Agent Management install flow.
+          </div>
+          <Alert
+            type="info" showIcon style={{ marginBottom: 8 }}
+            message="keepalived is installed automatically on Apply"
+            description={<>On Apply, any participating node that doesn’t already run keepalived will <b>install it from the node’s OS package repositories</b> (apt/dnf/yum/zypper/apk) — make sure the node can reach its repos (internet or an internal mirror). A node already running a <b>hand-managed</b> keepalived is left untouched (reported as “externally managed”).</>}
+          />
+          <Table
+            rowKey="agent_id"
+            size="small"
+            columns={memberColumns}
+            dataSource={memberRows}
+            pagination={false}
+            locale={{ emptyText: selectedPoolId ? 'No agents in this pool — install agents from Agent Management first.' : 'Select a pool to list its nodes.' }}
+          />
+        </Form>
+      </Modal>
+
+      {/* Delete confirmation with opt-in package uninstall. Enterprise-safe DEFAULT keeps the
+          package (just stop/disable + remove our config + release the VIP). */}
+      <Modal
+        title="Delete VIP — requires approval"
+        open={!!deleteTarget}
+        onCancel={() => setDeleteTarget(null)}
+        onOk={() => deleteVip(deleteTarget, false)}
+        okText="Stage deletion for approval"
+        okButtonProps={{ danger: true }}
+      >
+        {deleteTarget && (
+          <Space direction="vertical" size={12} style={{ width: '100%' }}>
+            <Alert
+              type="warning"
+              showIcon
+              message="This does NOT delete the VIP immediately"
+              description={<>It stages the deletion for approval. The VIP <b>keeps running, untouched</b>, on its
+                member node(s) until you <b>Approve</b> it on the <b>Apply Management</b> page — and you can
+                <b> Reject</b> it there to keep it. The node is changed <b>only after approval</b>, so an
+                accidental click can't tear down a production VIP.</>}
+            />
+            <Text>
+              Stage deletion of <Text strong>{deleteTarget.name}</Text> ({deleteTarget.virtual_ip}/{deleteTarget.prefix_length})?
+              When approved, the member node(s) <b>stop &amp; disable keepalived, remove the config we manage, and release the VIP</b>. The keepalived package itself is left installed, so re-adding a VIP later is instant.
+            </Text>
+          </Space>
+        )}
+      </Modal>
+
+      {/* Per-node keepalived deploy diagnostics + node-side log commands (esp. during SYNCING). */}
+      <Modal
+        title={diagVip ? `Diagnostics — ${diagVip.name}` : 'Diagnostics'}
+        open={!!diagVip}
+        onCancel={() => { setDiagVip(null); setDiagData(null); }}
+        width={780}
+        footer={[
+          <Button key="refresh" icon={<ReloadOutlined />} onClick={() => diagVip && openDiagnostics(diagVip)}>Refresh</Button>,
+          <Button key="close" type="primary" onClick={() => { setDiagVip(null); setDiagData(null); }}>Close</Button>,
+        ]}
+      >
+        {diagLoading && <div style={{ textAlign: 'center', padding: 24 }}><Spin /></div>}
+        {!diagLoading && diagData && (
+          <Space direction="vertical" size={12} style={{ width: '100%' }}>
+            <Text type="secondary">
+              Staging <Tag>{diagData.last_config_status}</Tag> — each node reports its deploy state after every poll; a fresh keepalived install can take ~30s.
+            </Text>
+            <Table
+              size="small" rowKey={(m) => m.agent_name} pagination={false}
+              dataSource={diagData.members || []}
+              columns={[
+                { title: 'Node', key: 'node', render: (_, m) => (
+                  <Space size={4}>
+                    <Text style={{ fontSize: 12 }}>{m.agent_name}</Text>
+                    {m.role === 'MASTER'
+                      ? <Tag color="green" icon={<CrownOutlined />} style={{ marginInlineEnd: 0 }}>MASTER</Tag>
+                      : <Tag color="orange" style={{ marginInlineEnd: 0 }}>BACKUP</Tag>}
+                  </Space>) },
+                { title: 'Agent', dataIndex: 'agent_status', key: 'agent',
+                  render: (s) => <Tag color={s === 'online' ? 'green' : 'red'}>{s || 'offline'}</Tag> },
+                { title: 'Live VRRP', dataIndex: 'keepalive_state', key: 'live',
+                  render: (s) => (s && s !== 'NONE')
+                    ? <Tag color={s === 'MASTER' ? 'green' : s === 'BACKUP' ? 'orange' : 'red'}>{s}</Tag>
+                    : <Text type="secondary">—</Text> },
+                { title: 'Deploy', key: 'deploy', render: (_, m) => {
+                    const st = m.deploy_state;
+                    const color = st === 'enabled' ? 'green' : st === 'error' ? 'red'
+                      : st === 'externally_managed' ? 'gold' : 'blue';
+                    return <Tooltip title={m.deploy_message || ''}><Tag color={color}>{m.convergence || st || 'pending'}</Tag></Tooltip>;
+                  } },
+                { title: 'Last ack', dataIndex: 'deploy_at', key: 'ack',
+                  render: (t) => t ? <Text style={{ fontSize: 11 }}>{new Date(t).toLocaleString()}</Text> : <Text type="secondary">—</Text> },
+              ]}
+            />
+            {(diagData.members || []).some((m) => m.deploy_message) && (
+              <Card size="small" title="Latest node messages" bodyStyle={{ padding: 8 }}>
+                {(diagData.members || []).filter((m) => m.deploy_message).map((m) => (
+                  <div key={m.agent_name} style={{ fontSize: 12 }}><Text strong>{m.agent_name}:</Text> {m.deploy_message}</div>
+                ))}
+              </Card>
+            )}
+            <Alert
+              type="info" showIcon
+              message="See the live install / VRRP logs on the node"
+              description={
+                <pre style={{ margin: 0, fontSize: 11, whiteSpace: 'pre-wrap' }}>{`systemctl status keepalived --no-pager
+journalctl -u keepalived --no-pager -n 50
+tail -n 100 /var/log/haproxy-agent/agent.log | grep -i keepalived`}</pre>
+              }
+            />
+          </Space>
+        )}
+        {!diagLoading && !diagData && <Text type="secondary">No diagnostics available.</Text>}
+      </Modal>
+    </div>
+  );
+};
+
+export default VIPManagement;

frontend/src/setupProxy.js

@@ -0,0 +1,31 @@
+/**
+ * CRA dev-server proxy.
+ *
+ * This file is consumed by Webpack DevServer (via `react-scripts start`)
+ * EXCLUSIVELY. It is NOT included in the production bundle (`npm run
+ * build` / `serve -s build`) and has zero effect in Kubernetes / Docker
+ * Compose deployments where nginx (`/api/*` → backend) handles routing.
+ *
+ * Default target is `http://localhost:8000` because the only realistic
+ * use of `npm start` is on a developer's host machine, where the backend
+ * is reachable on localhost. The target can be overridden with the
+ * `PROXY_TARGET` environment variable (e.g. `PROXY_TARGET=http://other-host:8000 npm start`).
+ *
+ * Why not `http://backend:8000`? That hostname only resolves inside the
+ * Docker Compose network. Defaulting to it caused all `/api/*` requests
+ * to fail with ENOTFOUND when running the dev-server on the host.
+ */
+const { createProxyMiddleware } = require('http-proxy-middleware');
+
+const target = process.env.PROXY_TARGET || 'http://localhost:8000';
+
+module.exports = function (app) {
+  app.use(
+    '/api',
+    createProxyMiddleware({
+      target,
+      changeOrigin: true,
+      logLevel: 'warn',
+    })
+  );
+};

frontend/src/utils/api.js

@@ -1,32 +1,28 @@
 /**
  * API Configuration
- * Centralized API URL management for the application
+ * Centralized API URL management for the application.
- * 
+ *
- * Environment Variables:
+ * Resolution strategy (same in dev and prod — keeps SPA same-origin):
- * - REACT_APP_API_URL: Full API URL (e.g., https://api.example.com)
+ *   1) REACT_APP_API_URL — explicit override at BUILD time. Use only when
- * - NODE_ENV: Environment (development, production)
+ *      the SPA must call a cross-origin API (CORS must be enabled there).
+ *   2) window.location.{protocol,host} — same-origin. In dev this routes
+ *      through CRA dev-server proxy (see frontend/src/setupProxy.js); in
+ *      prod through nginx ingress (`/api/*` → backend service).
+ *   3) Empty string — non-browser env (SSR/tests). Yields relative URLs.
+ *
+ * NOTE: Do NOT hardcode `http://localhost:8000` here. Even in unreachable
+ * branches CRA/Terser keeps string literals in the bundle, which would
+ * confuse anyone auditing the production artifact.
  */
-
-// Get API URL from environment or use default based on environment
 const getApiUrl = () => {
-  // Priority 1: Explicit environment variable
   if (process.env.REACT_APP_API_URL) {
     return process.env.REACT_APP_API_URL;
   }
-  
-  // Priority 2: Detect from window location (for production deployments)
-  // Use window.location.host (includes port) instead of hostname (excludes port)
-  // so non-standard ports like :8080 are preserved, preventing CORS issues
   if (typeof window !== 'undefined' && window.location) {
     const { protocol, host } = window.location;
-    
+    return `${protocol}//${host}`;
-    if (process.env.NODE_ENV === 'production') {
-      return `${protocol}//${host}`;
-    }
   }
-  
+  return '';
-  // Priority 3: Development default
-  return 'http://localhost:8000';
 };
 
 // API Base URL

k8s/manifests/03-secrets.yaml

@@ -23,6 +23,8 @@ metadata:
     app: haproxy-openmanager
     component: backend
 type: Opaque
-data:
+# Both values are replaced at deploy time by the CI/CD pipeline (sed step)
-  # your-secret-key-change-this-in-production
+# before `kubectl apply` runs. Never commit real secrets — use the placeholders.
-  SECRET_KEY: eW91ci1zZWNyZXQta2V5LWNoYW5nZS10aGlzLWluLXByb2R1Y3Rpb24= 
+stringData:
+  SECRET_KEY: secret_key_replace_me
+  MFA_ENCRYPTION_KEY: mfa_encryption_key_replace_me

k8s/manifests/07-configmaps.yaml

@@ -15,6 +15,33 @@ data:
   PUBLIC_URL: 'https://haproxy-openmanager.example.com'
   MANAGEMENT_BASE_URL: 'https://haproxy-openmanager.example.com'
 
+  # ---------------------------------------------------------------------------
+  # MFA rate-limits (slowapi format: "<count>/<second|minute|hour|day>").
+  # The shipped defaults assume the rate-limit key is per-USER (Bearer JWT)
+  # with a per-IP fallback. They are sized for thousands of concurrent
+  # operators in an org-wide MFA rollout. Pod restart required to apply.
+  #
+  # Defaults live in backend/middleware/mfa_rate_limits.py:
+  #   ENROLL_START='10/minute'      ENROLL_CONFIRM='10/minute'
+  #   DISABLE='10/minute'           REGENERATE_BACKUP_CODES='5/hour'
+  #   ADMIN_RESET='60/hour'         ADMIN_RESET_ALL='1/day'
+  # ---------------------------------------------------------------------------
+  # MFA_RATE_LIMIT_ENROLL_START: '10/minute'
+  # MFA_RATE_LIMIT_ENROLL_CONFIRM: '10/minute'
+  # MFA_RATE_LIMIT_DISABLE: '10/minute'
+  # MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES: '5/hour'
+  # MFA_RATE_LIMIT_ADMIN_RESET: '60/hour'
+  # MFA_RATE_LIMIT_ADMIN_RESET_ALL: '1/day'
+
+  # ---------------------------------------------------------------------------
+  # Trusted reverse-proxy CIDRs for X-Forwarded-For. When the request peer
+  # falls inside one of these CIDRs and the user is NOT authenticated, the
+  # first hop of XFF becomes the rate-limit bucket. Empty (default) disables
+  # XFF parsing entirely — the safe choice when the topology is unknown.
+  # Example (in-cluster service mesh + nginx ingress on the default ranges):
+  # ---------------------------------------------------------------------------
+  # MFA_TRUSTED_PROXY_CIDRS: '10.0.0.0/8,172.16.0.0/12,192.168.0.0/16'
+
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -25,9 +52,20 @@ metadata:
     app: haproxy-openmanager
     component: frontend
 data:
-  # Leave empty to use same-origin (window.location) in production
+  # NOTE — REACT_APP_* env vars are RUNTIME no-ops here.
-  # For development, set to: 'http://localhost:8000'
+  # Create-React-App inlines REACT_APP_* values into the bundle at BUILD time
-  REACT_APP_API_URL: ''
+  # (`npm run build`). The static bundle served by `serve -s build` does not
+  # consume runtime env vars, so anything set here has zero effect on
+  # window.location-based same-origin routing.
+  #
+  # The frontend always issues `/api/*` against window.location, which is
+  # routed by the ingress + nginx ConfigMap below to the backend service.
+  # No env vars are required for the frontend deployment.
+  #
+  # WARNING: Do NOT set REACT_APP_API_URL in your CI/CD pipeline either —
+  # if a value is supplied at build time it gets HARDCODED into the bundle
+  # and breaks every deployment that does not match that exact URL.
+  {}
 
 ---
 apiVersion: v1

k8s/manifests/10-nginx.yaml

@@ -22,7 +22,9 @@ spec:
       serviceAccountName: haproxy-openmanager-nginx
       containers:
       - name: nginx
-        image: nginx:alpine
+        # Pinned to a patched release for the nginx "poolslip" advisory
+        # (mainline <=1.31.0 affected; fixed in mainline 1.31.1+ / stable 1.30.2+).
+        image: nginx:1.31.1-alpine
         ports:
         - containerPort: 8080
           name: http

scripts/README.md

@@ -75,6 +75,35 @@ Test HAProxy stats parsing
 ### test-build.sh
 Run build tests for the project
 
+## 🔐 MFA Admin Scripts
+
+### admin-mfa-reset-all.sh
+**Purpose:** Emergency — disable Multi-Factor Authentication for **every** user
+in one call. Use only when there is a mass loss of authenticator devices /
+inherited platform without working operators (Issue #18, v1.6.0).
+
+**Usage:**
+```bash
+# Interactive prompts ask for the admin Bearer token + reason
+./scripts/admin-mfa-reset-all.sh
+
+# Non-interactive (still requires double confirmation typed at the keyboard)
+API_URL=https://hap.example.com \
+ADMIN_TOKEN=eyJhbGciOi... \
+  ./scripts/admin-mfa-reset-all.sh
+```
+
+**Features:**
+- Calls `POST /api/mfa/admin-reset-all` (requires `users.is_admin = TRUE`)
+- Double confirmation: type `yes`, then `RESET ALL MFA` exactly
+- Required reason is recorded in `user_activity_logs`
+  (`action='mfa.disabled.admin_bulk_reset'`)
+- Deletes every backup code and invalidates pending MFA challenges
+
+**Safety:**
+- Irreversible — all users must re-enroll MFA afterwards
+- All other authentication (password, JWT, roles) is unaffected
+
 ## 🚨 Emergency Use Cases
 
 **1. Cluster Migration/Cleanup:**
@@ -101,6 +130,12 @@ Run build tests for the project
 ./scripts/debug-agent-stats-function.sh
 ```
 
+**4. MFA Outage (mass lost authenticators):**
+```bash
+# Disable MFA for every user, then ask them to re-enroll
+./scripts/admin-mfa-reset-all.sh
+```
+
 ## ⚠️ Safety Notes
 
 - All cleanup scripts require admin authentication

scripts/admin-mfa-reset-all.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# scripts/admin-mfa-reset-all.sh
+# Issue #18 — v1.6.0 — Emergency: disable MFA for ALL users in one call.
+#
+# This is a break-glass tool. It calls POST /api/mfa/admin-reset-all on the
+# backend with a strict, double-confirmed body and writes the action into the
+# server's audit log (action=mfa.disabled.admin_bulk_reset). All users will be
+# able to log in with username/password alone afterwards and must re-enroll if
+# they want MFA again.
+#
+# Usage:
+#   ./scripts/admin-mfa-reset-all.sh
+#   API_URL=https://hap.example.com ADMIN_TOKEN=ey... ./scripts/admin-mfa-reset-all.sh
+#
+# Required: a JWT bearer token belonging to a user whose `users.is_admin = TRUE`.
+set -euo pipefail
+
+API_URL="${API_URL:-http://localhost:8000}"
+ADMIN_TOKEN="${ADMIN_TOKEN:-}"
+
+if [ -z "$ADMIN_TOKEN" ]; then
+  read -rsp "Admin Bearer token (user with is_admin=TRUE): " ADMIN_TOKEN
+  echo
+fi
+
+if [ -z "$ADMIN_TOKEN" ]; then
+  echo "ERROR: no admin token provided." >&2
+  exit 1
+fi
+
+cat <<'WARN'
+========================================================================
+  WARNING — IRREVERSIBLE BULK ACTION
+========================================================================
+  This will:
+    * set mfa_enabled = FALSE for every user
+    * delete every backup code
+    * invalidate all pending MFA login challenges and enrollments
+    * write a permanent entry to user_activity_logs
+  After this, all users can sign in with username/password only and must
+  re-enroll MFA from the Users page.
+========================================================================
+WARN
+
+read -rp "Type 'yes' to proceed: " confirm1
+if [ "$confirm1" != "yes" ]; then
+  echo "Aborted."
+  exit 1
+fi
+read -rp "Type 'RESET ALL MFA' (exact) to confirm: " confirm2
+if [ "$confirm2" != "RESET ALL MFA" ]; then
+  echo "Aborted."
+  exit 1
+fi
+read -rp "Reason (logged in audit): " reason
+if [ -z "$reason" ]; then
+  echo "Aborted: reason is required."
+  exit 1
+fi
+
+# Compose JSON safely (python3 for proper JSON escaping; reliably available
+# everywhere the backend already runs).
+payload=$(python3 -c "
+import json, sys
+print(json.dumps({'confirm': 'RESET ALL MFA', 'reason': sys.argv[1]}))
+" "$reason")
+
+echo "Calling $API_URL/api/mfa/admin-reset-all ..."
+http_response=$(curl -fsS -X POST "$API_URL/api/mfa/admin-reset-all" \
+  -H "Authorization: Bearer $ADMIN_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "$payload")
+
+echo "$http_response"
+echo
+echo "Done. Verify in user_activity_logs: action='mfa.disabled.admin_bulk_reset'."

version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.5.0",
+  "version": "1.8.6",
-  "releaseName": "ACME Diagnostics & Proxied Host Wizard",
+  "releaseName": "Opt-in API workers + heartbeat micro-optimization",
-  "releaseDate": "2026-05-08"
+  "releaseDate": "2026-07-06"
 }