fix(acme): remove stray paren breaking ACME completion task (v1.8.5, Issue #35)

The background order-completion task (complete_pending_acme_orders, 60s
cycle) failed on EVERY cycle since v1.8.0 with:

    [ACME-COMPLETE] Error in completion task: syntax error at or near ")"

Root cause: the bounded DNS-01 retry OR-arm added to the atomic order-claim
query in v1.8.0 (13b65d9) carried one extra closing parenthesis, making the
whole SELECT invalid PostgreSQL. The claim is the task's first statement, so
the generic except swallowed it each minute and NO background ACME work ever
ran on v1.8.0-v1.8.4:

- orders were never claimed for finalize -> download -> save (http-01 too);
- advance_dns01_order never ran, so the DNS-01 TXT record was never
  published - DNS-01 with an automated provider (e.g. Cloudflare) could
  never validate (exactly the report in Issue #35);
- wizard-staged orders never left wizard_staged (same try block);
- retry_invalid_dns01 / reconcile_dns01_cleanup never executed;
- hourly-created renewal orders could never complete in the background.

Fix: drop the stray ')' (one line). Query semantics are unchanged.

Why the suite missed it: the unit tests mock asyncpg, so raw SQL never
reaches a real parser. Added a regression test that AST-scans the ACME
modules' SQL string literals (comments/quoted literals stripped) and fails
on unbalanced parentheses - it is red on the pre-fix tree and would have
caught the v1.8.0 regression at commit time. Scanned all six ACME modules:
this query was the only unbalanced SQL.

Verification: full backend suite in docker green (1062 passed, 151
skipped); the fixed query EXPLAINs cleanly on postgres:15; live localtest
run shows zero completion-task errors and a seeded pending order is claimed
("[ACME-COMPLETE] Claimed 1 order(s)"). Backend-only, no schema/API/agent
changes; fully backward compatible.

Reported-by: @tkkost (GitHub Issue #35)

.env.template

@@ -17,6 +17,11 @@ REDIS_URL=redis://redis:6379
 # Change this to a strong random string in production
 SECRET_KEY=your-secret-key-change-this-in-production
 
+# Optional: dedicated Fernet key for encrypting VRRP secrets of HA/VIP (Issue #27).
+# If unset, it is derived from SECRET_KEY (HKDF), exactly like MFA. Set an explicit
+# key (urlsafe-base64, 32 bytes) in production if you want independent key rotation.
+# VIP_ENCRYPTION_KEY=
+
 # ============================================================================
 # PUBLIC URL CONFIGURATION
 # ============================================================================

.github/workflows/docker-build.yml

@@ -7,6 +7,8 @@ on:
 jobs:
   build_and_push:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
 
     steps:
       - name: checkout
@@ -26,6 +28,20 @@ jobs:
           fi
           echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
 
+      # The backend image is built with `context: ./backend`, so the
+      # repo-root version.json is OUTSIDE the build context and never
+      # reaches the container. Backend `main.py` falls back to a
+      # compile-time constant when /app/version.json is missing, which
+      # caused a real production drift: a redeploy of the v1.5.2 tree
+      # silently still reported "v1.5.0" in `/api/version` because the
+      # constant in main.py had been bumped but the file was not
+      # available to read. Stage version.json into the backend
+      # context here so the canonical file IS shipped and the
+      # constant only serves as a defensive fallback. The staged file
+      # is gitignored to keep `git status` clean for developers.
+      - name: stage version.json into backend build context
+        run: cp version.json backend/version.json
+
       - name: set up qemu
         uses: docker/setup-qemu-action@v3
 
@@ -62,3 +78,30 @@ jobs:
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.version.outputs.TAG }}
             taylanbakircioglu/haproxy-openmanager-frontend:${{ steps.prodversion.outputs.VERSION }}
 
+      # Keep the GitHub Releases/Tags in sync with version.json. The docker
+      # images above are tagged with the product version, but nothing here
+      # created the matching git tag, so the repo's Tags/Releases drifted
+      # behind (stuck at the last manually-created tag). After the images are
+      # pushed, cut a Release (which also creates the tag) for the current
+      # version.json, but only if one does not already exist, so re-runs
+      # without a version bump are a no-op.
+      - name: create github release from version.json
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          VERSION="${{ steps.prodversion.outputs.VERSION }}"
+          TAG="v${VERSION}"
+          RELEASE_NAME=$(jq -r '.releaseName // empty' version.json)
+          if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "Release $TAG already exists, skipping."
+          else
+            TITLE="$TAG"
+            [ -n "$RELEASE_NAME" ] && TITLE="$TAG — $RELEASE_NAME"
+            gh release create "$TAG" \
+              --repo "$GITHUB_REPOSITORY" \
+              --target "$GITHUB_SHA" \
+              --title "$TITLE" \
+              --notes "Automated release for $TAG (from version.json)."
+            echo "Created release $TAG"
+          fi
+

.gitignore

@@ -39,8 +39,10 @@ venv.bak/
 *.sqlite
 *.sqlite3
 
-# Docker
-.dockerignore
+# Build-time staged version.json (CI `cp version.json backend/`).
+# The canonical file lives at repo root; this path is a transient
+# copy for the backend Docker build context.
+backend/version.json
 
 # IDE
 .vscode/

README.md

@@ -45,6 +45,7 @@ Modern, web-based management interface for HAProxy load balancers with multi-clu
    - [ACME Auto SSL](#acme-auto-ssl---automated-certificate-management)
    - [WAF Management](#waf-management---web-application-firewall)
    - [IP Inventory](#ip-inventory---cross-cluster-ip-search--discovery)
+   - [HA / VIP Management](#ha--vip-management---keepalived-virtual-ip-failover)
    - [User Management](#user-management---access-control--authentication)
    - [Settings](#settings---system-configuration)
 6. [Getting Started - First-Time Usage](#getting-started---first-time-usage)
@@ -105,12 +106,13 @@ This architecture provides better security (no inbound connections to HAProxy se
 ✅ **Real-Time Monitoring** - Live stats, health checks, and performance dashboards  
 ✅ **SSL Certificate Management** - Centralized SSL with expiration tracking  
 ✅ **ACME Auto SSL (Let's Encrypt)** - Automated certificate issuance, renewal, and deployment via ACME protocol  
+✅ **ACME DNS-01 Challenge** *(v1.8.0)* - TXT-record validation for internal/isolated clusters (no public port 80) and wildcard certificates; pluggable DNS providers (Manual + Cloudflare), opt-in, HTTP-01 unchanged  
 ✅ **ACME Certificate Diagnostic Panel** - Automated preflight that checks agent readiness, DNS resolution, port 80 reachability, and ACME challenge ACL before issuing certificates  
 ✅ **WAF Rules** - Web Application Firewall management and deployment  
 ✅ **Agent Script Versioning** - Update agents via UI (Monaco editor) with auto-upgrade  
 ✅ **Token-Based Agent Auth** - Secure token management with revoke/renew  
 ✅ **IP Inventory** - Cross-cluster IP search to identify agents, VIPs, and backend servers by IP  
-✅ **Keepalived VRRP Detection** - Automatic MASTER/BACKUP state and VIP detection from keepalived  
+✅ **HA / VIP (Keepalived) Management** - Create virtual IPs from the UI; the agent installs & configures Keepalived (unicast VRRP) with a HAProxy health-check so the VIP fails over automatically; live MASTER/BACKUP detection per node  
 ✅ **Role-Based User Management** - Admin and user roles with granular permissions and access control  
 ✅ **User Activity Audit Logs** - Complete audit trail of all system events  
 ✅ **REST API** - Full programmatic access for automation and CI/CD integration
@@ -229,6 +231,7 @@ This architecture provides better security (no inbound connections to HAProxy se
 - **Agent Token Management**: Secure token-based agent installation with token revoke and renew capabilities
 - **Agent Script Editor**: Monaco code editor for updating agent scripts via UI (not binary) - create new versions, rollback, and auto-upgrade all agents
 - **Platform-Specific Scripts**: Auto-generated installation scripts for Linux and macOS (x86_64/ARM64)
+- **HA / VIP (Keepalived) Management**: Create a virtual IP from the UI and the agents install & configure Keepalived (VRRP) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops — opt-in, with live MASTER/BACKUP per node, multi-distro install, cluster-driven config path, and Apply/Reject staging
 
 #### Version Control & Change Management
 - **Apply Management**: Centralized change tracking and deployment status monitoring across all agents
@@ -252,10 +255,11 @@ This architecture provides better security (no inbound connections to HAProxy se
 - **Stuck Order Detection** *(v1.4.0)*: Setup wizard surfaces orders that the CA has validated but not yet downloaded, with one-click `Complete` action and automatic 60-second retry
 - **Multi-Provider Support**: Configurable ACME directory URL supports Let's Encrypt, ZeroSSL, Google Trust Services, Buypass, and custom CAs
 - **HTTP-01 Challenge**: Built-in challenge responder with automatic HAProxy routing injection; reserved backend name `_acme_challenge_backend` is auto-managed and protected from manual edits / agent sync collisions
+- **DNS-01 Challenge** *(v1.8.0 — Issue #35)*: Validate via a DNS TXT record instead of HTTP on port 80, for **internal/isolated clusters with no public ingress** and for **wildcard** certificates (`*.example.com`). Pluggable per-account DNS provider (Manual + Cloudflare to start; credentials encrypted at rest and verified on save), same PENDING → APPLIED pipeline, bounded automatic retry on propagation lag, and a DNS-01 event timeline. Opt-in via a global setting; HTTP-01 behaviour is unchanged. (See the *DNS-01 Challenge* subsection under ACME Auto SSL below.)
 - **ACME Account Management**: Register, view, and deactivate ACME accounts from the UI
 - **Staging Mode**: Test certificate issuance with Let's Encrypt staging environment before production
 - **Custom Staging Endpoint** *(v1.4.0)*: Optional `staging_url_override` setting lets you point staging mode at a private ACME test CA (e.g. Pebble) without touching the production directory URL
-- **External Account Binding (EAB)**: Support for CAs that require EAB (ZeroSSL, Google Trust Services)
+- **External Account Binding (EAB)**: Support for CAs that require EAB (ZeroSSL, Google Trust Services). Enter the EAB Key ID and HMAC Key globally in Settings, or per-account in the Register Account dialog (a per-account value overrides the global setting; leave it blank to use the global one)
 - **Structured Error Diagnostics** *(v1.4.0)*: All ACME failures (challenge, finalize, download) persist structured JSON to `letsencrypt_orders.error_detail` for clear post-mortem analysis
 - **Audit Logging** *(v1.4.0)*: Every ACME operation (request, revoke, CA-chain import, account ops) is captured in `user_activity_logs` for compliance review
 - **ACME Diagnostic Panel** *(v1.5.0 — Issue #13)*: Live pre-flight + post-failure diagnostics (DNS / port-80 / routing / account / agents) and merged event timeline (`acme_order_events` + correlated `user_activity_logs`) accessible from the ACME Automation page; humanized error rendering for 11+ RFC8555 problem types with backwards-compatible fallback for legacy plain-string `error_detail`; per-user 5/min rate-limit
@@ -964,6 +968,24 @@ Understanding how HTTP-01 challenges work in a distributed HAProxy environment i
 | Behind NAT/VIP | VIP: 1.2.3.4:80 | Internal:5000 | VIP address |
 | Multi-cluster | Multiple HAProxy nodes | Central OpenManager | Each domain → respective HAProxy |
 
+#### DNS-01 Challenge — Internal/Isolated Clusters & Wildcards *(v1.8.0 — Issue #35)*
+
+The default **HTTP-01** challenge validates over **port 80**, so the domain must resolve publicly to an HAProxy node with ACME Challenge Routing enabled. **DNS-01** validates via a **DNS TXT record** (`_acme-challenge.<domain>`) instead, so it needs **no inbound port 80 and no public ingress** to the HAProxy node. Use it for:
+
+- **Internal / isolated clusters** (behind a VPN/firewall, no public port 80) where you still control the domain's DNS.
+- **Wildcard certificates** (`*.example.com`) — which can *only* be issued via DNS-01.
+
+DNS-01 is **opt-in** and fully backward compatible: it is disabled until an administrator enables it, and existing HTTP-01 certificates are completely unaffected.
+
+- **Enable it**: Settings → ACME / SSL Automation → **DNS-01 Challenge (advanced)** → turn on *Enable DNS-01 Challenge* and Save. While off, DNS-01 options are hidden and no DNS-01 orders can be created.
+- **Per-account provider**: in ACME Automation, create (or reconfigure) an ACME account with **Challenge Method = DNS-01** and a **DNS Provider**. Provider credentials are **verified before saving** and **encrypted at rest** (Fernet, mirroring the VRRP/MFA secret pattern); they are never returned by the API or written to logs.
+- **Supported providers**: **Manual** (publish the TXT record yourself in any DNS — including fully internal DNS — then click *Verify*; works everywhere but cannot auto-renew unattended) and **Cloudflare** (API token with `Zone:DNS:Edit` + `Zone:Read`; the TXT record is created and cleaned up automatically and renews unattended). The provider interface is pluggable — more providers can be added without changing the issuance flow.
+- **Same pipeline**: after validation the certificate follows the normal PENDING → APPLIED flow (assign to clusters / Apply Management) and the agent serves it — identical to HTTP-01 from finalize onward, with **zero agent or rendered-config changes** for DNS-01.
+- **Manual flow**: the order detail shows the exact `_acme-challenge.<domain>` record name + TXT value (copyable); publish it and click *I've added the records — Verify*. For Cloudflare it is automatic.
+- **Resilience**: a propagation-lag failure is recovered by a **bounded fresh-order retry chain** (1 original + 3 retries with increasing backoff, kept under Let's Encrypt's rate limits); any orphaned TXT record is cleaned up by a reconcile sweep. The order detail shows a DNS-01 event timeline (publish → validation → cleanup).
+- **Wildcards**: `*.example.com` is validated at `_acme-challenge.example.com`; it does **not** cover the apex — add `example.com` as a separate name if you need both (the providers handle the two coexisting TXT values automatically).
+- **Scope (this release)**: the Site Wizard remains HTTP-01-only; issue DNS-01 / wildcard certificates from **ACME Automation**.
+
 #### ACME Quick Start Guide
 
 Follow these steps to obtain your first Let's Encrypt certificate:
@@ -1055,6 +1077,32 @@ The IP Inventory page provides a unified view of all IP addresses across every c
 - Locate a backend server IP across multiple clusters
 - Audit all IP addresses managed by the platform
 
+### HA / VIP Management - Keepalived Virtual IP Failover
+
+The HA / VIP page manages **highly-available virtual IPs** backed by Keepalived (VRRP) directly from the UI — no SSHing into nodes to install/configure Keepalived by hand. It builds on the agent pull-architecture: you define the VIP centrally, click Apply, and the agents converge.
+
+**What it does:**
+- **VIP dashboard**: lists each virtual IP with its pool, VRID, per-node interface, and a **live MASTER / BACKUP / FAULT** column per node (refreshed every 30s from the existing keepalive-state heartbeat pipeline).
+- **Creation form**: enter the virtual IP (+ prefix), pick the pool, and select which of the pool's HAProxy nodes participate — each with a network interface (from the node's reported interfaces), a VRRP role (MASTER/BACKUP) and a priority. VRID auto-allocates per pool; an optional VRRP secret can be set.
+- **Hands-off automation**: on **Apply**, each member's `keepalived.conf` is rendered and delivered; the agent installs Keepalived if missing, writes the config + a HAProxy health-check (`vrrp_script` + `track_script`), validates with `keepalived -t`, and starts the service. When HAProxy drops on the active node, the health-check lowers VRRP priority and the **VIP fails over automatically** to a backup.
+
+**How it works (pull-based, isolated):**
+- New tables `vip_instances` + `vip_members`; the agent polls `GET /api/agents/{name}/keepalived-config` (key-authenticated, agent-bound) and converges. It is fully **isolated** from the global HAProxy apply flow — VIP changes never regenerate `haproxy.cfg`.
+- **Cluster-driven path**: the `keepalived.conf` location is a cluster setting (`keepalived_config_path`, default `/etc/keepalived/keepalived.conf`) delivered to the agent dynamically — just like the HAProxy paths — so non-standard installs are supported with zero per-node effort. The default is what the keepalived service loads on every distro; if you set a non-default path, ensure the keepalived unit is configured to load it.
+
+**Apply / Reject staging:**
+- VIP edits are staged as **PENDING** and only go live on **Apply**, which records an applied snapshot. **Reject** discards pending changes and **restores the previous applied state** (a never-applied VIP is discarded). Editing a live VIP never disrupts it until you re-Apply.
+
+**Enterprise safety:**
+- **Opt-in & backward compatible** — nodes/clusters without a VIP do nothing new.
+- **Never clobbers a hand-managed Keepalived** — if an unmanaged `keepalived.conf` exists, the agent reports "externally managed" and leaves it untouched.
+- **Multi-distro install** — Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine (apt/dnf/yum/zypper/apk).
+- **Reliable detection across platforms** — MASTER/BACKUP is detected via systemd journal, syslog, and a portable interface-based check (exact-match, all VIPs).
+- **Secure** — the VRRP secret is encrypted at rest and never returned by the API; the delivery endpoint requires the node's own API key.
+- **Scope** — VRRP targets bare-metal / VMware / on-prem L2 networks; the UI clearly notes that AWS/Azure/GCP don't honor VRRP/gratuitous-ARP. macOS agents can't run Keepalived and are excluded (the UI warns if one is selected).
+
+**RBAC:** gated by the `vip` permission group (`vip.read/create/update/delete/apply`); admins bypass.
+
 ### User Management - Access Control & Authentication
 - **User Accounts**: Create, edit, and manage user accounts
 - **Role-based Access**: Admin, user, and custom role definitions
@@ -1065,6 +1113,115 @@ The IP Inventory page provides a unified view of all IP addresses across every c
 - **API Keys**: User API key generation and management
 - **Role Assignment**: Dynamic role assignment and permission updates
 
+#### Multi-Factor Authentication (MFA) — v1.6.0 (Issue #18)
+
+MFA is **optional per account** and **default OFF**. Existing users keep their
+single-factor (username/password) login unless they choose to enable it. The
+feature is fully additive: nothing changes for accounts that don't opt in.
+
+**For end-users**
+
+- Open **Users** → find your own row → click **Enable MFA**.
+- A wizard opens with three steps:
+  1. **Set up** — scan the QR code with Google Authenticator / Authy /
+     1Password / Microsoft Authenticator, or paste the displayed secret manually.
+  2. **Verify** — enter the current 6-digit code from your app.
+  3. **Backup codes** — save the 10 single-use recovery codes (format
+     `XXXX-YYYY`). They are shown only once. Use the **Copy all** /
+     **Download .txt** buttons.
+- After enrollment your sign-in becomes two-step: username/password →
+  6-digit TOTP (or a backup code).
+- To turn MFA off again, open your row's **Disable MFA** action and enter a
+  current TOTP or backup code.
+
+**For admins**
+
+- On any user with MFA enabled, the **Reset MFA** action wipes the user's
+  TOTP secret, backup codes, and pending challenges. A required *reason* is
+  written to the audit log. After reset the user logs in with their password
+  and may re-enroll.
+
+**Emergency: reset MFA for every user**
+
+Use the CLI helper when an authenticator outage / mass key loss happens.
+Double confirmation is required; the action is irreversible.
+
+```bash
+API_URL=https://hap.example.com ADMIN_TOKEN=eyJ... \
+  ./scripts/admin-mfa-reset-all.sh
+# Prompts ask for: 'yes' → 'RESET ALL MFA' → reason
+# Audit log:      action='mfa.disabled.admin_bulk_reset'
+```
+
+**Configuration**
+
+- Backend env var `MFA_ENCRYPTION_KEY` — a 44-char Fernet key used to encrypt
+  TOTP secrets at rest. Generate with:
+  ```bash
+  python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
+  ```
+  On Kubernetes the key lives in the `backend-secret` Secret
+  (`k8s/manifests/03-secrets.yaml`). The shipped manifest uses the placeholder
+  `mfa_encryption_key_replace_me`; replace it via your CI/CD pipeline (e.g.
+  `sed` step) before `kubectl apply`.
+- Optional `MFA_ACCOUNT_LABEL_DOMAIN` — overrides the per-user otpauth label
+  domain so QR codes show e.g. `alice@hap.example.com` instead of the request
+  hostname.
+
+**API endpoints (all additive)**
+
+| Method | Path | Notes |
+|---|---|---|
+| `POST` | `/api/auth/login` | Returns `mfa_required:true`+`mfa_token` for MFA users; legacy shape otherwise |
+| `POST` | `/api/auth/login/mfa-verify` | Submits TOTP or backup code; returns JWT |
+| `GET`  | `/api/mfa/status` | Self status (enabled / method / backup codes remaining) |
+| `POST` | `/api/mfa/enroll/start` | Begin TOTP enrollment |
+| `POST` | `/api/mfa/enroll/confirm` | Confirm enrollment, returns 10 backup codes once |
+| `POST` | `/api/mfa/disable` | Self-disable (TOTP or backup required) |
+| `POST` | `/api/mfa/backup-codes/regenerate` | Issue 10 fresh backup codes (TOTP only) |
+| `GET`  | `/api/mfa/admin/status/{id}` | Admin: any user's MFA status |
+| `POST` | `/api/mfa/admin-reset/{id}` | Admin: reset a single user's MFA |
+| `POST` | `/api/mfa/admin-reset-all` | Admin: emergency reset for all users |
+
+**Rate limits (per-user, ingress-aware, operationally tunable)**
+
+MFA endpoints are rate-limited via slowapi+Redis with a **user-aware key
+function** (`backend/middleware/mfa_rate_limit_key.py`):
+
+1. If the request carries a valid Bearer JWT → bucket is `user:<id>`.
+   Each operator gets an isolated bucket; an org-wide MFA rollout is no
+   longer bottlenecked by the shared ingress IP.
+2. Else, if the TCP peer is in `MFA_TRUSTED_PROXY_CIDRS` → bucket is the
+   first `X-Forwarded-For` hop (real client IP behind the ingress).
+3. Else → bucket is the TCP peer (slowapi default).
+
+Defaults live in `backend/middleware/mfa_rate_limits.py` and are sized for
+**enterprise-scale** rollouts (thousands of operators). Each one is
+overridable via env var; an invalid string logs a `WARNING` and falls back
+to the default without crashing.
+
+| Endpoint | Env var | Default | Bucket |
+|---|---|---|---|
+| `POST /api/mfa/enroll/start` | `MFA_RATE_LIMIT_ENROLL_START` | `10/minute` | per user |
+| `POST /api/mfa/enroll/confirm` | `MFA_RATE_LIMIT_ENROLL_CONFIRM` | `10/minute` | per user |
+| `POST /api/mfa/disable` | `MFA_RATE_LIMIT_DISABLE` | `10/minute` | per user |
+| `POST /api/mfa/backup-codes/regenerate` | `MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES` | `5/hour` | per user |
+| `POST /api/mfa/admin-reset/{id}` | `MFA_RATE_LIMIT_ADMIN_RESET` | `60/hour` | per admin |
+| `POST /api/mfa/admin-reset-all` | `MFA_RATE_LIMIT_ADMIN_RESET_ALL` | `1/day` | per admin |
+
+Limit string format follows slowapi: `<count>/<second|minute|hour|day>`.
+
+**Trusted-proxy configuration**
+`MFA_TRUSTED_PROXY_CIDRS` — comma-separated CIDR list, e.g.
+`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. Empty (default) disables XFF
+parsing — XFF from any peer is then ignored, which is the safe choice when
+the topology is unknown. Set this when your backend sits behind a known
+ingress / load balancer so anonymous flows still get per-real-IP buckets.
+
+The pre-existing `/api/auth/login` rate-limiting policy is unchanged. On
+Kubernetes, see commented overrides in
+`k8s/manifests/07-configmaps.yaml::backend-config`.
+
 ### Settings - System Configuration
 - **Theme Settings**: Light/dark mode toggle and UI customization
 - **ACME / SSL Automation**: Configure ACME provider, directory URL, staging mode, auto-renewal, EAB credentials, and test CA connectivity
@@ -1524,12 +1681,25 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 ```
 
 #### Frontend Configuration
-```bash
-# API Endpoint (auto-detected if empty)
-# Leave empty in production to use same-origin (window.location)
-REACT_APP_API_URL=""  # For development: "http://localhost:8000"
 
-# Environment
+The frontend is a Create-React-App single-page app served as a static bundle
+(`serve -s build`). It uses **same-origin** (`window.location.host`) for all
+`/api/*` calls — no env vars are needed in production. Routing is handled
+entirely by the nginx reverse proxy in front of the frontend pod (Kubernetes
+ingress + `nginx-config` ConfigMap, or `nginx/nginx.conf` for Docker Compose).
+
+> ⚠️ **Do NOT set `REACT_APP_API_URL` in your CI/CD pipeline.** CRA inlines
+> `REACT_APP_*` values into the bundle at **build time**, so any value baked
+> in there overrides the runtime same-origin detection and breaks every
+> deployment whose URL does not match the inlined string. Leave the variable
+> unset; the bundle will resolve to whatever host the user is browsing.
+
+```bash
+# Optional, only when you intentionally need a cross-origin API
+# (then CORS_ORIGINS on the backend must include the SPA's origin):
+# REACT_APP_API_URL="https://api.example.com"
+
+# Build settings
 NODE_ENV="production"
 GENERATE_SOURCEMAP="false"
 ```
@@ -2245,6 +2415,31 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.8.5** (2026-07-03) — **ACME completion-task SQL fix** (Issue #35 follow-up): the background order-completion task (`complete_pending_acme_orders`, runs every 60s) died on **every cycle** with `syntax error at or near ")"` — an extra closing parenthesis introduced in v1.8.0's bounded DNS-01 retry claim query. Because that query is the task's first database call, **no background ACME work ran at all from v1.8.0 through v1.8.4**: orders were never claimed for finalize/download, the DNS-01 TXT record was never published (so DNS-01 with an automated provider such as Cloudflare could never validate), Site Wizard staged orders never left `wizard_staged`, and DNS-01 retry/TXT-cleanup never executed. The stray parenthesis is removed and a regression test now scans all ACME modules' SQL for unbalanced parentheses (the unit suite mocks the database, which is why a raw-SQL syntax error could slip through). One-line backend query fix; no schema, API, or agent changes — fully backward compatible.
+- **v1.8.4** (2026-06-27) — **Agent installer self-kill fix** (Issue #31): the Linux/macOS agent installer could abort during "pre-installation cleanup" (terminal showed `Killing processes matching: haproxy-agent` then `Killed`) when the install script's own filename contained "haproxy-agent". The cleanup killed processes by matching the bare string "haproxy-agent" against full command lines, which also matched the running installer (and a `sudo`/PAM ancestor the self-exclusion did not cover), so the installer terminated itself. Cleanup now targets only the installed agent (the `$INSTALL_DIR/haproxy-agent` binary and the agent service), never the bare string, and the UI now names the downloaded scripts `install-agent-<platform>.sh` / `uninstall-agent-<platform>.sh`. Installer-only change; the running agent and its privilege model (it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade) are unchanged.
+- **v1.8.3** (2026-06-25) — **Agent heartbeat JSON fix** (Issue #31): a self-hosted agent could fail every heartbeat with `HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host, leaving a stray comma in the hand-built heartbeat JSON. The agent script now substitutes a valid placeholder when that block is empty so it can no longer emit a stray comma, and the backend heartbeat endpoint now parses valid payloads as-is and, only when a body fails to parse, tolerates that specific malformed pattern (a leading or doubled comma) so an already-deployed agent recovers on its next heartbeat after this build is deployed. Backend + agent-script only; healthy agents of every version are byte-for-byte unaffected.
+- **v1.8.2** (2026-06-25) — **ACME nonce fix** (Issue #35 follow-up): the ACME client now scopes the anti-replay nonce **per certificate authority** so a nonce issued by one CA is never sent to another. This fixes ZeroSSL/Google account registration failing with `malformed: The Replay Nonce could not be base64url-decoded` (the client previously shared one nonce across CAs and only auto-retried on `badNonce`). Account registration now always uses a fresh nonce from the target CA, and the retry covers this case too. Backend-only; HTTP-01 and Let's Encrypt are unaffected.
+- **v1.8.1** (2026-06-24) — **ACME DNS-01 fixes** (Issue #35 follow-up): Cloudflare API tokens are now sanitized so a pasted token with quotes/spaces no longer fails with "Invalid request headers"; ZeroSSL/Google **External Account Binding (EAB)** can be entered per-account in the register dialog and EAB-required failures show a clear message; and **Apply Management** now categorizes cluster ACME enable/disable changes under their own "ACME Challenge Routing" section and **Apply/Reject All** correctly process them (previously "Rejected 0 HA/VIP change(s)"), consistent with every other entity. Fully backward compatible.
+- **v1.8.0** (2026-06-23) — **ACME DNS-01 challenge support** (Issue #35): Auto SSL can now validate via a **DNS TXT record** (`_acme-challenge.<domain>`) instead of HTTP-01 on port 80, enabling certificates for **internal/isolated clusters with no public ingress** and **wildcard** certificates (`*.example.com`). Pluggable **per-account DNS provider** (Manual + Cloudflare to start; credentials verified on save and **encrypted at rest**, never returned by the API or logged), the same **PENDING → APPLIED** pipeline, a **bounded automatic retry** on propagation lag, and a **DNS-01 event timeline** in the order detail. **Opt-in** via Settings → ACME (global switch, default off); **HTTP-01 is byte-for-byte unchanged**, with **zero agent or rendered-config changes**. Manual DNS-01 certificates cannot auto-renew unattended; the UI states this and disables auto-renew for them.
+- **v1.7.8** (2026-06-07) — HA / VIP apply progress now shows **per-node** convergence: a multi-node VIP's apply popup reads "Syncing HA/VIP… 1/2 node(s) converged" (matching the HA/VIP table) instead of a coarse per-change count. Frontend-only.
+- **v1.7.7** (2026-06-07) — HA / VIP apply-progress consistency: applying a VIP change (or approving a delete) used to flash the progress popup green instantly while the HA/VIP page still showed `SYNCING (0/1)` for a couple of minutes. The popup now **keeps showing "Syncing HA/VIP… X/Y node(s) converged"** until each member node reports the VIP `ACTIVE` (create/edit) or fully torn down (delete) — exactly like the HAProxy agent-sync widget — then completes green. It's a fire-and-forget background poll (the Apply button is released immediately), bounded at ~5 min so an offline node can't spin forever (then it completes with an informational "still converging — track on the HA/VIP page"). Frontend-only; no backend/agent/schema change.
+- **v1.7.6** (2026-06-07) — HA / VIP UX + accuracy polish: (1) the on-prem/L2 cloud caveat is now a **subtle, collapsed-by-default "Network requirements" info link** instead of a prominent yellow warning. (2) The delete dialog is simplified — deletion is **always a graceful teardown** (stop & disable keepalived, remove our config, release the VIP, keep the package); the confusing "also uninstall the package" checkbox was removed (it was a no-op on any node whose keepalived predates the install marker, and package removal is better handled as a deliberate node-decommission step — the `purge_package` API remains for that). (3) The agent now reports keepalived **FAULT** state (e.g. when the chosen interface has no usable IPv4) instead of misreporting it as BACKUP, so a misconfigured VIP shows red/FAULT in the UI. (1)+(2) are frontend-only; (3) is an additive agent-script change — push it via **Agent Script Management → Reset to Defaults**, then **Upgrade**.
+- **v1.7.5** (2026-06-07) — HA / VIP "View Change" fix: editing a VIP (e.g. priority + virtual IP) now shows a **real line diff — only the lines that actually changed** — in Apply Management's View Change, instead of rendering the whole `keepalived.conf` as "added". Also fixes a doubled `+ +` prefix (the VIP diff now stores lines without a +/- prefix, matching the standard config diff). View-only; no schema, agent, apply, or render change.
+- **v1.7.4** (2026-06-06) — HAProxy config-generator robustness fix: a frontend that uses a stick counter (`track-sc<N>` or an `sc_*_rate(...)` fetch, e.g. a rate-limit `http-request deny if { sc_http_req_rate(0) gt N }`) but declares **no `stick-table`** caused HAProxy to fatally reject the whole cluster config with *"table '&lt;frontend&gt;' used but not configured"*. This happened where rate-limit directives had been baked into a frontend's stored `request_headers`/`options` (by an older version or a config import). The generator now **auto-injects a default `stick-table`** in that case. Purely additive — it only fires when a counter is used and no table exists (a config that was already invalid), so frontends that already declare a stick-table or don't rate-limit are byte-unchanged.
+- **v1.7.3** (2026-06-06) — HA / VIP backward-compat fix: the v1.7.2 deletion-tracking list now only resurfaces VIPs deleted through the **new approval flow** (gated on `last_config_status='APPLIED'`), so a VIP soft-deleted under an earlier version's immediate-delete is no longer shown as `DELETING`. Display-only; no schema or agent change.
+- **v1.7.2** (2026-06-06) — HA / VIP safety & visibility follow-up:
+  - **Approval-gated deletion (safety).** Deleting a *running* VIP from the UI no longer takes effect immediately — it is **staged for Apply Management** and the VIP **keeps running, untouched**, until you **Approve** it (Reject keeps it). The agent is told to tear keepalived down **only after approval**, so a misclick can never tear down a production VIP — an agent never deletes without an explicit human approval. The deletion is trackable through the standard Apply popup and a **DELETING** status on the HA/VIP tab. (A VIP that was never applied is removed at once — nothing is running to tear down.)
+  - **Diagnostics view** — a per-VIP search-icon modal showing each node's keepalived deploy state, the message it reported, last-ack time and live VRRP state (handy while a fresh install is SYNCING, ~30s), plus the exact node-side log commands.
+  - **Opt-in package uninstall** — the safe default keeps the keepalived package (stop & disable, remove our config, release the VIP); a default-off checkbox additionally uninstalls the package **only on nodes where OpenManager installed it** (an admin's pre-existing keepalived is never removed, tracked via an install marker).
+  - Adds additive `purge_on_teardown` + `pending_delete` columns; `SCHEMA_VERSION` 5 → 7 (idempotent; existing data/passwords unaffected). Agent script updated — push it via **Agent Script Management → Reset to Defaults**, then **Upgrade** agents.
+- **v1.7.1** (2026-06-06) — HA / VIP follow-up: a VIP can now be created on a **single node** — a Keepalived-managed floating IP **without** failover (e.g. a one-box HAProxy that wants a stable address, or before a second node is added). Add a second node anytime for real VRRP failover. A single-node VIP renders a clean **multicast** config (no bare `unicast_src_ip`, which `keepalived -t` rejects); multi-node behaviour is unchanged. The Create-VIP form now auto-selects the first chosen node as **MASTER** and shows an in-UI notice that, on Apply, Keepalived is **installed automatically from the node's OS package repositories** (apt/dnf/yum/zypper/apk — the node must reach its repos / an internal mirror), while a hand-managed Keepalived is still left untouched. No schema change.
+- **v1.7.0** (2026-06-05) — Feature (Issue #27): **HA / VIP (Keepalived) management** from the UI. A new "HA / VIP" tab lets you create a virtual IP, pick a per-node interface, and select which pool nodes participate (with MASTER/BACKUP roles + priorities); on Apply, the agent installs & configures Keepalived (unicast VRRP, cloud-safe default) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops, and the tab shows live MASTER/BACKUP per node. Fully **opt-in and backward compatible** — nodes/clusters without a VIP are untouched, and a node already running a hand-managed Keepalived is detected and never overwritten (reported as "externally managed"). Pending VIP changes can be **Rejected** to fully restore the last applied state. Keepalived is installed across the major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine), and live MASTER/BACKUP detection works across distros/init systems (journald + log files + portable interface-based detection). On-prem/L2 scope (VRRP); a clear in-UI notice covers the cloud caveat. Adds two new tables (`vip_instances`, `vip_members`) — `SCHEMA_VERSION` bumps to 3 (idempotent re-run; existing data and passwords unaffected). A new "HA / VIP" tab lets you create a virtual IP, pick a per-node interface, and select which pool nodes participate (with MASTER/BACKUP roles + priorities); on Apply, the agent installs & configures Keepalived (unicast VRRP, cloud-safe default) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops, and the tab shows live MASTER/BACKUP per node. Fully **opt-in and backward compatible** — nodes/clusters without a VIP are untouched, and a node already running a hand-managed Keepalived is detected and never overwritten (reported as "externally managed"). Pending VIP changes can be **Rejected** to fully restore the last applied state. Keepalived is installed across the major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine), and live MASTER/BACKUP detection works across distros/init systems (journald + log files + portable interface-based detection). On-prem/L2 scope (VRRP); a clear in-UI notice covers the cloud caveat. Adds two new tables (`vip_instances`, `vip_members`) — `SCHEMA_VERSION` bumps to 3 (idempotent re-run; existing data and passwords unaffected).
+- **v1.6.5** (2026-06-02) — Security: re-pinned the bundled nginx reverse-proxy image to `nginx:1.31.1-alpine` (mainline patched release) for the nginx "poolslip" advisory (fixed in mainline 1.31.1+ / stable 1.30.2+). Supersedes the v1.6.4 stable pin. No config, schema, or behavior changes.
+- **v1.6.4** (2026-06-02) — Security: pinned the bundled nginx reverse-proxy image to a patched stable release (`nginx:1.30.2-alpine`) for the nginx "poolslip" advisory (mainline ≤ 1.31.0 affected; fixed in stable 1.30.2+). The product's nginx config uses no `rewrite` capture groups, so the config-level mitigation did not apply — the fix is the version pin. No config, schema, or behavior changes.
+- **v1.6.3** (2026-06-01) — Bugfix: a backend **server** toggled OFF (`is_active=false`) disappeared from the UI with no way to reactivate it. `GET /api/backends` now honors `include_inactive` for servers (previously only backends), so disabled servers stay visible with an OFF switch + "Inactive" tag and can be re-enabled; soft-deleted (pending-delete) servers stay hidden. A Reject of a server toggle now correctly rolls back `is_active` (entity snapshot). Startup migrations are hardened for multiple replicas / rolling deploys (advisory lock + schema-version gate, so an already-current schema isn't re-migrated under a serving peer's load). Version is reported consistently across all layers. No schema changes; config generation unchanged (disabled servers stay commented out).
+- **v1.6.2** (2026-05-30) — Bugfixes: (1) agents (which authenticate with their `X-API-Key` token) could not reach `GET /api/clusters` / `/api/clusters/{id}` after the v1.5.x cluster-read hardening, breaking agent assignment ("401: Authorization header missing"); these endpoints now accept either a user JWT or an agent token (anonymous access is still rejected). (2) The uninstall-script generator returned 400 for macOS agents (which report platform `darwin`); it now normalizes the platform the same way the install generator does. No UI or schema changes.
+- **v1.6.1** (2026-05-21) — Security patch: bump `axios` to 1.16.x (prototype-pollution hardening, header-injection fix, keep-alive memory leak fix) and `fast-uri` to 3.1.2 (GHSA-v39h-62p7-jpjc). No functional changes.
+
 For full release notes and the list of features delivered in each version (v1.5.x Site Wizard + ACME Diagnostic Panel, v1.4.0 ACME stability + enterprise audit, v1.3.0, ...) see the [GitHub Releases](https://github.com/taylanbakircioglu/haproxy-openmanager/releases) page.
 
 ---

UPGRADE_GUIDE.md

@@ -1,3 +1,31 @@
+# Upgrade Notes — v1.7.0 (HA / VIP Keepalived management, Issue #27)
+
+**Backward compatible & opt-in.** Upgrading to v1.7.0 changes nothing for existing
+clusters/agents until you create a VIP:
+
+- **Schema:** `SCHEMA_VERSION` bumps to `3`, so on first start the (idempotent)
+  migration sequence re-runs once and adds two **new** tables (`vip_instances`,
+  `vip_members`) plus an additive `vip_instances.applied_snapshot` column (enables
+  rejecting a pending VIP change and restoring the previous applied state). No existing
+  table is altered. Existing rows and the **admin password
+  are not reset** (default users are create-if-missing). The only data effect is that
+  the **four built-in system roles** (`super_admin`/`operator`/`security_admin`/`viewer`)
+  are re-seeded to their canonical permission sets **plus** the new `vip.*` permissions —
+  this is the long-standing behavior of the role seeder; **custom roles are untouched**.
+- **Agents:** the agent script gains an opt-in keepalived deploy that is a **no-op** on
+  any node without an applied VIP, and it **never overwrites a hand-managed
+  `/etc/keepalived/keepalived.conf`** (it reports "externally managed" instead).
+- **Scope:** VRRP VIPs target bare-metal / VMware / on-prem L2 networks. On AWS/Azure/GCP
+  the cloud fabric doesn't honor VRRP/gratuitous-ARP; the UI surfaces this. Ensure host
+  firewalls permit VRRP (IP protocol 112).
+- **Optional env:** `VIP_ENCRYPTION_KEY` (see `.env.template`) — if unset, the VRRP secret
+  encryption key is derived from `SECRET_KEY` (like MFA).
+
+No rollback steps are required to *disable* the feature: simply don't create VIPs (or
+delete them — agents tear down their managed keepalived on the next poll).
+
+---
+
 # Agent Upgrade Guide - Dashboard Stats Fix
 
 ## Problem

backend/database/migrations.py

@@ -50,9 +50,62 @@ async def ensure_agents_table():
         await conn.execute("ALTER TYPE config_status ADD VALUE IF NOT EXISTS 'DELETION';")
         logger.info("Ensured REJECTED and DELETION values exist in config_status enum.")
 
-        # First, create essential tables if they don't exist
-        await create_essential_tables(conn)
-        
+        # First, create essential tables if they don't exist.
+        #
+        # Rolling-restart resilience: create_essential_tables() runs idempotent
+        # CREATE ... IF NOT EXISTS statements on every startup. Its CREATE INDEX
+        # block needs a SHARE lock that conflicts with concurrent writes (e.g.
+        # agent heartbeats updating backend_servers/agents). During a redeploy a
+        # writer can hold that lock, so the DDL blocked for the full 60s
+        # command_timeout -> TimeoutError -> startup crash -> crash-loop.
+        #
+        # Fix: fail fast on locks (lock_timeout), retry briefly, and on
+        # persistent contention SKIP the idempotent bootstrap and continue — on
+        # an established DB the objects already exist; a fresh DB has no writers
+        # so the first attempt always succeeds. lock_timeout is scoped to this
+        # call and RESET afterwards, so every other migration below keeps its
+        # original (wait-indefinitely) behavior. Non-lock errors still propagate
+        # (genuine schema problems must NOT be masked).
+        import asyncio as _asyncio
+        _lock_excs = (_asyncio.TimeoutError,)
+        try:
+            import asyncpg as _asyncpg
+            _lock_excs = _lock_excs + (
+                _asyncpg.exceptions.LockNotAvailableError,
+                _asyncpg.exceptions.QueryCanceledError,
+            )
+        except Exception:
+            pass
+        try:
+            await conn.execute("SET lock_timeout = '10s'")
+        except Exception:
+            pass
+        try:
+            for _attempt in range(1, 4):
+                try:
+                    await create_essential_tables(conn)
+                    break
+                except _lock_excs as _lock_err:
+                    if _attempt < 3:
+                        logger.warning(
+                            f"create_essential_tables: lock contention "
+                            f"(attempt {_attempt}/3), retrying in 3s "
+                            f"({type(_lock_err).__name__})"
+                        )
+                        await _asyncio.sleep(3)
+                    else:
+                        logger.warning(
+                            "create_essential_tables: persistent lock contention; "
+                            "skipping idempotent schema bootstrap and continuing "
+                            "startup (objects already exist on an established DB) "
+                            f"({type(_lock_err).__name__})"
+                        )
+        finally:
+            try:
+                await conn.execute("RESET lock_timeout")
+            except Exception:
+                pass
+
         # Ensure status column exists in config_versions table
         status_column_exists = await conn.fetchval("""
             SELECT 1 FROM information_schema.columns 
@@ -431,6 +484,7 @@ async def ensure_agents_table():
                 connection_type VARCHAR(50) DEFAULT 'agent',
                 stats_socket_path VARCHAR(500) DEFAULT '/run/haproxy/admin.sock',
                 haproxy_config_path VARCHAR(500) DEFAULT '/etc/haproxy/haproxy.cfg',
+                keepalived_config_path VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf',
                 pool_id INTEGER REFERENCES haproxy_cluster_pools(id) ON DELETE SET NULL,
                 haproxy_user VARCHAR(255) DEFAULT 'haproxy',
                 haproxy_group VARCHAR(255) DEFAULT 'haproxy',
@@ -1298,6 +1352,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk', 'apply.emergency',
                     'agents.read', 'agents.create', 'agents.update', 'agents.delete', 'agents.script', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.create', 'clusters.update', 'clusters.delete', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.upload', 'config.backup', 'config.restore', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'users.read', 'users.create', 'users.update', 'users.delete', 'users.password', 'users.roles',
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
@@ -1319,6 +1374,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history', 'apply.bulk',
                     'agents.read', 'agents.update', 'agents.toggle', 'agents.upgrade', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch', 'clusters.config',
+                    'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read'
@@ -1336,6 +1392,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.execute', 'apply.reject', 'apply.history',
                     'agents.read', 'agents.version', 'agents.logs',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
@@ -1354,6 +1411,7 @@ async def update_system_roles_to_enterprise_rbac():
                     'apply.read', 'apply.history',
                     'agents.read',
                     'clusters.read', 'clusters.switch',
+                    'vip.read',
                     'config.read', 'config.history', 'config.view_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read',
@@ -1639,13 +1697,126 @@ async def ensure_agent_activity_logs_table():
             await close_database_connection(conn)
         # Don't raise - this is not critical for system operation
 
+# Schema-version gate for the migration runner.
+#
+# >>> BUMP THIS whenever you add/modify ANY step in _run_all_migrations_inner()
+# >>> that changes the schema (table/column/index/constraint) OR seeded/role data
+# >>> (e.g. update_system_roles_to_enterprise_rbac). Otherwise the new step will
+# >>> NOT run on databases already marked at the current version.
+#
+# When the DB already records >= this version, run_all_migrations() skips the
+# whole (lock-heavy) idempotent sequence, so redeploys/scale-ups issue NO DDL and
+# a concurrently-serving replica's traffic cannot block ALTER / CREATE INDEX (the
+# rolling-deploy startup crash that motivated this gate).
+#
+# Backward compatibility (the product runs at many versions across companies):
+#  - First start on this code: no marker -> applied_version is NULL -> the FULL
+#    sequence runs (upgrades any prior version), THEN the marker is written. So
+#    upgrading from any older version is unaffected.
+#  - The marker is written ONLY after _run_all_migrations_inner() completes with
+#    no exception, so an interrupted/failed migration never marks an incomplete
+#    schema as done — the next start retries.
+#  - Behavior change vs the historical "re-run every idempotent ensure_* on every
+#    start": once marked, same-version restarts no longer re-run (and therefore no
+#    longer auto-repair manual drift). To force a re-run, bump SCHEMA_VERSION or
+#    delete the schema_migrations row.
+#
+# v1.7.0 (Issue #27 — HA/VIP Keepalived management): bumped 1 -> 2 so the new
+# additive ensure_vip_tables() step (two brand-new tables) actually runs on
+# databases already marked at version 1. The whole re-run is idempotent.
+# v1.7.0 self-review: bumped 2 -> 3 so the additive `applied_snapshot` column on
+# vip_instances (enables VIP reject/restore-to-previous) lands on DBs marked at 2.
+# v1.7.0 self-review: bumped 3 -> 4 for the additive `keepalived_config_path` column on
+# haproxy_clusters (cluster-driven keepalived.conf path, like haproxy_config_path).
+# v1.7.0 self-review: bumped 4 -> 5 to drop the table-level UNIQUE on vip_instances.name
+# and replace it with a partial unique index (active rows only), so a soft-deleted VIP's
+# name is reusable — consistent with the address/VRID partial indexes. Idempotent re-run.
+# v1.7.2: bumped 5 -> 6 for the additive `purge_on_teardown` column on vip_instances
+# (opt-in "also uninstall the keepalived package on delete"; default FALSE keeps the safe
+# graceful-teardown behaviour). Additive + idempotent.
+# v1.7.2: bumped 6 -> 7 for the additive `pending_delete` column on vip_instances
+# (approval-gated VIP deletion: a delete is staged for Apply Management and the VIP keeps
+# running until APPROVED, so an agent never tears down without explicit human approval).
+# v1.8.0 (Issue #35 — ACME DNS-01 challenge support): bumped 7 -> 8 for additive DNS-01
+# columns on letsencrypt_accounts/letsencrypt_orders/acme_challenges and the brand-new
+# letsencrypt_account_dns_credentials table (ensure_letsencrypt_dns_credentials step).
+# All additive + idempotent; default challenge_type 'http-01' keeps existing flows byte-identical.
+SCHEMA_VERSION = 8
+
+
 async def run_all_migrations():
-    """Run all database migrations"""
+    """Run all database migrations.
+
+    Hardened for multiple backend replicas / rolling deploys:
+    - A session-level advisory lock serializes the run so only one pod migrates
+      at a time (others wait, then hit the version gate and skip). It is
+      session-scoped, so it auto-releases if a pod dies mid-migration.
+    - A schema-version marker (schema_migrations) gates the run: when the DB is
+      already at SCHEMA_VERSION the whole idempotent sequence is skipped, so no
+      DDL is issued and a serving replica's traffic can't block it.
+    If the advisory lock or marker can't be used, we fall back to running the
+    (idempotent) migrations rather than crashing startup.
+    """
     logger.info("Starting database migrations...")
-    
+    MIGRATION_ADVISORY_LOCK_KEY = 1836016242  # single-key advisory space ("migr"); distinct from the (ns,id) locks used elsewhere
+    lock_conn = None
+    lock_acquired = False
+    try:
+        lock_conn = await get_database_connection()
+        try:
+            await lock_conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            lock_acquired = True
+            logger.info("Acquired migration advisory lock (migrations serialized across pods)")
+        except Exception as _lock_e:
+            logger.warning(f"Could not acquire migration advisory lock; proceeding (migrations are idempotent): {_lock_e}")
+
+        # Schema-version gate: skip the lock-heavy sequence if the DB is current.
+        applied_version = None
+        try:
+            await lock_conn.execute("""
+                CREATE TABLE IF NOT EXISTS schema_migrations (
+                    id INTEGER PRIMARY KEY DEFAULT 1,
+                    version INTEGER NOT NULL,
+                    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                    CONSTRAINT schema_migrations_singleton CHECK (id = 1)
+                )
+            """)
+            applied_version = await lock_conn.fetchval("SELECT version FROM schema_migrations WHERE id = 1")
+        except Exception as _mk_e:
+            logger.warning(f"schema_migrations marker unavailable; running full migrations: {_mk_e}")
+            applied_version = None
+
+        if applied_version is not None and applied_version >= SCHEMA_VERSION:
+            logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
+            return
+
+        await _run_all_migrations_inner()
+
+        try:
+            await lock_conn.execute("""
+                INSERT INTO schema_migrations (id, version, applied_at)
+                VALUES (1, $1, CURRENT_TIMESTAMP)
+                ON CONFLICT (id) DO UPDATE SET version = EXCLUDED.version, applied_at = EXCLUDED.applied_at
+            """, SCHEMA_VERSION)
+            logger.info(f"Recorded schema version {SCHEMA_VERSION} in schema_migrations.")
+        except Exception as _wr_e:
+            logger.warning(f"Could not record schema version marker (migrations still applied): {_wr_e}")
+    finally:
+        if lock_acquired and lock_conn is not None:
+            try:
+                await lock_conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_ADVISORY_LOCK_KEY)
+            except Exception:
+                pass
+        if lock_conn is not None:
+            await close_database_connection(lock_conn)
+
+
+async def _run_all_migrations_inner():
+    """The full idempotent migration sequence. Runs under the migration advisory
+    lock and is gated by the schema-version marker in run_all_migrations()."""
     # First, ensure basic database schema exists
     await run_init_sql()
-    
+
     # Then run additional migrations
     await ensure_agents_table()
     await ensure_config_versions_metadata_column()
@@ -1684,6 +1855,9 @@ async def run_all_migrations():
     await ensure_system_settings_table()
     await ensure_acme_tables()
     await ensure_acme_columns_on_existing_tables()
+    # Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+    # MUST run after ensure_acme_tables() (FK references letsencrypt_accounts).
+    await ensure_letsencrypt_dns_credentials()
     # Issue #11 cleanup: must run AFTER acme_tables/columns to ensure FK refs exist
     await cleanup_orphan_acme_challenge_backend()
     # v1.5.0 Feature A (ACME diagnostics) + Feature B (site wizard)
@@ -1700,8 +1874,212 @@ async def run_all_migrations():
     # (cluster_id, bind_address, bind_port) WHERE is_active.
     await ensure_frontends_bind_unique_constraint()
 
+    # Issue #18 — TOTP MFA (v1.6.0): additive columns + 3 new tables
+    await ensure_mfa_columns()
+
+    # Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
+    # MUST stay last: FK-references haproxy_cluster_pools/agents/users, all created above.
+    await ensure_vip_tables()
+
     logger.info("Database migrations completed successfully.")
 
+
+async def ensure_mfa_columns():
+    """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
+
+    All operations are idempotent (ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS).
+    Default behavior preserved: every existing user gets mfa_enabled=FALSE, so login
+    flow is byte-identical for accounts that don't opt in.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            ALTER TABLE users
+                ADD COLUMN IF NOT EXISTS mfa_enabled BOOLEAN DEFAULT FALSE NOT NULL,
+                ADD COLUMN IF NOT EXISTS mfa_method VARCHAR(20),
+                ADD COLUMN IF NOT EXISTS mfa_secret_encrypted TEXT,
+                ADD COLUMN IF NOT EXISTS mfa_enrolled_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_at TIMESTAMP,
+                ADD COLUMN IF NOT EXISTS mfa_last_used_totp_step BIGINT;
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_backup_codes (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                code_hash VARCHAR(255) NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute("""
+            CREATE INDEX IF NOT EXISTS idx_mfa_backup_codes_user
+              ON mfa_backup_codes(user_id);
+        """)
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_logins (
+                id SERIAL PRIMARY KEY,
+                user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+                challenge_token VARCHAR(64) UNIQUE NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                used_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                ip_address INET
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_token ON mfa_pending_logins(challenge_token);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_expires ON mfa_pending_logins(expires_at);"
+        )
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS mfa_pending_enrollments (
+                user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
+                secret_encrypted TEXT NOT NULL,
+                attempts INTEGER DEFAULT 0 NOT NULL,
+                expires_at TIMESTAMP NOT NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+            );
+        """)
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_mfa_pending_enroll_expires ON mfa_pending_enrollments(expires_at);"
+        )
+
+        logger.info("✅ MFA migration completed (Issue #18 — Phase 1)")
+    except Exception as e:
+        logger.error(f"Failed to ensure MFA columns: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_user_activity_logs_table
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+async def ensure_vip_tables():
+    """Issue #27 — HA/VIP (Keepalived) management (v1.7.0). Additive only:
+    two brand-new tables (vip_instances, vip_members) + indexes. No ALTER of any
+    existing table, so the entire current fleet is byte-identical. Fully idempotent
+    (CREATE TABLE/INDEX IF NOT EXISTS). FK targets (haproxy_cluster_pools, agents,
+    users) are created earlier in the sequence — this function is registered LAST.
+
+    Backward-compat: a cluster/agent with no VIP row is unaffected; the agent
+    delivery endpoint returns 'not_configured' for every node without a membership.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Cluster-driven keepalived.conf path (mirrors haproxy_config_path): additive +
+        # idempotent, with a universal default so operators need set nothing. The agent
+        # pulls this from its cluster, exactly like the HAProxy paths.
+        await conn.execute(
+            "ALTER TABLE haproxy_clusters ADD COLUMN IF NOT EXISTS keepalived_config_path "
+            "VARCHAR(500) DEFAULT '/etc/keepalived/keepalived.conf';")
+
+        # VIP instance: one row per virtual IP (one VRRP group), anchored to a pool.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_instances (
+                id SERIAL PRIMARY KEY,
+                name VARCHAR(255) NOT NULL,
+                description TEXT,
+                pool_id INTEGER NOT NULL REFERENCES haproxy_cluster_pools(id) ON DELETE CASCADE,
+                virtual_ip VARCHAR(45) NOT NULL,
+                prefix_length INTEGER NOT NULL DEFAULT 24,
+                virtual_router_id INTEGER NOT NULL,
+                advert_int INTEGER NOT NULL DEFAULT 1,
+                auth_pass_encrypted TEXT,
+                use_unicast BOOLEAN NOT NULL DEFAULT TRUE,
+                track_haproxy BOOLEAN NOT NULL DEFAULT TRUE,
+                is_active BOOLEAN NOT NULL DEFAULT TRUE,
+                last_config_status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
+                applied_snapshot JSONB,
+                created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_vrid_range CHECK (virtual_router_id BETWEEN 1 AND 255)
+            );
+        """)
+        # Additive (idempotent) for DBs that created vip_instances before applied_snapshot
+        # existed (v1.7.0 self-review): holds the field-level state as of the last Apply so
+        # a pending edit can be rejected and fully reverted to the previous applied state.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS applied_snapshot JSONB;")
+        # Opt-in package removal (v1.7.2): when an operator deletes a VIP and explicitly ticks
+        # "also uninstall keepalived from the node(s)", we set this flag so the teardown
+        # delivery tells the agent to purge the OS package. Default FALSE = the safe enterprise
+        # default (stop+disable+remove our config, but KEEP the package). Additive + idempotent;
+        # a node we never managed stays untouched regardless.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Approval-gated deletion (v1.7.2): deleting a RUNNING VIP from the UI does NOT take
+        # effect immediately — it sets pending_delete=TRUE and stages a vip-*-delete version
+        # for Apply Management. The VIP stays is_active=TRUE (agents keep serving it, NOTHING
+        # is torn down) until the operator APPROVES; only then does apply flip is_active=FALSE
+        # and the agents tear down. Reject clears the flag and the VIP keeps running untouched.
+        # This guarantees an agent never tears a VIP down without an explicit human approval —
+        # protecting production. Additive + idempotent.
+        await conn.execute("ALTER TABLE vip_instances ADD COLUMN IF NOT EXISTS pending_delete BOOLEAN NOT NULL DEFAULT FALSE;")
+        # Uniqueness as PARTIAL indexes on active rows so a soft-deleted VIP frees its
+        # name/address/VRID for immediate reuse (a table-level UNIQUE would keep blocking it).
+        # NAME (v1.7.0 self-review): the original CREATE used a table-level UNIQUE on name,
+        # which left a soft-deleted VIP's name blocked (you couldn't re-create a VIP with the
+        # same name) — inconsistent with addr/VRID. Drop that constraint and use a partial
+        # index instead. Idempotent: no-op on a fresh table (no inline UNIQUE) and on re-run.
+        await conn.execute("ALTER TABLE vip_instances DROP CONSTRAINT IF EXISTS vip_instances_name_key;")
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_name_active ON vip_instances(name) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_addr_active ON vip_instances(virtual_ip) WHERE is_active=TRUE;"
+        )
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_vrid_active ON vip_instances(pool_id, virtual_router_id) WHERE is_active=TRUE;"
+        )
+
+        # Per-node membership: which agents participate + their VRRP role/priority,
+        # the applied (delivered) config snapshot, and the agent's deploy ack.
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS vip_members (
+                id SERIAL PRIMARY KEY,
+                vip_id INTEGER NOT NULL REFERENCES vip_instances(id) ON DELETE CASCADE,
+                agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
+                network_interface VARCHAR(64) NOT NULL,
+                role VARCHAR(10) NOT NULL DEFAULT 'BACKUP',
+                priority INTEGER NOT NULL DEFAULT 100,
+                applied_config_content TEXT,
+                applied_config_hash VARCHAR(64),
+                last_deploy_state VARCHAR(24),
+                last_deploy_message TEXT,
+                last_deploy_hash VARCHAR(64),
+                last_deploy_at TIMESTAMP,
+                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+                CONSTRAINT vip_member_role CHECK (role IN ('MASTER','BACKUP')),
+                CONSTRAINT vip_member_priority_range CHECK (priority BETWEEN 1 AND 254),
+                CONSTRAINT vip_member_unique UNIQUE (vip_id, agent_id)
+            );
+        """)
+        # Last line of defense against split-brain: at most one MASTER per VIP.
+        await conn.execute(
+            "CREATE UNIQUE INDEX IF NOT EXISTS uq_vip_one_master ON vip_members(vip_id) WHERE role='MASTER';"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_vip ON vip_members(vip_id);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_vip_members_agent ON vip_members(agent_id);"
+        )
+
+        logger.info("✅ VIP tables ensured (Issue #27 — HA/VIP Keepalived management)")
+    except Exception as e:
+        logger.error(f"Failed to ensure VIP tables: {e}")
+        # Don't raise — follow the same defensive pattern as ensure_mfa_columns
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 async def add_ssl_certificate_id_to_backend_servers():
     """Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
     conn = None
@@ -3278,6 +3656,23 @@ async def ensure_acme_columns_on_existing_tables():
             ('last_attempt_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS last_attempt_at TIMESTAMPTZ"),
             # Commit 3a: track auto-completion task lock/poll timestamps for atomic claim across replicas
             ('orders_updated_at_idx', "CREATE INDEX IF NOT EXISTS idx_letsencrypt_orders_status_updated ON letsencrypt_orders(status, updated_at) WHERE status = 'valid' AND ssl_certificate_id IS NULL"),
+            # Issue #35 (v1.8.0 — ACME DNS-01): per-account challenge method + DNS provider selection
+            ('acct_challenge_type', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('acct_dns_provider', "ALTER TABLE letsencrypt_accounts ADD COLUMN IF NOT EXISTS dns_provider VARCHAR(50)"),
+            # per-order challenge method + bounded DNS-01 retry chain (dns01_parent_order_id is a PLAIN INTEGER, not a FK,
+            # to avoid a self-referential cascade interacting with account/order bulk DELETEs)
+            ('order_challenge_type', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('order_dns01_attempts', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_attempts INTEGER DEFAULT 0"),
+            ('order_dns01_last_attempt_at', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_last_attempt_at TIMESTAMPTZ"),
+            ('order_dns01_parent_order_id', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_parent_order_id INTEGER"),
+            ('order_dns01_retry_claimed', "ALTER TABLE letsencrypt_orders ADD COLUMN IF NOT EXISTS dns01_retry_claimed BOOLEAN DEFAULT FALSE"),
+            # per-challenge DNS-01 lifecycle state
+            ('chal_challenge_type', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS challenge_type VARCHAR(20) DEFAULT 'http-01'"),
+            ('chal_dns_txt_value', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_txt_value TEXT"),
+            ('chal_dns_record_published', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_published BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_record_cleaned', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_record_cleaned BOOLEAN DEFAULT FALSE"),
+            ('chal_dns_published_at', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS dns_published_at TIMESTAMPTZ"),
+            ('chal_manual_confirm_deadline', "ALTER TABLE acme_challenges ADD COLUMN IF NOT EXISTS manual_confirm_deadline TIMESTAMPTZ"),
         ]:
             try:
                 await conn.execute(sql)
@@ -3293,6 +3688,34 @@ async def ensure_acme_columns_on_existing_tables():
         logger.error(f"Error adding ACME columns: {e}")
 
 
+async def ensure_letsencrypt_dns_credentials():
+    """Issue #35 (v1.8.0 — ACME DNS-01): per-account encrypted DNS provider credentials.
+
+    Idempotent (CREATE TABLE IF NOT EXISTS). FK to letsencrypt_accounts (created earlier by
+    ensure_acme_tables). Credentials are Fernet-encrypted at rest (backend/utils/dns_credentials.py);
+    only the provider name + timestamps are ever surfaced to the API.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS letsencrypt_account_dns_credentials (
+                id SERIAL PRIMARY KEY,
+                account_id INTEGER NOT NULL UNIQUE REFERENCES letsencrypt_accounts(id) ON DELETE CASCADE,
+                dns_provider VARCHAR(50) NOT NULL,
+                credentials_encrypted TEXT NOT NULL,
+                created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
+                updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
+            )
+        """)
+        logger.info("Ensured letsencrypt_account_dns_credentials table")
+        await close_database_connection(conn)
+    except Exception as e:
+        if conn:
+            await close_database_connection(conn)
+        logger.error(f"Error ensuring letsencrypt_account_dns_credentials: {e}")
+
+
 async def cleanup_orphan_acme_challenge_backend():
     """
     Issue #11: One-time cleanup of orphan `_acme_challenge_backend` rows that may

backend/main.py

@@ -8,7 +8,8 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.5.2", "releaseName": "ACME Diagnostics Panel Hardening", "releaseDate": "2026-05-13"}
+# Build/deploy marker for the v1.8.x (Issue #35, DNS-01) rollout — ensures the pipeline ships this commit's image.
+_version_info = {"version": "1.8.4", "releaseName": "Agent installer self-kill fix", "releaseDate": "2026-06-27"}
 for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
     try:
         with open(_vpath) as _vf:
@@ -40,6 +41,8 @@ from routers.settings import router as settings_router
 from routers.letsencrypt import router as letsencrypt_router
 from routers.acme_diagnostics import router as acme_diagnostics_router
 from routers.site_wizard import router as site_wizard_router
+from routers.mfa import router as mfa_router
+from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -302,6 +305,21 @@ async def complete_pending_acme_orders():
                         WHERE (
                             status IN ('pending', 'processing', 'ready')
                             OR (status = 'valid' AND ssl_certificate_id IS NULL)
+                            -- Issue #35: bounded DNS-01 retry. ONLY dns-01 invalids with remaining
+                            -- budget + elapsed backoff are claimed; http-01 invalids are NEVER matched
+                            -- (their existing skip-and-log is preserved).
+                            OR (
+                                status = 'invalid' AND challenge_type = 'dns-01'
+                                AND ssl_certificate_id IS NULL
+                                AND COALESCE(dns01_retry_claimed, FALSE) = FALSE
+                                AND COALESCE(dns01_attempts, 0) < 3
+                                AND (
+                                    dns01_last_attempt_at IS NULL
+                                    OR dns01_last_attempt_at < NOW() - (
+                                        (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
+                                        || ' minutes')::INTERVAL
+                                )
+                            )
                         )
                         AND created_at > NOW() - INTERVAL '7 days'
                         AND (updated_at IS NULL OR updated_at < NOW() - INTERVAL '30 seconds')
@@ -335,9 +353,17 @@ async def complete_pending_acme_orders():
                 continue
 
             logger.info(f"[ACME-COMPLETE] Claimed {len(claimed_ids)} order(s) for completion: {claimed_ids}")
-            
+
+            from services.dns01_orchestrator import (
+                advance_dns01_order, retry_invalid_dns01, reconcile_dns01_cleanup,
+            )
+
             for oid in claimed_ids:
                 try:
+                    # Issue #35: advance the DNS-01 publish->confirm->respond state machine for
+                    # pending dns-01 orders (no-op for http-01 or non-pending orders).
+                    await advance_dns01_order(oid)
+
                     status_info = await acme_svc.check_order_status(oid)
                     current_status = status_info.get('status')
 
@@ -350,12 +376,21 @@ async def complete_pending_acme_orders():
                         result = await _complete_certificate(oid)
                         logger.info(f"[ACME-COMPLETE] Order {oid} completed - {result.get('message', '')}")
                     elif current_status == 'invalid':
-                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid, skipping")
+                        # Issue #35: bounded DNS-01 fresh-order retry (no-op for http-01).
+                        await retry_invalid_dns01(oid)
+                        logger.warning(f"[ACME-COMPLETE] Order {oid} is invalid")
                     elif current_status in ('pending', 'processing'):
                         logger.info(f"[ACME-COMPLETE] Order {oid} still {current_status}, will retry next cycle")
                 except Exception as poll_err:
                     logger.error(f"[ACME-COMPLETE] Failed to complete order {oid}: {poll_err}")
 
+            # Issue #35: best-effort cleanup of TXT records left published on terminal orders
+            # (covers a failed cleanup or the kill-switch being flipped off). NOT gated by the switch.
+            try:
+                await reconcile_dns01_cleanup()
+            except Exception as rec_err:
+                logger.debug(f"[ACME-COMPLETE] DNS-01 reconcile skipped: {rec_err}")
+
             # NOTE: v1.5.0 wizard-staged processing now runs BEFORE the
             # claimed_ids early-continue above (Bulgu #2 fix), so it executes
             # every cycle regardless of pending/processing volume.
@@ -671,7 +706,9 @@ async def check_letsencrypt_renewals():
                     skip = False
                     try:
                         order = await conn2.fetchrow(
-                            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1",
+                            "SELECT o.account_id, o.domains, o.cluster_ids, o.challenge_type, a.dns_provider "
+                            "FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id "
+                            "WHERE o.id = $1",
                             order_id
                         )
                         if order:
@@ -687,15 +724,37 @@ async def check_letsencrypt_renewals():
                             if existing:
                                 logger.info(f"[ACME-RENEWAL] Skipping cert {cert['id']} - order {existing['id']} already in progress")
                                 skip = True
+                            elif (order['challenge_type'] == 'dns-01'):
+                                # Issue #35: manual DNS-01 cannot auto-renew unattended; and for an
+                                # automated provider, don't re-mint hourly if a recent retry chain already
+                                # exhausted its budget (avoids tripping the CA new-order rate limit).
+                                if (order['dns_provider'] or 'manual') == 'manual':
+                                    logger.warning(f"[ACME-RENEWAL] cert {cert['id']} uses manual DNS-01; cannot auto-renew unattended (publish the TXT and renew manually)")
+                                    skip = True
+                                else:
+                                    exhausted = await conn2.fetchrow("""
+                                        SELECT id FROM letsencrypt_orders
+                                        WHERE domains::text = $1::text AND challenge_type = 'dns-01'
+                                        AND status = 'invalid' AND COALESCE(dns01_attempts, 0) >= 3
+                                        AND created_at > NOW() - INTERVAL '24 hours'
+                                        LIMIT 1
+                                    """, json.dumps(domains))
+                                    if exhausted:
+                                        logger.warning(f"[ACME-RENEWAL] cert {cert['id']} DNS-01 renewal recently failed (check DNS); skipping re-mint for 24h")
+                                        skip = True
                     finally:
                         await close_database_connection(conn2)
 
                     if not order or skip:
                         continue
 
-                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids)
-                    await acme_svc.respond_to_challenges(new_order['order_id'])
-                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} for cert {cert['id']} ({cert['name']})")
+                    challenge_type = order['challenge_type'] or 'http-01'
+                    new_order = await acme_svc.create_order(order['account_id'], domains, cluster_ids, challenge_type=challenge_type)
+                    # http-01 responds immediately (token served continuously); dns-01 is driven by the
+                    # orchestrator AFTER the TXT is published (never respond before publish).
+                    if challenge_type != 'dns-01':
+                        await acme_svc.respond_to_challenges(new_order['order_id'])
+                    logger.info(f"[ACME-RENEWAL] Initiated renewal order {new_order['order_id']} ({challenge_type}) for cert {cert['id']} ({cert['name']})")
                 except Exception as cert_err:
                     logger.error(f"[ACME-RENEWAL] Failed to initiate renewal for cert {cert['id']}: {cert_err}")
 
@@ -820,6 +879,7 @@ app.include_router(config_router)  # Configuration management
 app.include_router(maintenance_router, prefix="/api", tags=["maintenance"])  # Database cleanup & maintenance
 app.include_router(auth_router)
 app.include_router(user_router)
+app.include_router(mfa_router)
 app.include_router(frontend_router)
 app.include_router(backend_router)
 app.include_router(cluster_router)
@@ -834,6 +894,7 @@ app.include_router(settings_router)
 app.include_router(letsencrypt_router)
 app.include_router(acme_diagnostics_router)  # v1.5.0 Issue #13: ACME Diagnostic Panel
 app.include_router(site_wizard_router)  # v1.5.0 Issue #14: New Site Setup Wizard
+app.include_router(vip_router)  # v1.7.0 Issue #27: HA/VIP (Keepalived) management
 
 
 # Legacy URL alias: /api/proxied-hosts/* → 308 redirect to /api/sites/*.
@@ -1051,7 +1112,7 @@ async def serve_acme_challenge(token: str):
     try:
         conn = await get_database_connection()
         row = await conn.fetchrow(
-            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) LIMIT 1",
+            "SELECT key_authorization FROM acme_challenges WHERE token = $1 AND (status IN ('pending', 'processing') OR status IS NULL) AND (challenge_type = 'http-01' OR challenge_type IS NULL) LIMIT 1",
             token
         )
         if row:

backend/middleware/error_handler.py

@@ -126,10 +126,26 @@ class GlobalExceptionHandler:
             except Exception as body_error:
                 logger.debug(f"Could not extract raw body for debugging: {body_error}")
         
-        # Enhanced log message for agent heartbeats
-        log_message = f"Validation error: {str(exc)}"
+        # Build a sanitized log summary. The raw `str(exc)` from Pydantic
+        # contains the user-supplied `input` value for each failed field —
+        # which leaks secrets like TOTP codes, backup codes, mfa_token, and
+        # passwords to plaintext logs. Use only field NAMES + types here;
+        # `validation_details` (already sanitized to {field, message, type})
+        # is attached separately for downstream structured logging.
+        _field_names = [
+            err.get("field", "unknown")
+            for err in error_details.get("validation_errors", [])
+        ]
+        _err_count = len(error_details.get("validation_errors", []))
+        log_message = (
+            f"Validation error: {_err_count} field(s) failed validation: "
+            f"[{', '.join(_field_names)}]"
+        )
         if agent_name != "unknown":
-            log_message = f"Agent '{agent_name}' heartbeat validation error: {str(exc)}"
+            log_message = (
+                f"Agent '{agent_name}' heartbeat validation error: "
+                f"{_err_count} field(s) failed: [{', '.join(_field_names)}]"
+            )
         
         # Log validation error with enhanced details
         log_with_correlation(

backend/middleware/mfa_rate_limit_key.py

@@ -0,0 +1,147 @@
+"""MFA rate-limit key extraction (user-aware + ingress-aware).
+
+Problem with the default ``slowapi.util.get_remote_address``:
+
+  * Behind an ingress / reverse proxy / load balancer, every request looks
+    like it originates from the same upstream IP (the proxy itself). A
+    5/minute IP-bucket therefore becomes a 5/minute *whole-organization*
+    bucket. In a 500-user enterprise rolling out MFA, this stalls the
+    rollout to a trickle.
+
+Strategy (combined B + C from the design review):
+
+  1. **User-aware key (preferred):** if the request carries a valid Bearer
+     JWT, derive the bucket from ``user:<id>``. Each authenticated user
+     gets an isolated bucket, regardless of source IP. A malicious user
+     burning their own quota cannot starve the other 499.
+
+  2. **Trusted-proxy XFF fallback:** if the request is unauthenticated
+     (e.g. ``/login`` flow, future endpoints) and the TCP peer is in
+     ``MFA_TRUSTED_PROXY_CIDRS``, peel the first hop off ``X-Forwarded-For``.
+     This preserves real-IP buckets behind a known ingress without
+     accepting spoofed headers from the public internet.
+
+  3. **Default fallback:** plain ``request.client.host`` (slowapi default).
+
+JWT decode is intentionally signature-verified (replay/spoof protection)
+and *sync* — slowapi's decorator hook is sync, and our JWT library
+(python-jose) is sync as well. Failed verification silently downgrades
+to IP-based bucketing — never crashes the decorator.
+
+Env vars:
+  * ``MFA_TRUSTED_PROXY_CIDRS`` — comma-separated CIDR list of trusted
+    upstream proxies. Empty (default) disables XFF parsing entirely,
+    which is the safe choice when the deployment topology is unknown.
+    Examples:
+        ``MFA_TRUSTED_PROXY_CIDRS=10.0.0.0/8,172.16.0.0/12``
+        ``MFA_TRUSTED_PROXY_CIDRS=192.168.0.0/16``
+"""
+from __future__ import annotations
+
+import logging
+import os
+from ipaddress import ip_address, ip_network
+from typing import List
+
+from fastapi import Request
+from jose import jwt
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_trusted_cidrs() -> List:
+    """Parse the trusted-proxy CIDR list at import time.
+
+    A malformed entry is logged and skipped — we never crash the process
+    over a typo in operational config.
+    """
+    raw = os.getenv("MFA_TRUSTED_PROXY_CIDRS", "").strip()
+    if not raw:
+        return []
+    nets = []
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        try:
+            nets.append(ip_network(entry, strict=False))
+        except ValueError:
+            logger.warning(
+                "MFA_TRUSTED_PROXY_CIDRS: ignoring invalid CIDR %r", entry
+            )
+    return nets
+
+
+_TRUSTED_NETS = _parse_trusted_cidrs()
+
+
+def _peer_ip(request: Request) -> str:
+    """The TCP peer IP — never raises; falls back to ``0.0.0.0``."""
+    return request.client.host if request.client else "0.0.0.0"
+
+
+def _is_trusted_peer(peer: str) -> bool:
+    if not _TRUSTED_NETS:
+        return False
+    try:
+        peer_ip = ip_address(peer)
+    except ValueError:
+        return False
+    return any(peer_ip in net for net in _TRUSTED_NETS)
+
+
+def _real_ip(request: Request) -> str:
+    """If TCP peer is in a trusted proxy CIDR, peel off the first
+    ``X-Forwarded-For`` IP; otherwise return the peer.
+
+    ``X-Forwarded-For`` from an *untrusted* peer is intentionally ignored —
+    accepting it would let any client spoof their bucket.
+    """
+    peer = _peer_ip(request)
+    if not _is_trusted_peer(peer):
+        return peer
+    xff = request.headers.get("X-Forwarded-For")
+    if not xff:
+        return peer
+    first = xff.split(",")[0].strip()
+    return first or peer
+
+
+def _user_id_from_jwt(request: Request) -> str | None:
+    """Sync JWT decode → ``user_id`` (or ``sub``) claim. None on failure.
+
+    Uses the same secret + algorithm as ``auth_middleware`` so a token that
+    is valid for the API surface is also valid for the rate-limit key.
+    Bad / missing / expired tokens silently return None — slowapi falls
+    back to IP bucketing.
+    """
+    auth = request.headers.get("Authorization", "")
+    if not auth.startswith("Bearer "):
+        return None
+    token = auth[7:].strip()
+    if not token or token in {"null", "undefined"}:
+        return None
+    # Late import to avoid pulling jose into module-load when not needed.
+    try:
+        from config import JWT_ALGORITHM, JWT_SECRET_KEY
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None
+    uid = payload.get("user_id") or payload.get("sub")
+    if uid is None:
+        return None
+    return str(uid)
+
+
+def mfa_rate_limit_key(request: Request) -> str:
+    """slowapi ``key_func`` for MFA endpoints.
+
+    Order:
+      1. Authenticated → ``user:<id>``
+      2. Trusted-proxy XFF → ``ip:<first hop>``
+      3. TCP peer → ``ip:<peer>``
+    """
+    uid = _user_id_from_jwt(request)
+    if uid is not None:
+        return f"user:{uid}"
+    return f"ip:{_real_ip(request)}"

backend/middleware/mfa_rate_limits.py

@@ -0,0 +1,79 @@
+"""MFA rate-limit configuration (env-overridable).
+
+Best-practice pattern:
+  - Secure-by-default values live in code (kept in sync with the threat model).
+  - Operations can override per-environment via env vars (ConfigMap on K8s)
+    WITHOUT a code change / re-release.
+  - All limits funnel through a single named constant so the decorator stays
+    declarative (``@limiter.limit(MFA_LIMITS.enroll_start)``).
+
+Env-var precedence::
+
+    MFA_RATE_LIMIT_<NAME>  >  default in code
+
+slowapi limit string syntax: ``<count>/<period>`` where period is
+``second|minute|hour|day``. Example: ``"5/minute"``.
+
+NOTE: slowapi binds limits at import time. A change to an env var requires a
+backend restart (rolling restart on K8s, ``docker compose restart backend``
+locally). This is consistent with how ``SECRET_KEY`` / ``MFA_ENCRYPTION_KEY``
+behave.
+"""
+from __future__ import annotations
+
+import logging
+import os
+import re
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+# slowapi limit-string format guard. Keeps a typo from silently disabling
+# rate-limiting at process start.
+_LIMIT_RE = re.compile(r"^\d+/(second|minute|hour|day)$")
+
+
+def _env(name: str, default: str) -> str:
+    """Read ``MFA_RATE_LIMIT_<NAME>``; fall back to ``default``.
+
+    Validates the limit string. On bad input, logs a warning and returns the
+    secure default instead of crashing the process.
+    """
+    value = os.getenv(f"MFA_RATE_LIMIT_{name}", default).strip()
+    if not _LIMIT_RE.match(value):
+        logger.warning(
+            "MFA_RATE_LIMIT_%s='%s' is not a valid slowapi limit string "
+            "(expected '<n>/<second|minute|hour|day>'); using default '%s'.",
+            name, value, default,
+        )
+        return default
+    return value
+
+
+@dataclass(frozen=True)
+class MfaRateLimits:
+    """Aggregate of MFA endpoint rate-limit strings (slowapi format).
+
+    Defaults assume the rate-limit ``key_func`` is ``mfa_rate_limit_key``
+    (user-aware + ingress-aware), NOT raw IP. Per-user buckets are safe to
+    keep generous because a misbehaving user only burns their own quota and
+    cannot starve the rest of the org. If you re-key on raw IP, retighten
+    these values (see README + ``MFA_RATE_LIMIT_<NAME>`` env overrides).
+    """
+
+    # Enrollment lifecycle — per-user buckets, large enough for org-wide rollout
+    enroll_start: str = _env("ENROLL_START", "10/minute")
+    enroll_confirm: str = _env("ENROLL_CONFIRM", "10/minute")
+
+    # Self-service maintenance
+    disable: str = _env("DISABLE", "10/minute")
+    regenerate_backup_codes: str = _env("REGENERATE_BACKUP_CODES", "5/hour")
+
+    # Admin operations (per-admin bucket; bulk reset stays tight because
+    # it is an emergency-only flow).
+    admin_reset: str = _env("ADMIN_RESET", "60/hour")
+    admin_reset_all: str = _env("ADMIN_RESET_ALL", "1/day")
+
+
+# Module-level singleton — import this from routers/mfa.py.
+MFA_LIMITS = MfaRateLimits()

backend/middleware/rate_limiter.py

@@ -2,6 +2,7 @@
 Rate Limiting Middleware for Production Security
 Protects API endpoints from abuse and DDoS attacks
 """
+import os
 import time
 import logging
 from typing import Callable
@@ -16,10 +17,15 @@ from database.connection import redis_client
 
 logger = logging.getLogger(__name__)
 
-# Rate limiter instance using Redis backend  
+# Resolve the Redis storage URI from REDIS_URL (set by docker-compose / k8s
+# ConfigMap). Falls back to the compose service hostname for backward
+# compatibility when REDIS_URL is unset.
+_REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379").rstrip("/")
+_LIMITER_STORAGE_URI = f"{_REDIS_URL}/0" if "/" not in _REDIS_URL.split("//", 1)[-1] else _REDIS_URL
+
 limiter = Limiter(
     key_func=get_remote_address,
-    storage_uri="redis://redis:6379/0",
+    storage_uri=_LIMITER_STORAGE_URI,
     default_limits=["1000/hour"],  # Default global limit
     retry_after=lambda name, t: int(t) + 10
 )

backend/models/cluster.py

@@ -8,6 +8,7 @@ class HAProxyClusterCreate(BaseModel):
     stats_socket_path: str = "/run/haproxy/admin.sock"
     haproxy_config_path: str = "/etc/haproxy/haproxy.cfg"
     haproxy_bin_path: str = "/usr/sbin/haproxy"  # HAProxy binary path
+    keepalived_config_path: str = "/etc/keepalived/keepalived.conf"  # HA/VIP: keepalived.conf path (Issue #27)
     pool_id: Optional[int] = None  # Which pool this cluster belongs to
 
 class HAProxyClusterUpdate(BaseModel):
@@ -17,6 +18,7 @@ class HAProxyClusterUpdate(BaseModel):
     stats_socket_path: Optional[str] = None
     haproxy_config_path: Optional[str] = None
     haproxy_bin_path: Optional[str] = None
+    keepalived_config_path: Optional[str] = None
     pool_id: Optional[int] = None
     is_active: Optional[bool] = None
     acme_enabled: Optional[bool] = None

backend/models/mfa.py

@@ -0,0 +1,73 @@
+"""MFA-specific Pydantic models — Issue #18, v1.6.0.
+
+Kept in a separate module so the existing User / UserUpdate contracts in
+``backend/models/user.py`` stay byte-identical for backwards compatibility.
+"""
+from typing import List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+class MfaVerifyRequest(BaseModel):
+    """Body of POST /api/auth/login/mfa-verify (pre-auth — no JWT)."""
+
+    mfa_token: str = Field(..., min_length=64, max_length=64)
+    # 6 digits for TOTP or 8 alphanumerics (with optional dash) for backup codes
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaEnrollStartResponse(BaseModel):
+    """Returned by POST /api/mfa/enroll/start."""
+
+    secret: str
+    otpauth_uri: str
+    expires_in: int  # pending enrollment TTL in seconds
+
+
+class MfaEnrollConfirmRequest(BaseModel):
+    """Body of POST /api/mfa/enroll/confirm (TOTP only — backup codes not yet issued)."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaEnrollConfirmResponse(BaseModel):
+    enabled: bool
+    backup_codes: List[str]
+    method: Literal["totp"] = "totp"
+
+
+class MfaDisableRequest(BaseModel):
+    """Body of POST /api/mfa/disable — TOTP or backup."""
+
+    code: str = Field(..., min_length=6, max_length=10)
+
+
+class MfaRegenerateBackupRequest(BaseModel):
+    """Body of POST /api/mfa/backup-codes/regenerate — TOTP only."""
+
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class MfaRegenerateBackupResponse(BaseModel):
+    backup_codes: List[str]
+
+
+class MfaAdminResetRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset/{user_id}."""
+
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaAdminResetAllRequest(BaseModel):
+    """Body of POST /api/mfa/admin-reset-all (emergency)."""
+
+    confirm: Literal["RESET ALL MFA"]
+    reason: str = Field(..., min_length=3, max_length=500)
+
+
+class MfaStatusResponse(BaseModel):
+    enabled: bool
+    method: Optional[str] = None
+    enrolled_at: Optional[str] = None
+    last_used_at: Optional[str] = None
+    backup_codes_remaining: int = 0

backend/models/vip.py

@@ -0,0 +1,245 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Pydantic request/response models for the VIP management API. Field validation
+is strict because several values flow into a generated keepalived.conf and into
+root-run agent commands — we reuse the same FORBIDDEN-metacharacter discipline as
+models/agent.py (Bulgu #81) and validate the VIP as a real IPv4 address.
+
+Pydantic idiom: the project runs pydantic>=2.5; this module uses the v2-native
+@field_validator/@model_validator style (matching models/ssl.py).
+"""
+import ipaddress
+from typing import List, Optional
+
+from pydantic import BaseModel, field_validator, model_validator
+
+# Shell/keepalived.conf metacharacters that must never appear in a value that
+# reaches the generated config or a root-run agent command (mirrors
+# models/agent.py:202, the Bulgu #81 convention).
+_FORBIDDEN = set('$`;&|<>"\'\\\n\r\x00*?')
+
+
+def _validate_iface(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('network_interface must be a non-empty string')
+    s = v.strip()
+    if len(s) > 64:
+        raise ValueError('network_interface too long (max 64 chars)')
+    if any(c in _FORBIDDEN for c in s):
+        raise ValueError('network_interface contains a forbidden character')
+    # Linux iface names: letters, digits, and . _ - : @ (vlans/aliases/altnames)
+    import re as _re
+    if not _re.match(r'^[A-Za-z0-9][A-Za-z0-9._:@-]{0,63}$', s):
+        raise ValueError(
+            'network_interface must start alphanumeric and contain only '
+            'letters, digits, and . _ - : @'
+        )
+    return s
+
+
+def _validate_ipv4(v: str) -> str:
+    if not isinstance(v, str) or not v.strip():
+        raise ValueError('virtual_ip must be a non-empty string')
+    s = v.strip()
+    try:
+        addr = ipaddress.ip_address(s)
+    except ValueError:
+        raise ValueError(f'virtual_ip={v!r} is not a valid IP address')
+    if addr.version != 4:
+        raise ValueError('virtual_ip must be IPv4 — IPv6 VIPs are not supported yet')
+    return s
+
+
+class VIPMemberIn(BaseModel):
+    agent_id: int
+    network_interface: str
+    role: str = 'BACKUP'
+    priority: int = 100
+
+    @field_validator('network_interface')
+    @classmethod
+    def _iface(cls, v):
+        return _validate_iface(v)
+
+    @field_validator('role')
+    @classmethod
+    def _role(cls, v):
+        u = (v or '').strip().upper()
+        if u not in ('MASTER', 'BACKUP'):
+            raise ValueError("role must be 'MASTER' or 'BACKUP'")
+        return u
+
+    @field_validator('priority')
+    @classmethod
+    def _priority(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 254):
+            raise ValueError('priority must be an integer between 1 and 254')
+        return v
+
+
+def _validate_members(members: List['VIPMemberIn']) -> List['VIPMemberIn']:
+    # >=1 node: a single-node VIP is a keepalived-managed floating IP without failover
+    # (valid, e.g. a one-box HAProxy that wants a stable VIP, or before a 2nd node is added).
+    # Two or more nodes give actual VRRP failover. The UI flags the single-node case.
+    if not members:
+        raise ValueError('a VIP needs at least 1 member node')
+    agent_ids = [m.agent_id for m in members]
+    if len(set(agent_ids)) != len(agent_ids):
+        raise ValueError('each node may appear at most once in a VIP')
+    masters = [m for m in members if m.role == 'MASTER']
+    if len(masters) != 1:
+        raise ValueError('exactly one member must be MASTER')
+    master_prio = masters[0].priority
+    if any(m.role == 'BACKUP' and m.priority >= master_prio for m in members):
+        raise ValueError('the MASTER must have a strictly higher priority than every BACKUP')
+    return members
+
+
+def _validate_auth_pass(v: Optional[str]) -> Optional[str]:
+    if v is None or v == '':
+        return None
+    # keepalived PASS auth_pass is silently truncated to 8 chars (B-3) — reject longer
+    # so MASTER/BACKUP never silently disagree.
+    if not (1 <= len(v) <= 8):
+        raise ValueError('auth_pass must be 1-8 characters (keepalived PASS limit)')
+    if any(c in _FORBIDDEN for c in v):
+        raise ValueError('auth_pass contains a forbidden character')
+    # No whitespace: keepalived PASS auth_pass is a single token, and a whitespace-containing
+    # secret would only partially redact in the masked config diff (review HIGH-2).
+    if any(c.isspace() for c in v):
+        raise ValueError('auth_pass must not contain whitespace')
+    return v
+
+
+class VIPCreate(BaseModel):
+    name: str
+    description: Optional[str] = None
+    pool_id: int
+    virtual_ip: str
+    prefix_length: int = 24
+    virtual_router_id: Optional[int] = None   # auto-allocated within the pool when omitted
+    advert_int: int = 1
+    auth_pass: Optional[str] = None           # plaintext on the wire; stored Fernet-encrypted
+    use_unicast: bool = True
+    track_haproxy: bool = True
+    members: List[VIPMemberIn]
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if not isinstance(v, str) or not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255:
+            raise ValueError('name too long (max 255 chars)')
+        if any(c in _FORBIDDEN for c in s):
+            raise ValueError('name contains a forbidden character')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v)
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 32):
+            raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be an integer between 1 and 255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if not isinstance(v, int) or not (1 <= v <= 255):
+            raise ValueError('advert_int must be an integer between 1 and 255 (seconds)')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        _validate_members(self.members)
+        return self
+
+
+class VIPUpdate(BaseModel):
+    """All fields optional — only provided fields are changed. Any change sets the
+    VIP back to last_config_status='PENDING' (router-side)."""
+    name: Optional[str] = None
+    description: Optional[str] = None
+    virtual_ip: Optional[str] = None
+    prefix_length: Optional[int] = None
+    virtual_router_id: Optional[int] = None
+    advert_int: Optional[int] = None
+    auth_pass: Optional[str] = None           # provide only to rotate; omit to keep existing
+    use_unicast: Optional[bool] = None
+    track_haproxy: Optional[bool] = None
+    members: Optional[List[VIPMemberIn]] = None
+
+    @field_validator('name')
+    @classmethod
+    def _name(cls, v):
+        if v is None:
+            return v
+        if not v.strip():
+            raise ValueError('name must be a non-empty string')
+        s = v.strip()
+        if len(s) > 255 or any(c in _FORBIDDEN for c in s):
+            raise ValueError('name invalid (too long or forbidden character)')
+        return s
+
+    @field_validator('virtual_ip')
+    @classmethod
+    def _vip(cls, v):
+        return _validate_ipv4(v) if v is not None else v
+
+    @field_validator('prefix_length')
+    @classmethod
+    def _prefix(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 32):
+            raise ValueError('prefix_length must be 1-32 (IPv4)')
+        return v
+
+    @field_validator('virtual_router_id')
+    @classmethod
+    def _vrid(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('virtual_router_id must be 1-255')
+        return v
+
+    @field_validator('advert_int')
+    @classmethod
+    def _advert(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 255):
+            raise ValueError('advert_int must be 1-255 seconds')
+        return v
+
+    @field_validator('auth_pass')
+    @classmethod
+    def _auth(cls, v):
+        return _validate_auth_pass(v)
+
+    @model_validator(mode='after')
+    def _members_consistent(self):
+        if self.members is not None:
+            _validate_members(self.members)
+        return self

backend/requirements.txt

@@ -15,4 +15,5 @@ bcrypt>=4.0.1
 slowapi>=0.1.9
 psutil>=5.9.8
 pytz>=2023.3
-josepy>=1.14.0
+josepy>=1.14.0
+pyotp>=2.9.0

backend/routers/acme_diagnostics.py

@@ -79,7 +79,7 @@ async def _load_order(conn, order_id: int) -> dict:
             """
             SELECT id, account_id, status, domains, cluster_ids, error_detail,
                    post_completion_actions, pending_apply_version_name,
-                   wizard_staged_until, created_by
+                   wizard_staged_until, created_by, challenge_type
             FROM letsencrypt_orders
             WHERE id = $1
             """,
@@ -220,6 +220,7 @@ async def run_diagnostics(order_id: int, authorization: str = Header(None)):
                 domains=domains,
                 cluster_ids=cluster_ids,
                 account_id=order["account_id"],
+                challenge_type=(order.get("challenge_type") or "http-01"),
             )
         except Exception as exc:  # noqa: BLE001 — diagnostic boundary
             # run_checks now wraps individual checks, but a top-level
@@ -335,6 +336,7 @@ async def rerun_diagnostic_check(
                 cluster_ids=cluster_ids,
                 account_id=order["account_id"],
                 only=[check_id],
+                challenge_type=(order.get("challenge_type") or "http-01"),
             )
         except Exception as exc:  # noqa: BLE001 — diagnostic boundary
             logger.exception(

backend/routers/agent.py

@@ -29,6 +29,40 @@ AGENT_VERSIONS = {
     "linux": "2.0.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -730,14 +764,15 @@ async def generate_uninstall_script(platform: str, authorization: str = Header(N
     ```
     """
     try:
-        # Validate platform
-        platform_lower = platform.lower()
-        if platform_lower not in ['linux', 'macos']:
-            raise HTTPException(
-                status_code=400, 
-                detail=f"Invalid platform: {platform}. Must be 'linux' or 'macos'"
-            )
-        
+        # Normalize platform to a canonical key (always 'linux' or 'macos').
+        # macOS agents register with platform 'darwin' (from `uname -s`), so the
+        # strict ['linux','macos'] check used to 400 on the UI's uninstall flow
+        # (GET /generate-uninstall-script/darwin). Reuse the same get_platform_key()
+        # helper the install-script generator uses, so darwin/osx/mac and the
+        # linux distro variants all resolve correctly. Backward-compatible:
+        # 'linux'/'macos' still map to themselves.
+        platform_lower = get_platform_key(platform)
+
         # Read uninstall script from agent_scripts directory (same as install scripts)
         import os
         script_filename = f"uninstall-agent-{platform_lower}.sh"
@@ -861,7 +896,25 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
         # Validate cluster access if agent belongs to a cluster
         if agent['cluster_id']:
             await validate_user_cluster_access(current_user['id'], agent['cluster_id'], conn)
-        
+
+        # HA/VIP (Issue #27): block deleting a node that's still a member of an active VIP.
+        # Otherwise the CASCADE would silently drop it from the VIP (breaking the one-MASTER
+        # topology with no signal) and the still-running node would keep advertising the VIP
+        # with no way to be told to tear down (review MED-3). Make the operator remove it
+        # from the VIP first — that stages a clean PENDING change they can apply.
+        try:
+            vip_member = await conn.fetchrow(
+                "SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+                "WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
+        except Exception:  # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
+            vip_member = None
+        if vip_member:
+            await close_database_connection(conn)
+            raise HTTPException(
+                status_code=409,
+                detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
+                        f"VIP on the HA / VIP page (and apply) before deleting the agent."))
+
         await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
         
         await close_database_connection(conn)
@@ -1436,47 +1489,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1749,9 +1788,14 @@ async def agent_heartbeat_by_name(
             heartbeat_data.operating_system, heartbeat_data.kernel_version,
             heartbeat_data.uptime, heartbeat_data.cpu_count, heartbeat_data.memory_total,
             heartbeat_data.disk_space, 
-            # Convert lists to JSON for JSONB columns
-            heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str) else json.dumps(heartbeat_data.network_interfaces or []),
-            heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str) else json.dumps(heartbeat_data.capabilities or []),
+            # Convert lists to JSON for JSONB columns. Don't WIPE network_interfaces/capabilities
+            # when a heartbeat omits them: send NULL so COALESCE keeps the existing value (a bare
+            # `or []` would store "[]" and erase the reported NICs / keepalived_management on every
+            # daemon heartbeat that doesn't include them — issue #27 corporate test).
+            (heartbeat_data.network_interfaces if isinstance(heartbeat_data.network_interfaces, str)
+             else (json.dumps(heartbeat_data.network_interfaces) if heartbeat_data.network_interfaces else None)),
+            (heartbeat_data.capabilities if isinstance(heartbeat_data.capabilities, str)
+             else (json.dumps(heartbeat_data.capabilities) if heartbeat_data.capabilities else None)),
             agent_ip, heartbeat_data.haproxy_status, heartbeat_data.haproxy_version,
             heartbeat_data.applied_config_version, new_status, update_applied_version,
             heartbeat_data.keepalive_state, heartbeat_data.keepalive_ip)
@@ -2180,13 +2224,171 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
             logger.info(f"SSL INCREMENTAL: No certificates updated since {since}")
             
         return response_data
-        
+
     except HTTPException:
         raise
     except Exception as e:
         logger.error(f"SSL certificates retrieval failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
+
+@router.get("/{agent_name}/keepalived-config")
+async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
+    teardown / no-op signal.
+
+    Snapshot-based (T-1): keys off vip_instances.is_active + the member's
+    applied_config_content, NOT the live last_config_status — so a PENDING edit never
+    flips a running member to not_configured (no mid-edit teardown). Auth is MANDATORY (a
+    valid agent key is required), but — like the /config and /ssl-certificates endpoints —
+    the agent API key is a SHARED/global install token, so the config is resolved by the
+    requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
+    (a hard 403 would break every VIP member whose name isn't the one row the shared token
+    resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
+    the agent stays inert; a node with no membership row always gets not_configured.
+    """
+    conn = None
+    try:
+        # Auth FIRST and MANDATORY: a valid agent key is REQUIRED (the response carries the
+        # VRRP secret). The token is shared/global, so resolve by agent_name and only LOG a
+        # name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
+        # unauthenticated caller can't probe which agent names exist.
+        from auth_middleware import validate_agent_api_key
+        if not x_api_key:
+            raise HTTPException(status_code=401, detail="Agent API key required")
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        # The agent API key is a SHARED/global install token (many agent rows per token),
+        # so validate_agent_api_key resolves it to one arbitrary agent for that token. Resolve
+        # the keepalived config strictly by the requested agent_name and treat a token/name
+        # mismatch as an advisory audit log — exactly like the /config and /ssl-certificates
+        # endpoints. (A hard 403 here would reject every VIP member whose name isn't the one
+        # row the shared token happens to return, so the VIP could never converge — review HIGH-1.)
+        if agent_auth['name'] == agent_name:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using its own API key")
+        else:
+            logger.info(f"Agent '{agent_name}' fetching keepalived config using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        # Resolve the agent + its cluster's keepalived.conf path (cluster-driven, like the
+        # HAProxy paths). config_path is returned in EVERY response so the agent knows where
+        # to write/own-marker-check even on not_configured/teardown.
+        agent = await conn.fetchrow("""
+            SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
+                   hc.keepalived_config_path
+            FROM agents a
+            LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
+            WHERE a.name = $1
+        """, agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+        config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
+        if not agent['enabled']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        row = await conn.fetchrow("""
+            SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
+                   v.purge_on_teardown,
+                   m.applied_config_content, m.applied_config_hash
+            FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
+            WHERE m.agent_id = $1
+            -- Active VIP first (an agent has at most one). With NO active VIP, pick the most
+            -- RECENTLY updated inactive membership so a teardown reflects the latest delete
+            -- (incl. its purge flag) — not a stale older VIP the node was once part of.
+            ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
+            LIMIT 1
+        """, agent['id'])
+
+        if not row:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+        if not row['is_active']:
+            # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
+            # the agent still only purges on nodes where IT installed keepalived (install marker).
+            return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
+                    "config_path": config_path, "keepalived": None,
+                    "purge": bool(row['purge_on_teardown'])}
+        if not row['applied_config_content']:
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+
+        from services.keepalived_config import build_haproxy_check_script
+        check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
+        return {
+            "agent_name": agent_name,
+            "status": "available",
+            "config_path": config_path,
+            "keepalived": {
+                "desired_state": "enabled",
+                "install_if_missing": True,
+                "vip_id": row['vip_id'],
+                "vip_name": row['vip_name'],
+                "config_content": row['applied_config_content'],
+                "config_hash": row['applied_config_hash'],
+                "check_script": check_script,
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
+        # Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
+        return {"agent_name": agent_name, "status": "not_configured",
+                "config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+@router.post("/{agent_name}/keepalived-status")
+async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key: Optional[str] = Header(None)):
+    """Issue #27 (v1.7.0) — agent reports the outcome of a keepalived deploy/teardown.
+
+    Auth mirrors config-applied's post-Bulgu-#75 guard (reject a MISSING key — never the
+    `and` short-circuit that accepted no-key requests). The token is a shared/global install
+    token, so the status is recorded strictly for the requested agent_name and a token/name
+    mismatch is an advisory audit log (like /config-applied), not a 403 — review HIGH-1.
+    """
+    conn = None
+    try:
+        from auth_middleware import validate_agent_api_key
+        agent_auth = await validate_agent_api_key(x_api_key)
+        if not x_api_key or not agent_auth:
+            raise HTTPException(status_code=401, detail="Invalid API key")
+        if agent_auth['name'] != agent_name:
+            logger.info(f"Agent '{agent_name}' reporting keepalived status using API key from agent '{agent_auth['name']}'")
+
+        conn = await get_database_connection()
+        agent = await conn.fetchrow("SELECT id FROM agents WHERE name = $1", agent_name)
+        if not agent:
+            raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
+
+        vip_id = status_data.get("vip_id")
+        state = (status_data.get("state") or "").strip()[:24]
+        config_hash = (status_data.get("config_hash") or "")[:64]
+        message = status_data.get("message")
+        if vip_id is None:
+            # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
+            await conn.execute("""
+                UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
+                    last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                WHERE agent_id=$1
+            """, agent['id'], state, message, config_hash)
+        else:
+            await conn.execute("""
+                UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
+                    last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
+                WHERE agent_id=$1 AND vip_id=$2
+            """, agent['id'], int(vip_id), state, message, config_hash)
+        return {"status": "ok"}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"keepalived-status update failed for '{agent_name}': {e}")
+        raise HTTPException(status_code=500, detail="keepalived-status update failed")
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
 @router.get("/script-version")
 async def get_latest_script_version(platform: str = "macos"):
     """Get the latest available agent script version for specified platform"""

backend/routers/auth.py

@@ -9,14 +9,48 @@ from datetime import datetime, timedelta
 # Import database and models
 from database.connection import get_database_connection, close_database_connection
 from models.user import LoginRequest, User, UserCreate, UserUpdate, UserPasswordUpdate
+from models.mfa import MfaVerifyRequest
 from utils.activity_log import log_user_activity
 from auth_middleware import get_current_user_from_token
+from services import mfa_service
 
 # Rate limiting temporarily disabled
 
 router = APIRouter(prefix="/api/auth", tags=["Authentication"])
 logger = logging.getLogger(__name__)
 
+MFA_PENDING_TTL_SECONDS = 300  # 5 minutes — pre-verification challenge lifetime
+MFA_PENDING_MAX_ATTEMPTS = 5  # invalidate token after this many wrong codes
+
+
+async def _fetch_mfa_state(conn, user_id: int):
+    """Return (mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step) or None
+    when the MFA columns aren't yet present (pre-migration deploys).
+    """
+    try:
+        return await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+    except Exception as exc:
+        logger.warning(f"MFA columns not available (assuming disabled): {exc}")
+        return None
+
+
+async def _cleanup_expired_pending_logins(conn, user_id: int) -> None:
+    """Lazy cleanup of expired pending MFA challenges for this user."""
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_logins WHERE user_id = $1 AND expires_at < NOW()",
+            user_id,
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-login cleanup skipped: {exc}")
+
 # Security scheme
 security = HTTPBearer()
 
@@ -96,7 +130,7 @@ async def login(login_request: LoginRequest, request: Request):
                 SELECT id, username, email, password_hash, is_active, role, 
                        created_at, updated_at, last_login_at
                 FROM users 
-                WHERE username = $1
+                WHERE username = $1 AND is_active = TRUE
             """, login_request.username)
         except Exception as schema_error:
             logger.warning(f"Schema error, trying fallback query: {schema_error}")
@@ -106,7 +140,7 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login_at
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
             except Exception as column_error:
                 logger.warning(f"last_login_at column error, trying last_login: {column_error}")
@@ -115,26 +149,67 @@ async def login(login_request: LoginRequest, request: Request):
                     SELECT id, username, email, password_hash, is_active,
                            created_at, updated_at, last_login
                     FROM users 
-                    WHERE username = $1
+                    WHERE username = $1 AND is_active = TRUE
                 """, login_request.username)
         
         if not user:
+            # Covers both "no such user" and "soft-deleted (is_active=FALSE)".
+            # We deliberately return the same generic 401 in either case to
+            # avoid leaking whether an account exists (account enumeration
+            # prevention). Soft-deleted rows are filtered out by the
+            # `AND is_active = TRUE` predicate above.
             await close_database_connection(conn)
             logger.warning(f"Failed login attempt for username: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
-        if not user['is_active']:
-            await close_database_connection(conn)
-            logger.warning(f"Login attempt for inactive account: {login_request.username}")
-            raise HTTPException(status_code=401, detail="Account is deactivated")
-        
+
         # Verify password
         import bcrypt
         if not bcrypt.checkpw(login_request.password.encode('utf-8'), user['password_hash'].encode('utf-8')):
             await close_database_connection(conn)
             logger.warning(f"Wrong password for user: {login_request.username}")
             raise HTTPException(status_code=401, detail="Invalid username or password")
-        
+
+        # Issue #18 — MFA branch (v1.6.0): if the user opted in, defer JWT mint and
+        # last_login update until /api/auth/login/mfa-verify completes.
+        mfa_state = await _fetch_mfa_state(conn, user['id'])
+        if mfa_state and mfa_state.get('mfa_enabled'):
+            await _cleanup_expired_pending_logins(conn, user['id'])
+            challenge_token = mfa_service.generate_challenge_token()
+            try:
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_logins (user_id, challenge_token, expires_at, ip_address)
+                    VALUES ($1, $2, NOW() + ($3 || ' seconds')::interval, $4)
+                    """,
+                    user['id'],
+                    challenge_token,
+                    str(MFA_PENDING_TTL_SECONDS),
+                    str(request.client.host) if request.client else None,
+                )
+            except Exception as exc:
+                await close_database_connection(conn)
+                logger.error(f"Failed to create MFA pending login: {exc}")
+                raise HTTPException(status_code=500, detail="MFA challenge creation failed")
+
+            await close_database_connection(conn)
+
+            await log_user_activity(
+                user_id=user['id'],
+                action='mfa.login.challenge_issued',
+                resource_type='mfa',
+                resource_id=str(user['id']),
+                details={'login_method': 'username_password'},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get('user-agent'),
+            )
+
+            return {
+                "mfa_required": True,
+                "mfa_token": challenge_token,
+                "methods": ["totp", "backup"],
+                "expires_in": MFA_PENDING_TTL_SECONDS,
+            }
+
         # Update last login (try different column names)
         try:
             await conn.execute("""
@@ -242,6 +317,348 @@ async def login(login_request: LoginRequest, request: Request):
         logger.error(f"Login error: {e}")
         raise HTTPException(status_code=500, detail="Login failed")
 
+@router.post(
+    "/login/mfa-verify",
+    summary="MFA Verification (Step 2 of Login)",
+    response_description="JWT access token after successful TOTP/backup verification",
+)
+async def login_mfa_verify(payload: MfaVerifyRequest, request: Request):
+    """
+    # MFA Verification — Step 2 of the two-step login flow
+
+    Submit a 6-digit TOTP code OR an 8-character backup code (with optional dash)
+    together with the ``mfa_token`` returned by ``POST /api/auth/login`` for an
+    MFA-enabled account. On success, returns the same response shape as a
+    non-MFA login (Branch A).
+
+    ## Request Body
+    - **mfa_token**: 64-char challenge token from /login response
+    - **code**: 6 digits (TOTP) or `XXXX-YYYY` (backup)
+
+    ## Error Responses
+    - **401**: Invalid code (attempts counter increments)
+    - **410**: Challenge expired or invalidated (too many wrong attempts)
+    """
+    ip_address = str(request.client.host) if request.client else None
+    user_agent = request.headers.get('user-agent')
+
+    # Outcome captured from the transactional block so we can do JWT mint /
+    # activity logging AFTER commit (no side effects on rollback).
+    success_payload: Optional[dict] = None
+    failure: Optional[dict] = None  # { user_id, attempts, invalidated, reason, http_status, detail }
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        # Round 1 audit fix — wrap the whole verify+update in a single
+        # transaction with row-level locks (FOR UPDATE) so two concurrent
+        # /mfa-verify calls cannot both consume the same TOTP step or the
+        # same pending challenge. We never raise inside the transaction once
+        # we've started mutating the pending row (would rollback the mark);
+        # instead we capture `failure` and raise after commit.
+        async with conn.transaction():
+            pending = await conn.fetchrow(
+                """
+                SELECT id, user_id, attempts, expires_at, used_at
+                  FROM mfa_pending_logins
+                 WHERE challenge_token = $1
+                 FOR UPDATE
+                """,
+                payload.mfa_token,
+            )
+
+            if not pending:
+                failure = {
+                    'user_id': None,
+                    'attempts': 0,
+                    'invalidated': True,
+                    'reason': 'challenge_not_found',
+                    'http_status': 410,
+                    'detail': 'MFA challenge not found or expired',
+                }
+            elif pending['used_at'] is not None:
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_already_used',
+                    'http_status': 410,
+                    'detail': 'MFA challenge already used',
+                }
+            elif pending['expires_at'] and pending['expires_at'] < datetime.utcnow():
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'challenge_expired',
+                    'http_status': 410,
+                    'detail': 'MFA challenge expired',
+                }
+            elif pending['attempts'] >= MFA_PENDING_MAX_ATTEMPTS:
+                await conn.execute(
+                    "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                    pending['id'],
+                )
+                failure = {
+                    'user_id': pending['user_id'],
+                    'attempts': pending['attempts'],
+                    'invalidated': True,
+                    'reason': 'too_many_attempts_pre_check',
+                    'http_status': 410,
+                    'detail': 'MFA challenge invalidated (too many attempts)',
+                }
+
+            if failure is None:
+                # Lock the user row so the atomic TOTP-step bump cannot race a
+                # parallel verify on a different pending challenge for the
+                # same account.
+                user_row = await conn.fetchrow(
+                    """
+                    SELECT id, username, email, role, is_active,
+                           created_at, updated_at, last_login_at,
+                           mfa_secret_encrypted, mfa_last_used_totp_step
+                      FROM users
+                     WHERE id = $1
+                     FOR UPDATE
+                    """,
+                    pending['user_id'],
+                )
+                if not user_row or not user_row['is_active']:
+                    failure = {
+                        'user_id': pending['user_id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': False,
+                        'reason': 'user_unavailable',
+                        'http_status': 401,
+                        'detail': 'User not available',
+                    }
+                elif not user_row['mfa_secret_encrypted']:
+                    failure = {
+                        'user_id': user_row['id'],
+                        'attempts': pending['attempts'],
+                        'invalidated': True,
+                        'reason': 'mfa_not_configured',
+                        'http_status': 410,
+                        'detail': 'MFA not configured for this user',
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(user_row['mfa_secret_encrypted'])
+                    verified_method: Optional[str] = None
+                    codes_remaining: Optional[int] = None
+
+                    if secret_plain:
+                        ok, step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, user_row['mfa_last_used_totp_step']
+                        )
+                        if ok:
+                            # Atomic step bump — refuse if another request already
+                            # consumed this (or a newer) TOTP step.
+                            bumped = await conn.fetchval(
+                                """
+                                UPDATE users
+                                   SET mfa_last_used_totp_step = $1,
+                                       mfa_last_used_at = NOW()
+                                 WHERE id = $2
+                                   AND (mfa_last_used_totp_step IS NULL
+                                        OR mfa_last_used_totp_step < $1)
+                                 RETURNING id
+                                """,
+                                step,
+                                user_row['id'],
+                            )
+                            if bumped:
+                                verified_method = 'totp'
+
+                    if verified_method is None:
+                        # Backup codes — atomic single-use consumption.
+                        rows = await conn.fetch(
+                            """
+                            SELECT id, code_hash FROM mfa_backup_codes
+                             WHERE user_id = $1 AND used_at IS NULL
+                            """,
+                            user_row['id'],
+                        )
+                        for row in rows:
+                            if await mfa_service.check_backup_code(payload.code, row['code_hash']):
+                                consumed_id = await conn.fetchval(
+                                    """
+                                    UPDATE mfa_backup_codes
+                                       SET used_at = NOW()
+                                     WHERE id = $1 AND used_at IS NULL
+                                     RETURNING id
+                                    """,
+                                    row['id'],
+                                )
+                                if consumed_id:
+                                    verified_method = 'backup'
+                                    codes_remaining = await conn.fetchval(
+                                        "SELECT COUNT(*) FROM mfa_backup_codes WHERE user_id = $1 AND used_at IS NULL",
+                                        user_row['id'],
+                                    )
+                                break
+
+                    if verified_method is None:
+                        new_attempts = pending['attempts'] + 1
+                        invalidated = new_attempts >= MFA_PENDING_MAX_ATTEMPTS
+                        await conn.execute(
+                            """
+                            UPDATE mfa_pending_logins
+                               SET attempts = $1,
+                                   used_at = CASE WHEN $2 THEN NOW() ELSE used_at END
+                             WHERE id = $3
+                            """,
+                            new_attempts,
+                            invalidated,
+                            pending['id'],
+                        )
+                        failure = {
+                            'user_id': user_row['id'],
+                            'attempts': new_attempts,
+                            'invalidated': invalidated,
+                            'reason': 'invalid_code',
+                            'http_status': 410 if invalidated else 401,
+                            'detail': 'MFA challenge invalidated (too many attempts)'
+                            if invalidated else 'Invalid MFA code',
+                        }
+                    else:
+                        # Verified — finalize state inside the transaction so a
+                        # concurrent verify sees used_at on retry.
+                        await conn.execute(
+                            "UPDATE mfa_pending_logins SET used_at = NOW() WHERE id = $1",
+                            pending['id'],
+                        )
+                        if verified_method != 'totp':
+                            await conn.execute(
+                                "UPDATE users SET mfa_last_used_at = NOW() WHERE id = $1",
+                                user_row['id'],
+                            )
+                        try:
+                            await conn.execute(
+                                "UPDATE users SET last_login_at = CURRENT_TIMESTAMP WHERE id = $1",
+                                user_row['id'],
+                            )
+                        except Exception as exc:
+                            logger.warning(f"last_login_at update failed (continuing): {exc}")
+
+                        user_roles = await conn.fetch(
+                            """
+                            SELECT r.id, r.name, r.display_name, r.permissions
+                              FROM user_roles ur
+                              JOIN roles r ON ur.role_id = r.id
+                             WHERE ur.user_id = $1 AND ur.is_active = TRUE AND r.is_active = TRUE
+                            """,
+                            user_row['id'],
+                        )
+
+                        permissions: dict = {}
+                        roles_list: list = []
+                        for role_row in user_roles:
+                            roles_list.append({
+                                'id': role_row['id'],
+                                'name': role_row['name'],
+                                'display_name': role_row['display_name'],
+                            })
+                            role_permissions = role_row['permissions']
+                            if isinstance(role_permissions, str):
+                                import json
+                                role_permissions = json.loads(role_permissions)
+                            if role_permissions:
+                                for perm in role_permissions:
+                                    if '.' in perm:
+                                        resource, action = perm.split('.', 1)
+                                        if resource not in permissions:
+                                            permissions[resource] = {}
+                                        permissions[resource][action] = True
+
+                        success_payload = {
+                            'user': dict(user_row),
+                            'roles_list': roles_list,
+                            'permissions': permissions,
+                            'method': verified_method,
+                            'codes_remaining': codes_remaining,
+                        }
+
+        # ------------------------------------------------------------------
+        # Transaction has committed. Side-effects (JWT mint, audit log) below.
+        # ------------------------------------------------------------------
+        await close_database_connection(conn)
+        conn = None
+
+        if failure is not None:
+            if failure['user_id'] is not None:
+                await log_user_activity(
+                    user_id=failure['user_id'],
+                    action='mfa.login.failed',
+                    resource_type='mfa',
+                    resource_id=str(failure['user_id']),
+                    details={
+                        'reason': failure['reason'],
+                        'attempts': failure['attempts'],
+                        'invalidated': failure['invalidated'],
+                    },
+                    ip_address=ip_address,
+                    user_agent=user_agent,
+                )
+            raise HTTPException(status_code=failure['http_status'], detail=failure['detail'])
+
+        # Success path
+        assert success_payload is not None  # for type checkers; transaction guarantees this
+        user_row = success_payload['user']
+
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        token_payload = {
+            "user_id": user_row['id'],
+            "username": user_row['username'],
+            "email": user_row['email'],
+            "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+            "exp": datetime.utcnow() + timedelta(hours=24),
+        }
+        token = jwt.encode(token_payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+        await log_user_activity(
+            user_id=user_row['id'],
+            action='mfa.login.success',
+            resource_type='mfa',
+            resource_id=str(user_row['id']),
+            details={
+                'method': success_payload['method'],
+                'codes_remaining': success_payload['codes_remaining']
+                if success_payload['method'] == 'backup' else None,
+            },
+            ip_address=ip_address,
+            user_agent=user_agent,
+        )
+
+        return {
+            "access_token": token,
+            "token_type": "bearer",
+            "expires_in": 86400,
+            "user": {
+                "id": user_row['id'],
+                "username": user_row['username'],
+                "email": user_row['email'],
+                "role": user_row['role'] if 'role' in user_row.keys() else 'admin',
+                "is_active": user_row['is_active'],
+                "created_at": user_row['created_at'].isoformat() if user_row.get('created_at') else None,
+                "last_login_at": datetime.utcnow().isoformat(),
+            },
+            "roles": success_payload['roles_list'],
+            "permissions": success_payload['permissions'],
+        }
+
+    except HTTPException:
+        raise
+    except Exception as exc:
+        logger.error(f"MFA verify error: {exc}")
+        raise HTTPException(status_code=500, detail="MFA verification failed")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
 @router.post("/logout", summary="User Logout", response_description="Logout confirmation")
 async def logout(request: Request, authorization: str = Header(None)):
     """

backend/routers/backend.py

@@ -333,31 +333,38 @@ async def get_backends(
                 """)
         
         result = []
+        # Issue #24: servers must honor include_inactive exactly like the backend
+        # queries above. Pre-fix these sub-queries hardcoded `is_active = TRUE`
+        # (added in f34a6ee to hide soft-deleted entities), so a server toggled
+        # OFF (is_active=false) vanished from the UI with no way to reactivate it.
+        # Default callers (include_inactive=false) keep the is_active filter →
+        # byte-identical behavior; include_inactive=true now also returns inactive
+        # (disabled / soft-deleted) servers. last_config_status is selected so the
+        # frontend can tell DISABLED (re-enableable) from DELETION (pending delete).
+        server_active_filter = "" if include_inactive else "AND is_active = TRUE"
         for backend in backends:
-            # Get servers for this backend with cluster_id (ONLY show active servers)
-            # CRITICAL FIX: Add is_active = TRUE filter to prevent soft-deleted servers from appearing
             if cluster_id:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND cluster_id = $2 AND is_active = TRUE ORDER BY server_name
+                    FROM backend_servers
+                    WHERE backend_name = $1 AND cluster_id = $2 {server_active_filter} ORDER BY server_name
                 """, backend["name"], cluster_id)
             else:
-                servers = await conn.fetch("""
+                servers = await conn.fetch(f"""
                     SELECT id, server_name, server_address, server_port, weight, maxconn,
                            check_enabled, check_port, backup_server, ssl_enabled, ssl_verify, ssl_certificate_id,
                            ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers,
                            cookie_value, inter, fall, rise,
-                           is_active, cluster_id,
+                           is_active, cluster_id, last_config_status,
                            haproxy_status, haproxy_status_updated_at, backend_name
-                    FROM backend_servers 
-                    WHERE backend_name = $1 AND is_active = TRUE ORDER BY server_name
-                """, backend["name"]) 
+                    FROM backend_servers
+                    WHERE backend_name = $1 {server_active_filter} ORDER BY server_name
+                """, backend["name"])
             
             # Prepare server list with real-time HAProxy status from agents
             server_list = []
@@ -413,6 +420,7 @@ async def get_backends(
                     "fall": s.get("fall"),
                     "rise": s.get("rise"),
                     "is_active": s["is_active"],
+                    "last_config_status": s.get("last_config_status") or "APPLIED",  # Issue #24: lets UI distinguish DISABLED (re-enableable) from DELETION
                     "status": server_status,
                     "status_age_minutes": status_age_minutes,
                     "last_status_update": s.get("haproxy_status_updated_at").isoformat().replace('+00:00', 'Z') if s.get("haproxy_status_updated_at") else None,
@@ -1909,12 +1917,12 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
 
         conn = await get_database_connection()
         
-        # Get server info
+        # Get server info. Issue #24: fetch the FULL row (not just 5 columns) so we
+        # can snapshot the pre-toggle state for reject-rollback (see config version below).
         server = await conn.fetchrow("""
-            SELECT id, server_name, backend_name, is_active, cluster_id
-            FROM backend_servers WHERE id = $1
+            SELECT * FROM backend_servers WHERE id = $1
         """, server_id)
-        
+
         if not server:
             await close_database_connection(conn)
             raise HTTPException(status_code=404, detail="Server not found")
@@ -1975,22 +1983,45 @@ async def toggle_server(server_id: int, request: Request, authorization: str = H
                 
                 # Generate new HAProxy config (after database commit)
                 config_content = await generate_haproxy_config_for_cluster(cluster_id)
-                
+
                 # Create new config version
                 config_hash = hashlib.sha256(config_content.encode()).hexdigest()
                 version_name = f"server-{server_id}-toggle-{int(time.time())}"
-                
+
                 # Get system admin user ID for created_by
                 conn2 = await get_database_connection()
                 admin_user_id = await conn2.fetchval("SELECT id FROM users WHERE username = 'admin' LIMIT 1") or 1
-                
+
+                # Issue #24: persist an entity snapshot so a Reject of this toggle
+                # rolls back is_active to its pre-toggle value. Pre-fix the toggle's
+                # config version carried NO metadata, so reject only reset
+                # last_config_status and the server stayed disabled (out of sync with
+                # the still-active live config). Mirrors the server-edit snapshot path;
+                # reject's rollback_entity_from_snapshot('server') restores is_active.
+                import json
+                from utils.entity_snapshot import save_entity_snapshot
+                entity_snapshot_metadata = await save_entity_snapshot(
+                    conn=conn2,
+                    entity_type="server",
+                    entity_id=server_id,
+                    old_values=dict(server),            # full pre-toggle row
+                    new_values={"is_active": new_status},
+                    operation="UPDATE",
+                )
+                old_config = await conn2.fetchval("""
+                    SELECT config_content FROM config_versions
+                    WHERE cluster_id = $1 AND status = 'APPLIED' AND is_active = TRUE
+                    ORDER BY created_at DESC LIMIT 1
+                """, cluster_id)
+                metadata = {"pre_apply_snapshot": old_config or "", **entity_snapshot_metadata}
+
                 # Create PENDING config version
                 config_version_id = await conn2.fetchval("""
-                    INSERT INTO config_versions 
-                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status)
-                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING')
+                    INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                    VALUES ($1, $2, $3, $4, $5, FALSE, 'PENDING', $6)
                     RETURNING id
-                """, cluster_id, version_name, config_content, config_hash, admin_user_id)
+                """, cluster_id, version_name, config_content, config_hash, admin_user_id, json.dumps(metadata))
                 
                 logger.error(f"SERVER TOGGLE DEBUG: Created PENDING config version {version_name} for cluster {cluster_id}")
                 

backend/routers/cluster.py

@@ -292,12 +292,14 @@ async def create_cluster(cluster: HAProxyClusterCreate, authorization: str = Hea
         
         # Create cluster
         cluster_id = await conn.fetchval("""
-            INSERT INTO haproxy_clusters (name, description, connection_type, is_active, 
-                                        stats_socket_path, haproxy_config_path, haproxy_bin_path, pool_id)
-            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7)
+            INSERT INTO haproxy_clusters (name, description, connection_type, is_active,
+                                        stats_socket_path, haproxy_config_path, haproxy_bin_path,
+                                        keepalived_config_path, pool_id)
+            VALUES ($1, $2, $3, TRUE, $4, $5, $6, $7, $8)
             RETURNING id
         """, cluster.name, cluster.description, cluster.connection_type,
-            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path, cluster.pool_id)
+            cluster.stats_socket_path, cluster.haproxy_config_path, cluster.haproxy_bin_path,
+            cluster.keepalived_config_path, cluster.pool_id)
         
         await close_database_connection(conn)
         
@@ -436,7 +438,12 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
             update_fields.append(f"haproxy_bin_path = ${param_counter}")
             update_values.append(cluster.haproxy_bin_path)
             param_counter += 1
-            
+
+        if cluster.keepalived_config_path is not None:
+            update_fields.append(f"keepalived_config_path = ${param_counter}")
+            update_values.append(cluster.keepalived_config_path)
+            param_counter += 1
+
         if cluster.pool_id is not None:
             update_fields.append(f"pool_id = ${param_counter}")
             update_values.append(cluster.pool_id)
@@ -517,7 +524,7 @@ async def update_cluster(cluster_id: int, cluster: HAProxyClusterUpdate, authori
 
 
 @router.get("/{cluster_id}", summary="Get Cluster by ID", response_description="Cluster details")
-async def get_cluster(cluster_id: int, authorization: str = Header(None)):
+async def get_cluster(cluster_id: int, authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get Specific HAProxy Cluster
     
@@ -528,8 +535,12 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters/1" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters/1" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -554,20 +565,30 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
     - **404**: Cluster not found
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 final convergence): authenticate
-        # the caller before fetching cluster topology by ID. Pre-fix
-        # this sibling of GET /api/clusters was anonymous, so an
-        # attacker could iterate cluster IDs to enumerate the same
-        # info (stats socket, paths, ACME flags, pool identity) the
-        # list endpoint just locked down. Closes the asymmetry.
+    # R18c audit fix (round 6 final convergence): authenticate the caller
+    # before fetching cluster topology by ID. Pre-fix this sibling of
+    # GET /api/clusters was anonymous, so an attacker could iterate cluster
+    # IDs to enumerate the same info (stats socket, paths, ACME flags, pool
+    # identity) the list endpoint just locked down.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         cluster = await conn.fetchrow("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name
             FROM haproxy_clusters c
@@ -591,6 +612,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
             "stats_socket_path": cluster["stats_socket_path"],
             "haproxy_config_path": cluster["haproxy_config_path"],
             "haproxy_bin_path": cluster["haproxy_bin_path"],
+            "keepalived_config_path": cluster.get("keepalived_config_path"),
             "pool_id": cluster["pool_id"],
             "pool_name": cluster["pool_name"],
             "acme_enabled": cluster.get("acme_enabled", False),
@@ -602,7 +624,7 @@ async def get_cluster(cluster_id: int, authorization: str = Header(None)):
         raise HTTPException(status_code=500, detail=str(e))
 
 @router.get("", summary="Get All Clusters", response_description="List of all clusters")
-async def get_clusters(authorization: str = Header(None)):
+async def get_clusters(authorization: str = Header(None), x_api_key: Optional[str] = Header(None)):
     """
     # Get All HAProxy Clusters
     
@@ -610,8 +632,12 @@ async def get_clusters(authorization: str = Header(None)):
     
     ## Example Request
     ```bash
+    # User (UI) authentication:
     curl -X GET "{BASE_URL}/api/clusters" \\
       -H "Authorization: Bearer eyJhbGciOiJIUz..."
+    # Agent authentication (agent token in X-API-Key):
+    curl -X GET "{BASE_URL}/api/clusters" \\
+      -H "X-API-Key: hap_..."
     ```
     
     ## Example Response
@@ -662,23 +688,34 @@ async def get_clusters(authorization: str = Header(None)):
     ## Error Responses
     - **500**: Server error
     """
-    try:
-        # R18c audit fix (round 6 #3 — KRITIK info leak): require an
-        # authenticated caller. Pre-fix the endpoint accepted
-        # anonymous GETs and returned cluster topology including
-        # internal HAProxy paths (stats socket, config path, bin
-        # path), pool ids, ACME flags, and agent counts. This is
-        # both reconnaissance for an attacker and the spine of the
-        # cluster-scoped RBAC the rest of the platform builds on,
-        # so guarding it at the read layer is essential after R18c
-        # round 5's roster + role guards.
+    # R18c audit fix (round 6 #3 — KRITIK info leak): require an authenticated
+    # caller. Pre-fix the endpoint accepted anonymous GETs and returned cluster
+    # topology including internal HAProxy paths (stats socket, config path, bin
+    # path), pool ids, ACME flags, and agent counts. This is both reconnaissance
+    # for an attacker and the spine of the cluster-scoped RBAC the rest of the
+    # platform builds on, so guarding it at the read layer is essential.
+    # Issue #22: agents send their token in the X-API-Key header (not a user
+    # JWT), so accept either credential — mirrors the dual-auth on
+    # POST /api/agents/generate-install-script. Anonymous is still rejected.
+    # (Guard kept OUTSIDE the try below: get_clusters' broad `except Exception`
+    # re-wraps raised HTTPExceptions into 500, which produced the "500 - 401"
+    # in the issue log; raising here yields a clean 401.)
+    if authorization:
         from auth_middleware import get_current_user_from_token
         await get_current_user_from_token(authorization)
+    elif x_api_key:
+        from auth_middleware import validate_agent_api_key
+        if not await validate_agent_api_key(x_api_key):
+            raise HTTPException(status_code=401, detail="Invalid agent API key")
+    else:
+        raise HTTPException(status_code=401, detail="Authorization header or X-API-Key required")
+    try:
         conn = await get_database_connection()
         
         clusters = await conn.fetch("""
-            SELECT c.id, c.name, c.description, c.connection_type, c.is_active, 
+            SELECT c.id, c.name, c.description, c.connection_type, c.is_active,
                    c.created_at, c.stats_socket_path, c.haproxy_config_path, c.haproxy_bin_path,
+                   c.keepalived_config_path,
                    c.pool_id, c.is_default, c.acme_enabled, c.acme_backend_url,
                    p.name as pool_name,
                    COALESCE(agent_counts.total_agents, 0) as total_agents,
@@ -737,6 +774,7 @@ async def get_clusters(authorization: str = Header(None)):
                 "stats_socket_path": cluster["stats_socket_path"],
                 "haproxy_config_path": cluster["haproxy_config_path"],
                 "haproxy_bin_path": cluster["haproxy_bin_path"],
+                "keepalived_config_path": cluster.get("keepalived_config_path"),
                 "pool_id": cluster["pool_id"],
                 "pool_name": cluster["pool_name"],
                 "acme_enabled": cluster.get("acme_enabled", False),
@@ -1313,6 +1351,8 @@ async def list_cluster_config_versions(cluster_id: int, authorization: str = Hea
                 version_type = "WAF Rule"
             elif "ssl-" in version['version_name']:
                 version_type = "SSL Certificate"
+            elif "vip-" in version['version_name']:
+                version_type = "HA / VIP"
             
             # Parse validation error if present
             validation_error = version.get("validation_error")
@@ -1417,10 +1457,14 @@ async def apply_pending_changes(
         # Users will handle configuration completeness through the centralized Apply Management page
         
         # Get all pending config versions for this cluster
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP apply/reject endpoints
+        # (keepalived is not part of haproxy.cfg). Exclude them so a generic cluster apply
+        # from any entity page never marks a VIP version APPLIED without enacting it. This
+        # is a no-op for every non-VIP cluster (no vip-* rows exist).
         pending_versions = await conn.fetch("""
             SELECT id, version_name, created_at, config_content, checksum, metadata
-            FROM config_versions 
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            FROM config_versions
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
             ORDER BY created_at ASC
         """, cluster_id)
         
@@ -2539,6 +2583,92 @@ async def get_config_version_diff(cluster_id: int, version_id: int, authorizatio
                     }
                 }
         
+        # HA/VIP (Issue #27): vip-{id}-{action} versions show the generated keepalived.conf
+        # each member node will deploy as the change content (VRRP secret masked). Mirrors
+        # the ssl-* special case above so VIP uses the STANDARD View Change diff modal.
+        vip_match = re.search(r'vip-(\d+)-(create|update|delete)', current_version['version_name'])
+        if vip_match:
+            vip_id = int(vip_match.group(1))
+            vip_action = vip_match.group(2)
+            rendered, vip_meta = None, None
+            old_content = ""
+            try:
+                from routers.vip import render_vip_config_masked
+                rendered, vip_meta = await render_vip_config_masked(conn, vip_id)
+                # For a create/update diff, fetch the PREVIOUS applied vip-* config for this VIP
+                # (the last-deployed keepalived.conf) so an EDIT shows ONLY the changed lines
+                # instead of the whole config as "added". Both sides are already secret-masked.
+                if vip_action != "delete":
+                    prev_applied = await conn.fetchrow(
+                        "SELECT config_content FROM config_versions WHERE version_name LIKE $1 "
+                        "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1",
+                        f"vip-{vip_id}-%", cluster_id, current_version['id'])
+                    if prev_applied and prev_applied['config_content']:
+                        old_content = prev_applied['config_content']
+            except Exception as vip_err:
+                logger.warning(f"VIP DIFF: render failed for vip {vip_id}: {vip_err}")
+            await close_database_connection(conn)
+
+            changes = []
+            line_number = 1
+            if vip_action == "delete" or rendered is None:
+                title = (vip_meta or {}).get("name") or f"VIP {vip_id}"
+                for line in [f"# HA/VIP change: {title}",
+                             "# keepalived will be stopped and the virtual IP released on each member node."
+                             if vip_action == "delete" else
+                             "# (configuration is not available to render yet)"]:
+                    changes.append({"type": "context", "line": line, "line_number": line_number})
+                    line_number += 1
+                summary = {"added": 0, "removed": 1 if vip_action == "delete" else 0,
+                           "total_changes": 1 if vip_action == "delete" else 0}
+            else:
+                # Real line diff. Match the STANDARD haproxy diff format: the line is stored
+                # WITHOUT a +/- prefix (the UI adds it from `type` — the old `+ {line}` here
+                # caused the doubled "+ +"). A CREATE (no previous applied config) shows
+                # everything as added; an UPDATE shows ONLY the lines that actually changed.
+                import difflib
+                new_content = current_version['config_content'] or rendered or ""
+                added_count = 0
+                removed_count = 0
+                line_number = 0
+                if not old_content:
+                    for i, l in enumerate(new_content.split('\n')):
+                        changes.append({"type": "added", "line": l, "line_number": i + 1})
+                        added_count += 1
+                else:
+                    for dl in difflib.unified_diff(old_content.split('\n'), new_content.split('\n'),
+                                                   lineterm='', n=3):
+                        if dl.startswith('@@'):
+                            mm = re.search(r'@@ -(\d+),?\d* \+(\d+),?\d* @@', dl)
+                            if mm:
+                                line_number = int(mm.group(2))
+                            continue
+                        if dl.startswith('---') or dl.startswith('+++'):
+                            continue
+                        if dl.startswith('+'):
+                            changes.append({"type": "added", "line": dl[1:], "line_number": line_number})
+                            added_count += 1
+                            line_number += 1
+                        elif dl.startswith('-'):
+                            changes.append({"type": "removed", "line": dl[1:], "line_number": line_number})
+                            removed_count += 1
+                        elif dl.startswith(' '):
+                            changes.append({"type": "context", "line": dl[1:], "line_number": line_number})
+                            line_number += 1
+                summary = {"added": added_count, "removed": removed_count,
+                           "total_changes": added_count + removed_count}
+
+            return {
+                "current_version": {
+                    "id": current_version['id'],
+                    "version_name": current_version['version_name'],
+                    "created_at": current_version['created_at'].isoformat().replace('+00:00', 'Z')
+                },
+                "previous_version": None,
+                "changes": changes,
+                "summary": summary,
+            }
+
         # Check if current version has config content
         if not current_version['config_content']:
             # Special handling for restore versions - they might not have content yet
@@ -4226,6 +4356,20 @@ async def undo_reject_config_version(
                 ),
             )
 
+        # HA/VIP (Issue #27): vip-* versions are owned by the VIP entity, so undo is handled
+        # by the VIP router — it re-stages the rejected change as PENDING from the version's
+        # captured pending_state (reactivating the VIP if a rejected create soft-deleted it,
+        # or re-applying a rejected edit). Returns an error string if it can't (e.g. the
+        # name/address/VRID was reused since reject) -> surface a clean 409.
+        if version_name.startswith('vip-'):
+            from routers.vip import restore_vip_from_rejected_version
+            err = await restore_vip_from_rejected_version(conn, version_id)
+            await close_database_connection(conn)
+            if err:
+                raise HTTPException(status_code=409, detail=f"Cannot undo this VIP change — {err}.")
+            return {"message": "VIP change restored to PENDING — review and Apply it from Apply Management",
+                    "version_name": version_name}
+
         async with conn.transaction():
             # Mark the version as PENDING again
             await conn.execute("""
@@ -4902,9 +5046,17 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         await validate_user_cluster_access(current_user['id'], cluster_id, conn)
 
         # Get all pending config versions for this cluster (CRITICAL: Include metadata for rollback!)
+        # HA/VIP (Issue #27): exclude vip-* versions — they are rejected/reverted by the
+        # VIP reject endpoint (which restores keepalived state), not the generic rollback.
+        # ORDER BY created_at ASC: the rollback loop dedups per entity and keeps the FIRST-processed
+        # snapshot, so the OLDEST snapshot must win — its old_values hold the true pre-change state.
+        # Critical when one entity has multiple pending versions (e.g. cluster ACME enable->disable->enable):
+        # rolling back to the oldest restores the original acme_enabled. (Matches the apply SELECT, which
+        # already orders created_at ASC.)
         pending_versions = await conn.fetch("""
             SELECT id, version_name, metadata FROM config_versions
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
+            ORDER BY created_at ASC
         """, cluster_id)
         
         # CRITICAL FIX: Detect and clean orphan config versions
@@ -5116,11 +5268,12 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         )
         
         # Mark all pending config versions as REJECTED (don't delete them)
+        # HA/VIP (Issue #27): leave vip-* versions to the VIP reject endpoint.
         rejected_count = len(pending_versions)
         await conn.execute("""
-            UPDATE config_versions 
+            UPDATE config_versions
             SET status = 'REJECTED'
-            WHERE cluster_id = $1 AND status = 'PENDING'
+            WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
         """, cluster_id)
 
         # Update WAF rules status to APPLIED (rolled back)

backend/routers/letsencrypt.py

@@ -1,6 +1,7 @@
 from fastapi import APIRouter, HTTPException, Header
-from pydantic import BaseModel, Field, field_validator
-from typing import Optional, List
+from pydantic import BaseModel, Field, field_validator, model_validator
+from typing import Optional, List, Dict
+import base64
 import json
 import logging
 import re
@@ -10,6 +11,40 @@ from datetime import datetime
 from database.connection import get_database_connection, close_database_connection
 from services.acme_service import acme_service
 from services.haproxy_config import generate_haproxy_config_for_cluster
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import encrypt_dns_credentials, decrypt_dns_credentials
+
+# Issue #35: DNS-01 challenge methods.
+_CHALLENGE_TYPES = ("http-01", "dns-01")
+
+
+async def _dns01_enabled() -> bool:
+    """Global kill-switch (system_settings acme.dns01_enabled, default False). Read via the ACME
+    settings dict so non-admins never need the admin-only /api/settings/acme endpoint."""
+    try:
+        settings = await acme_service._get_settings()
+        val = settings.get('dns01_enabled')
+        if isinstance(val, str):
+            return val.strip().lower() in ('1', 'true', 'yes', 'on')
+        return bool(val)
+    except Exception:
+        return False
+
+
+# Per-user sliding-window rate limit for the manual dns-confirm action (soft anti-abuse so a user
+# can't spam the CA via the confirm button). Per-process; sufficient for a manual UI action.
+_DNS_CONFIRM_RL: Dict[int, list] = {}
+_DNS_CONFIRM_LIMIT = 5
+_DNS_CONFIRM_WINDOW = 60.0
+
+
+async def _enforce_dns_confirm_rate_limit(user_id: int) -> None:
+    now = time.time()
+    bucket = [t for t in _DNS_CONFIRM_RL.get(user_id, []) if now - t < _DNS_CONFIRM_WINDOW]
+    if len(bucket) >= _DNS_CONFIRM_LIMIT:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded: dns-confirm allowed 5 requests per minute")
+    bucket.append(now)
+    _DNS_CONFIRM_RL[user_id] = bucket
 
 logger = logging.getLogger(__name__)
 
@@ -25,8 +60,55 @@ class AccountCreate(BaseModel):
     email: str
     directory_url: Optional[str] = None
     tos_agreed: bool = True
-    eab_kid: Optional[str] = None
-    eab_hmac_key: Optional[str] = None
+    # EAB (External Account Binding) for CAs that require it (ZeroSSL, Google). The KID is opaque
+    # (bound only); the HMAC key must be base64url so newAccount's _b64url_decode won't raise a
+    # cryptic binascii error (a common copy mistake is standard-base64 '+'/'/' vs urlsafe '-'/'_').
+    eab_kid: Optional[str] = Field(default=None, max_length=256)
+    eab_hmac_key: Optional[str] = Field(default=None, max_length=512)
+    # Issue #35: per-account default challenge method + DNS provider (for dns-01).
+    challenge_type: str = "http-01"
+    dns_provider: Optional[str] = None
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @field_validator('eab_hmac_key')
+    @classmethod
+    def _validate_eab_hmac_key(cls, v):
+        if not v:
+            return v
+        try:
+            base64.urlsafe_b64decode(v + '=' * (-len(v) % 4))
+        except Exception:
+            raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
+        return v
+
+    @model_validator(mode='after')
+    def _require_provider_for_dns01(self):
+        if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
+            raise ValueError("dns_provider is required when challenge_type is 'dns-01'")
+        return self
+
+
+class DnsCredentialsUpsert(BaseModel):
+    dns_provider: str = Field(..., min_length=1, max_length=50)
+    credentials: Dict[str, str] = Field(default_factory=dict)
+
+    @field_validator('credentials')
+    @classmethod
+    def _validate_credentials(cls, v):
+        if len(v) > 20:
+            raise ValueError("Too many credential fields")
+        for key, val in v.items():
+            if not isinstance(key, str) or not re.match(r'^[a-zA-Z0-9_]{1,50}$', key):
+                raise ValueError(f"Invalid credential field name: {key!r}")
+            if not isinstance(val, str) or len(val) > 4000:
+                raise ValueError(f"Credential value for {key!r} is missing or too long")
+        return v
 
 
 class CertificateRequest(BaseModel):
@@ -38,6 +120,8 @@ class CertificateRequest(BaseModel):
     account_id: Optional[int] = None
     cluster_ids: List[int] = Field(default_factory=list)
     auto_renew: bool = True
+    # Issue #35: optional override; when None the account's default method is used.
+    challenge_type: Optional[str] = None
 
     @field_validator('domains')
     @classmethod
@@ -54,7 +138,30 @@ class CertificateRequest(BaseModel):
             if not _DOMAIN_REGEX.match(d_norm):
                 raise ValueError(f"Invalid domain format: '{d}'")
             normalized.append(d_norm)
-        return normalized
+        # De-duplicate (case/whitespace variants normalize to the same value) while preserving order,
+        # so we don't submit a redundant SAN to the CA or render duplicate-keyed tags in the UI.
+        return list(dict.fromkeys(normalized))
+
+    @field_validator('challenge_type')
+    @classmethod
+    def _validate_challenge_type(cls, v):
+        if v is not None and v not in _CHALLENGE_TYPES:
+            raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
+        return v
+
+    @model_validator(mode='after')
+    def _wildcard_requires_dns01(self):
+        # Static cross-field guard: a wildcard SAN can ONLY be issued via dns-01 (the CA rejects
+        # wildcard over http-01). The runtime dns01_enabled gate + provider resolution happen in the
+        # endpoint (validators can't do async/DB). When challenge_type is None here, the effective
+        # method is resolved from the account in the endpoint, which re-checks this.
+        if any((d or '').startswith('*.') for d in (self.domains or [])):
+            # Only reject when the caller EXPLICITLY chose a non-dns-01 method. When challenge_type is
+            # None, the effective method is resolved from the account in the endpoint, which re-checks
+            # wildcard-requires-dns-01 — so account-default dns-01 inheritance still works for wildcards.
+            if self.challenge_type is not None and self.challenge_type != 'dns-01':
+                raise ValueError("Wildcard certificates require challenge_type 'dns-01'")
+        return self
 
 
 # --- Account management ---
@@ -77,7 +184,7 @@ async def list_accounts(authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         rows = await conn.fetch(
-            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at FROM letsencrypt_accounts ORDER BY id"
+            "SELECT id, email, directory_url, account_url, status, tos_agreed, eab_kid, created_at, updated_at, challenge_type, dns_provider FROM letsencrypt_accounts ORDER BY id"
         )
         return [dict(r) for r in rows]
     finally:
@@ -107,16 +214,36 @@ async def create_account(body: AccountCreate, authorization: str = Header(None))
         eab_kid = body.eab_kid or settings.get('eab_kid', '') or None
         eab_hmac_key = body.eab_hmac_key or settings.get('eab_hmac_key', '') or None
 
+        # Issue #35: a dns-01 account must name a supported DNS provider.
+        if body.challenge_type == 'dns-01':
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((body.dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {body.dns_provider}")
+
         result = await acme_service.register_account(
             email=body.email,
             directory_url=directory_url,
             tos_agreed=body.tos_agreed,
             eab_kid=eab_kid,
             eab_hmac_key=eab_hmac_key,
+            challenge_type=body.challenge_type,
+            dns_provider=(body.dns_provider or None),
         )
         return result
+    except HTTPException:
+        # Preserve deliberate status codes (e.g. 409 DNS-01 disabled, 422 unsupported provider) —
+        # the broad except below would otherwise downgrade them all to 400.
+        raise
     except Exception as e:
         logger.error(f"ACME account registration failed: {e}")
+        # Humanize the common EAB-required failure (ZeroSSL/Google). The ACME error propagates as a
+        # string ("Account registration failed: {<dict>}"), so match the URN substring in str(e).
+        if 'externalaccountrequired' in str(e).lower():
+            raise HTTPException(status_code=400, detail=(
+                "This CA requires External Account Binding (EAB). Enter the EAB Key ID and HMAC Key "
+                "from your ZeroSSL/Google account and retry."
+            ))
         raise HTTPException(status_code=400, detail=str(e))
 
 
@@ -186,6 +313,147 @@ async def remove_account(account_id: int, authorization: str = Header(None)):
         await close_database_connection(conn)
 
 
+# --- Issue #35: DNS-01 providers + per-account DNS credentials ---
+
+@router.get("/dns-providers")
+async def get_dns_providers(authorization: str = Header(None)):
+    """List supported DNS providers + their credential-field schema (for the UI). Also returns the
+    global dns01_enabled gate so a non-admin cert UI can read it without the admin-only settings API.
+    Authenticated (any user); not admin-only."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    return {"dns01_enabled": await _dns01_enabled(), "providers": list_providers()}
+
+
+@router.get("/accounts/{account_id}/dns-credentials")
+async def get_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Masked metadata only — provider + which credential fields are set + updated_at. NEVER returns
+    the ciphertext or any plaintext token. Read-only for any authenticated user (matches list_accounts)."""
+    from auth_middleware import get_current_user_from_token
+    await get_current_user_from_token(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            "SELECT dns_provider, credentials_encrypted, updated_at FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+            account_id,
+        )
+        if not row:
+            return {"configured": False, "dns_provider": None, "credential_fields_present": [], "updated_at": None}
+        present = []
+        decrypted = decrypt_dns_credentials(row["credentials_encrypted"])
+        if isinstance(decrypted, dict):
+            present = sorted(decrypted.keys())
+        return {
+            "configured": True,
+            "dns_provider": row["dns_provider"],
+            "credential_fields_present": present,
+            "updated_at": row["updated_at"],
+        }
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/accounts/{account_id}/dns-credentials")
+async def upsert_dns_credentials(account_id: int, body: DnsCredentialsUpsert, authorization: str = Header(None)):
+    """Store (encrypted) DNS provider credentials for an account. Admin-only. Verifies the
+    credentials against the provider BEFORE persisting; returns a sanitized result (never the token)."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    provider_name = body.dns_provider.strip()
+    if not is_supported(provider_name):
+        raise HTTPException(status_code=422, detail=f"Unsupported DNS provider: {provider_name}")
+
+    conn = await get_database_connection()
+    try:
+        exists = await conn.fetchval("SELECT 1 FROM letsencrypt_accounts WHERE id = $1", account_id)
+        if not exists:
+            raise HTTPException(status_code=404, detail="ACME account not found")
+
+        # Verify credentials synchronously; only persist on success. The detail is user-safe.
+        try:
+            provider = get_provider(provider_name, dict(body.credentials))
+            verify = await provider.verify_credentials()
+        except DnsProviderError as exc:
+            raise HTTPException(status_code=422, detail=str(exc))
+        except HTTPException:
+            raise
+        except Exception:
+            # Defensive: never let a provider-internal exception string (which could echo creds in a
+            # future provider) reach the client. Always a sanitized 422.
+            raise HTTPException(status_code=422, detail="DNS provider credential verification failed")
+        if not verify.get("ok"):
+            raise HTTPException(status_code=422, detail=verify.get("detail") or "DNS provider credential verification failed")
+
+        token = encrypt_dns_credentials(dict(body.credentials))
+        await conn.execute(
+            """INSERT INTO letsencrypt_account_dns_credentials (account_id, dns_provider, credentials_encrypted, updated_at)
+               VALUES ($1, $2, $3, NOW())
+               ON CONFLICT (account_id) DO UPDATE SET
+                   dns_provider = EXCLUDED.dns_provider,
+                   credentials_encrypted = EXCLUDED.credentials_encrypted,
+                   updated_at = NOW()""",
+            account_id, provider_name, token,
+        )
+        # Keep the account's provider selection in sync.
+        await conn.execute(
+            "UPDATE letsencrypt_accounts SET dns_provider = $1, updated_at = NOW() WHERE id = $2",
+            provider_name, account_id,
+        )
+        return {"ok": True, "dns_provider": provider_name, "detail": verify.get("detail", "Credentials stored.")}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/accounts/{account_id}/dns-credentials")
+async def delete_dns_credentials(account_id: int, authorization: str = Header(None)):
+    """Remove an account's stored DNS credentials. Admin-only."""
+    from auth_middleware import get_current_user_from_token
+    current_user = await get_current_user_from_token(authorization)
+    if not current_user.get('is_admin', False):
+        raise HTTPException(status_code=403, detail="Admin access required")
+    conn = await get_database_connection()
+    try:
+        await conn.execute("DELETE FROM letsencrypt_account_dns_credentials WHERE account_id = $1", account_id)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/orders/{order_id}/dns-confirm")
+async def confirm_dns_order(order_id: int, authorization: str = Header(None)):
+    """Manual DNS-01 only: the user asserts the TXT record is published; tell the CA to validate.
+    Requires ssl.create + a per-user rate limit; acts only on a dns-01 + manual + pending order."""
+    from auth_middleware import get_current_user_from_token, check_user_permission
+    current_user = await get_current_user_from_token(authorization)
+    has_perm = await check_user_permission(current_user['id'], 'ssl', 'create')
+    if not has_perm:
+        raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.create required")
+    await _enforce_dns_confirm_rate_limit(current_user['id'])
+
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+    finally:
+        await close_database_connection(conn)
+    if not order:
+        raise HTTPException(status_code=404, detail="Order not found")
+    if order['challenge_type'] != 'dns-01' or (order['dns_provider'] or 'manual') != 'manual':
+        raise HTTPException(status_code=409, detail="This order is not a manual DNS-01 order")
+    if order['status'] not in ('pending', 'processing'):
+        raise HTTPException(status_code=409, detail=f"Order is '{order['status']}' and cannot be confirmed")
+
+    from services.dns01_orchestrator import confirm_manual_dns01
+    await confirm_manual_dns01(order_id)
+    return {"ok": True, "message": "DNS-01 confirmation submitted; the CA will validate shortly."}
+
+
 # --- Certificate operations ---
 
 @router.post("/certificates")
@@ -222,32 +490,61 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
         logger.info(f"ACME: Using account_id={account_id} for certificate request")
 
         warnings = []
-        # Audit Tur 5 / Commit 8c: empty cluster_ids in UI means "global certificate".
-        # Resolve to all ACME-enabled active clusters; only fail if NONE exist.
+
+        # Issue #35: resolve the effective challenge method (request override, else account default).
+        conn_acct = await get_database_connection()
+        try:
+            acct = await conn_acct.fetchrow(
+                "SELECT challenge_type, dns_provider FROM letsencrypt_accounts WHERE id = $1", account_id
+            )
+        finally:
+            await close_database_connection(conn_acct)
+        effective_challenge = (body.challenge_type or (acct['challenge_type'] if acct else None) or 'http-01')
+        dns_provider = (acct['dns_provider'] if acct else None)
+        is_dns01 = (effective_challenge == 'dns-01')
+        has_wildcard = any(d.startswith('*.') for d in body.domains)
+
+        if is_dns01:
+            if not await _dns01_enabled():
+                raise HTTPException(status_code=409, detail="DNS-01 is disabled by an administrator (enable it in Settings).")
+            if not is_supported((dns_provider or '').strip()):
+                raise HTTPException(status_code=422, detail="The selected ACME account has no DNS provider configured for DNS-01.")
+            if (dns_provider or 'manual') == 'manual':
+                # Manual DNS-01 cannot be renewed unattended; auto-renew is forced off on the issued
+                # certificate (see _complete_certificate). Tell the requester so it isn't a surprise.
+                warnings.append(
+                    "Manual DNS-01 certificates cannot auto-renew unattended. Auto-renew will be disabled; "
+                    "re-publish the TXT record and request renewal before expiry."
+                )
+        elif has_wildcard:
+            raise HTTPException(status_code=422, detail="Wildcard certificates require a DNS-01 account.")
+
+        # Empty cluster_ids = "global certificate". For DNS-01 no ACME Challenge Routing / port 80 is
+        # needed, so resolve to ALL active clusters; http-01 still requires acme_enabled clusters.
         if not body.cluster_ids:
             conn_resolve = await get_database_connection()
             try:
-                acme_clusters_resolved = await conn_resolve.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
-                )
+                if is_dns01:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    resolved = await conn_resolve.fetch("SELECT id FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE")
             finally:
                 await close_database_connection(conn_resolve)
-            if not acme_clusters_resolved:
+            if not resolved:
+                if is_dns01:
+                    raise HTTPException(status_code=422, detail="Cannot issue certificate: no active clusters configured.")
                 raise HTTPException(
                     status_code=422,
                     detail="Cannot issue certificate: no ACME-enabled clusters configured. "
                            "Enable ACME Challenge Routing on at least one cluster in Cluster Management, "
                            "Apply the configuration change, then retry."
                 )
-            body.cluster_ids = [c['id'] for c in acme_clusters_resolved]
+            body.cluster_ids = [c['id'] for c in resolved]
             warnings.append(
-                f"No clusters specified — applied to all ACME-enabled cluster(s) ({len(body.cluster_ids)})"
+                f"No clusters specified — applied to all {'active' if is_dns01 else 'ACME-enabled'} cluster(s) ({len(body.cluster_ids)})"
             )
-            logger.warning(
-                f"ACME: Empty cluster_ids → global cert fallback to {len(body.cluster_ids)} ACME-enabled cluster(s)"
-            )
-        else:
-            # Validate that referenced clusters exist + are ACME-enabled (warn-only).
+        elif not is_dns01:
+            # http-01 only: warn if no cluster has ACME Challenge Routing enabled.
             try:
                 conn_warn = await get_database_connection()
                 try:
@@ -255,7 +552,6 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
                         "SELECT COUNT(*) FROM haproxy_clusters WHERE acme_enabled = TRUE AND is_active = TRUE"
                     )
                     if acme_clusters == 0:
-                        logger.warning("ACME: No clusters with ACME Challenge Routing enabled - certificate validation will likely fail")
                         warnings.append(
                             "No clusters have ACME Challenge Routing enabled. "
                             "Certificate validation will fail. Enable it in Cluster Management and Apply Changes first."
@@ -269,14 +565,30 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             account_id=account_id,
             domains=body.domains,
             cluster_ids=body.cluster_ids,
+            challenge_type=effective_challenge,
+            created_by=current_user['id'],
         )
 
-        challenges = await acme_service.respond_to_challenges(order['order_id'])
+        # Audit trail: record who requested the certificate + the method (esp. for DNS-01/wildcard,
+        # which has a wider blast radius than http-01). Never raises into the request path.
+        try:
+            from utils.activity_log import record_event
+            await record_event(
+                order['order_id'], "acme.order.requested",
+                message=f"Certificate requested ({effective_challenge}) by user {current_user['id']}",
+                details={"user_id": current_user['id'], "challenge_type": effective_challenge,
+                         "dns_provider": dns_provider, "domains": body.domains},
+            )
+        except Exception:
+            pass
+
+        # http-01 posts the challenge response immediately (token served continuously). dns-01 is
+        # driven by the orchestrator AFTER the TXT is published (manual waits for dns-confirm), so we
+        # must NOT respond here.
+        challenges = []
+        if not is_dns01:
+            challenges = await acme_service.respond_to_challenges(order['order_id'])
 
-        # Commit 5b: re-fetch the order's *current* status from DB. The status
-        # returned by create_order() reflects the moment of creation; after
-        # respond_to_challenges() the CA may have already advanced it (e.g. to
-        # 'processing'). Surfacing stale status leads UI to under-poll.
         conn_status = await get_database_connection()
         try:
             fresh_status = await conn_status.fetchval(
@@ -287,14 +599,23 @@ async def request_certificate(body: CertificateRequest, authorization: str = Hea
             await close_database_connection(conn_status)
         effective_status = fresh_status or order['status']
 
-        logger.info(f"ACME: Order {order['order_id']} created, {len(challenges)} challenge(s) posted, status={effective_status}")
+        if is_dns01:
+            msg = ("Order created. Publish the DNS TXT record shown for each domain, then confirm."
+                   if dns_provider == 'manual'
+                   else "Order created. The DNS TXT record(s) will be published automatically; waiting for CA validation.")
+        else:
+            msg = "Order created. ACME challenges have been posted. Waiting for CA validation."
+
+        logger.info(f"ACME: Order {order['order_id']} created ({effective_challenge}), status={effective_status}")
 
         return {
             "order_id": order['order_id'],
             "status": effective_status,
             "domains": body.domains,
+            "challenge_type": effective_challenge,
+            "dns_provider": dns_provider,
             "challenges": challenges,
-            "message": "Order created. ACME challenges have been posted. Waiting for CA validation.",
+            "message": msg,
             "warnings": warnings,
         }
     except HTTPException:
@@ -316,7 +637,8 @@ async def list_orders(authorization: str = Header(None)):
         rows = await conn.fetch("""
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.ssl_certificate_id, o.cluster_ids, o.error_detail,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             ORDER BY o.created_at DESC
@@ -345,7 +667,8 @@ async def get_order(order_id: int, authorization: str = Header(None)):
             SELECT o.id, o.account_id, o.order_url, o.status, o.domains,
                    o.certificate_url, o.finalize_url, o.expires_at,
                    o.error_detail, o.ssl_certificate_id, o.cluster_ids,
-                   o.created_at, o.updated_at, a.email as account_email
+                   o.created_at, o.updated_at, o.challenge_type, a.email as account_email,
+                   a.dns_provider
             FROM letsencrypt_orders o
             JOIN letsencrypt_accounts a ON o.account_id = a.id
             WHERE o.id = $1
@@ -353,13 +676,23 @@ async def get_order(order_id: int, authorization: str = Header(None)):
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
 
+        # Issue #35: include challenge_type + dns_txt_value (PUBLIC DNS data — NOT key_authorization,
+        # NOT the API token) so the UI can render manual DNS-01 instructions. Explicit column list.
         challenges = await conn.fetch(
-            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
+            "SELECT id, order_id, domain, token, challenge_url, status, validated_at, created_at, "
+            "challenge_type, dns_txt_value FROM acme_challenges WHERE order_id = $1 ORDER BY domain", order_id
         )
         result = dict(order)
         result['domains'] = json.loads(result['domains']) if isinstance(result['domains'], str) else result['domains']
         result['cluster_ids'] = json.loads(result['cluster_ids']) if isinstance(result['cluster_ids'], str) else result['cluster_ids']
-        result['challenges'] = [dict(c) for c in challenges]
+        ch_list = []
+        for c in challenges:
+            cd = dict(c)
+            if cd.get('challenge_type') == 'dns-01':
+                # Server computes the record name (wildcard *.-stripping lives server-side).
+                cd['dns_record_name'] = acme_service._challenge_dns_name(cd['domain'])
+            ch_list.append(cd)
+        result['challenges'] = ch_list
         return result
     finally:
         await close_database_connection(conn)
@@ -470,18 +803,23 @@ async def renew_order(order_id: int, authorization: str = Header(None)):
     conn = await get_database_connection()
     try:
         order = await conn.fetchrow(
-            "SELECT account_id, domains, cluster_ids FROM letsencrypt_orders WHERE id = $1", order_id
+            "SELECT account_id, domains, cluster_ids, challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
         )
         if not order:
             raise HTTPException(status_code=404, detail="Order not found")
         domains = json.loads(order['domains']) if isinstance(order['domains'], str) else order['domains']
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
+        challenge_type = order['challenge_type'] or 'http-01'
 
         new_order = await acme_service.create_order(
-            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids
+            account_id=order['account_id'], domains=domains, cluster_ids=cluster_ids,
+            challenge_type=challenge_type, created_by=current_user['id'],
         )
-        challenges = await acme_service.respond_to_challenges(new_order['order_id'])
-        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenges": challenges}
+        # dns-01 is driven by the orchestrator after the TXT is published; only http-01 responds here.
+        challenges = []
+        if challenge_type != 'dns-01':
+            challenges = await acme_service.respond_to_challenges(new_order['order_id'])
+        return {"message": "Renewal order created", "new_order_id": new_order['order_id'], "challenge_type": challenge_type, "challenges": challenges}
     finally:
         await close_database_connection(conn)
 
@@ -611,11 +949,17 @@ async def get_renewal_schedule(authorization: str = Header(None)):
         raise HTTPException(status_code=403, detail="Insufficient permissions: ssl.read required")
     conn = await get_database_connection()
     try:
+        # Issue #35: expose the challenge method/provider (via the originating order/account) so the
+        # UI can distinguish manual DNS-01 certs, which cannot auto-renew unattended. LEFT JOINs keep
+        # legacy certs (no order link / pre-DNS-01 columns) rendering as http-01.
         certs = await conn.fetch("""
-            SELECT id, name, primary_domain, expiry_date, auto_renew, days_until_expiry
-            FROM ssl_certificates
-            WHERE source = 'letsencrypt' AND is_active = TRUE
-            ORDER BY expiry_date ASC NULLS LAST
+            SELECT c.id, c.name, c.primary_domain, c.expiry_date, c.auto_renew, c.days_until_expiry,
+                   o.challenge_type, a.dns_provider
+            FROM ssl_certificates c
+            LEFT JOIN letsencrypt_orders o ON o.id = c.letsencrypt_order_id
+            LEFT JOIN letsencrypt_accounts a ON a.id = o.account_id
+            WHERE c.source = 'letsencrypt' AND c.is_active = TRUE
+            ORDER BY c.expiry_date ASC NULLS LAST
         """)
         return [dict(c) for c in certs]
     finally:
@@ -663,6 +1007,18 @@ async def _complete_certificate(order_id: int) -> dict:
         cluster_ids = json.loads(order['cluster_ids']) if isinstance(order['cluster_ids'], str) else order['cluster_ids']
         primary_domain = domains[0] if domains else 'unknown'
 
+        # Issue #35: a manual DNS-01 certificate cannot be auto-renewed unattended (the renewal task
+        # skips it — see main.py), so persist auto_renew=FALSE rather than storing a misleading
+        # "Enabled" that the user trusts while the cert silently expires. http-01 and automated
+        # DNS-01 (e.g. Cloudflare) keep auto_renew=TRUE, preserving existing behaviour.
+        auto_renew_value = True
+        if order.get('challenge_type') == 'dns-01':
+            acct_provider = await conn.fetchval(
+                "SELECT dns_provider FROM letsencrypt_accounts WHERE id = $1", order['account_id']
+            )
+            if (acct_provider or 'manual') == 'manual':
+                auto_renew_value = False
+
         # Commit 5g: guard against empty cert_private_key.
         # Inserting an SSL certificate row with an empty private_key would silently
         # produce an unusable certificate (HAProxy would fail to load on Apply, or
@@ -749,12 +1105,13 @@ async def _complete_certificate(order_id: int) -> dict:
                     certificate_content = $1, private_key_content = $2, chain_content = $3,
                     all_domains = $4::jsonb, expiry_date = $5, days_until_expiry = $6,
                     issuer = $7, fingerprint = $8, letsencrypt_order_id = $9,
-                    auto_renew = TRUE, is_active = TRUE, last_config_status = 'PENDING',
+                    auto_renew = $11, is_active = TRUE, last_config_status = 'PENDING',
                     updated_at = NOW()
                 WHERE id = $10
             """, cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id, cert_id,
+                auto_renew_value)
             logger.info(f"ACME RENEWAL: Updated certificate {cert_id} for {primary_domain}, status=PENDING")
         else:
             cert_row = await conn.fetchrow("""
@@ -764,11 +1121,12 @@ async def _complete_certificate(order_id: int) -> dict:
                      usage_type, source, letsencrypt_order_id, auto_renew, is_active,
                      last_config_status)
                 VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8, $9, $10,
-                        'frontend', 'letsencrypt', $11, TRUE, TRUE, 'PENDING')
+                        'frontend', 'letsencrypt', $11, $12, TRUE, 'PENDING')
                 RETURNING id
             """, f"le-{primary_domain}", cert_data['certificate_pem'], private_key_pem,
                 cert_data.get('chain_pem', ''), primary_domain, all_domains,
-                expiry_date, days_until_expiry, issuer, fingerprint, order_id)
+                expiry_date, days_until_expiry, issuer, fingerprint, order_id,
+                auto_renew_value)
             cert_id = cert_row['id']
 
         await conn.execute(
@@ -815,12 +1173,18 @@ async def _complete_certificate(order_id: int) -> dict:
             if mapped:
                 effective_cluster_ids = [r['cluster_id'] for r in mapped]
             else:
-                # Audit Tur 6 / Commit 5j: only fall back to ACME-enabled clusters.
-                # Applying renewal to ACME-disabled clusters could re-introduce Issue #11
-                # patterns and risks deploying certs to clusters where they can't be renewed.
-                all_clusters = await conn.fetch(
-                    "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                # Audit Tur 6 / Commit 5j: http-01 only falls back to ACME-enabled clusters.
+                # Issue #35: a dns-01 cert needs NO challenge routing / port 80, so it can renew on
+                # any active cluster — resolve to all active clusters for dns-01.
+                ch_type = await conn.fetchval(
+                    "SELECT challenge_type FROM letsencrypt_orders WHERE id = $1", order_id
                 )
+                if ch_type == 'dns-01':
+                    all_clusters = await conn.fetch("SELECT id FROM haproxy_clusters WHERE is_active = TRUE")
+                else:
+                    all_clusters = await conn.fetch(
+                        "SELECT id FROM haproxy_clusters WHERE is_active = TRUE AND acme_enabled = TRUE"
+                    )
                 effective_cluster_ids = [r['id'] for r in all_clusters]
             if effective_cluster_ids:
                 logger.info(f"ACME RENEWAL: Resolved {len(effective_cluster_ids)} cluster(s) for global cert {cert_id}")

backend/routers/mfa.py

@@ -0,0 +1,727 @@
+"""MFA router — Issue #18, v1.6.0.
+
+All endpoints are additive; nothing here breaks existing JWT or apply_service flows.
+Authentication is JWT-based (Bearer). Admin endpoints additionally require
+``users.is_admin == True`` (canonical super-admin flag).
+"""
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from middleware.mfa_rate_limit_key import mfa_rate_limit_key
+from middleware.mfa_rate_limits import MFA_LIMITS
+from middleware.rate_limiter import limiter
+from models.mfa import (
+    MfaAdminResetAllRequest,
+    MfaAdminResetRequest,
+    MfaDisableRequest,
+    MfaEnrollConfirmRequest,
+    MfaEnrollConfirmResponse,
+    MfaEnrollStartResponse,
+    MfaRegenerateBackupRequest,
+    MfaRegenerateBackupResponse,
+    MfaStatusResponse,
+)
+from services import mfa_service
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/api/mfa", tags=["MFA"])
+
+# Lifecycle constants (Plan section 5)
+PENDING_ENROLLMENT_TTL_SECONDS = 600  # 10 minutes — QR scan + verify window
+PENDING_ENROLLMENT_MAX_ATTEMPTS = 5
+
+
+# ---------------------------------------------------------------------------
+# Authentication helpers
+# ---------------------------------------------------------------------------
+
+
+async def _require_user(authorization: Optional[str]) -> dict:
+    user = await get_current_user_from_token(authorization)
+    if not user:
+        raise HTTPException(status_code=401, detail="Not authenticated")
+    return user
+
+
+async def _require_admin(authorization: Optional[str]) -> dict:
+    user = await _require_user(authorization)
+    if not user.get("is_admin"):
+        raise HTTPException(status_code=403, detail="Admin privileges required")
+    return user
+
+
+async def _cleanup_expired_pending_enrollments(conn) -> None:
+    try:
+        await conn.execute(
+            "DELETE FROM mfa_pending_enrollments WHERE expires_at < NOW()"
+        )
+    except Exception as exc:
+        logger.debug(f"Pending-enrollment cleanup skipped: {exc}")
+
+
+# ---------------------------------------------------------------------------
+# Self status
+# ---------------------------------------------------------------------------
+
+
+@router.get(
+    "/status",
+    summary="MFA status for the authenticated user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_status(authorization: str = Header(None)):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            current["id"],
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            current["id"],
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+@router.get(
+    "/admin/status/{user_id}",
+    summary="Admin: MFA status of any user",
+    response_model=MfaStatusResponse,
+)
+async def mfa_admin_status(user_id: int, authorization: str = Header(None)):
+    await _require_admin(authorization)
+    conn = await get_database_connection()
+    try:
+        row = await conn.fetchrow(
+            """
+            SELECT mfa_enabled, mfa_method, mfa_enrolled_at, mfa_last_used_at
+              FROM users
+             WHERE id = $1
+            """,
+            user_id,
+        )
+        remaining = await conn.fetchval(
+            """
+            SELECT COUNT(*) FROM mfa_backup_codes
+             WHERE user_id = $1 AND used_at IS NULL
+            """,
+            user_id,
+        )
+    finally:
+        await close_database_connection(conn)
+
+    if not row:
+        raise HTTPException(status_code=404, detail="User not found")
+
+    return MfaStatusResponse(
+        enabled=bool(row["mfa_enabled"]),
+        method=row["mfa_method"],
+        enrolled_at=row["mfa_enrolled_at"].isoformat() if row["mfa_enrolled_at"] else None,
+        last_used_at=row["mfa_last_used_at"].isoformat() if row["mfa_last_used_at"] else None,
+        backup_codes_remaining=int(remaining or 0),
+    )
+
+
+# ---------------------------------------------------------------------------
+# Enrollment
+# ---------------------------------------------------------------------------
+
+
+@router.post(
+    "/enroll/start",
+    summary="Begin TOTP enrollment (returns secret + otpauth URI)",
+    response_model=MfaEnrollStartResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_start, key_func=mfa_rate_limit_key)
+async def mfa_enroll_start(request: Request, authorization: str = Header(None)):
+    current = await _require_user(authorization)
+
+    # Round 7 audit fix — REFUSE re-enrollment if the user is already MFA-on.
+    # Without this guard a stolen JWT could silently rotate the victim's TOTP
+    # secret + invalidate all their backup codes via /enroll/start ->
+    # /enroll/confirm (overwriting `users.mfa_secret_encrypted` and replacing
+    # `mfa_backup_codes`). To re-enroll, the user must first call /api/mfa/disable
+    # (which requires a fresh TOTP) or an admin must run /api/mfa/admin-reset.
+    secret_plain = mfa_service.generate_totp_secret()
+    secret_encrypted = mfa_service.encrypt_secret(secret_plain)
+
+    conn = await get_database_connection()
+    blocked = False
+    try:
+        # Single transaction with SELECT FOR UPDATE closes the TOCTOU window
+        # between the mfa_enabled check and the pending_enrollment upsert.
+        async with conn.transaction():
+            row = await conn.fetchrow(
+                "SELECT mfa_enabled FROM users WHERE id = $1 FOR UPDATE",
+                current["id"],
+            )
+            if row and row["mfa_enabled"]:
+                blocked = True
+            else:
+                await _cleanup_expired_pending_enrollments(conn)
+                await conn.execute(
+                    """
+                    INSERT INTO mfa_pending_enrollments
+                        (user_id, secret_encrypted, attempts, expires_at)
+                    VALUES ($1, $2, 0, NOW() + ($3 || ' seconds')::interval)
+                    ON CONFLICT (user_id) DO UPDATE
+                      SET secret_encrypted = EXCLUDED.secret_encrypted,
+                          attempts = 0,
+                          expires_at = EXCLUDED.expires_at,
+                          created_at = CURRENT_TIMESTAMP
+                    """,
+                    current["id"],
+                    secret_encrypted,
+                    str(PENDING_ENROLLMENT_TTL_SECONDS),
+                )
+    finally:
+        await close_database_connection(conn)
+
+    if blocked:
+        raise HTTPException(
+            status_code=400,
+            detail="MFA is already enabled. Disable it first (via /api/mfa/disable or admin reset) to re-enroll.",
+        )
+
+    hostname_hint = request.url.hostname if request.url else None
+    label = mfa_service.build_account_label(current["username"], hostname_hint)
+    otpauth_uri = mfa_service.build_otpauth_uri(label, secret_plain)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.started",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"secret_len": len(secret_plain)},  # NEVER log the secret itself
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+
+    return MfaEnrollStartResponse(
+        secret=secret_plain,
+        otpauth_uri=otpauth_uri,
+        expires_in=PENDING_ENROLLMENT_TTL_SECONDS,
+    )
+
+
+@router.post(
+    "/enroll/confirm",
+    summary="Confirm enrollment with a TOTP code; returns 10 backup codes once",
+    response_model=MfaEnrollConfirmResponse,
+)
+@limiter.limit(MFA_LIMITS.enroll_confirm, key_func=mfa_rate_limit_key)
+async def mfa_enroll_confirm(
+    payload: MfaEnrollConfirmRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # Pre-generate plain codes & hashes outside the DB transaction so we
+    # never hold a row lock for ~2-3s of bcrypt work.
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    # failure: { reason, attempts, http_status, detail }; success when None.
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            # Lock the pending row so concurrent /enroll/confirm calls for the
+            # same user can't both consume the same pending enrollment.
+            pending = await conn.fetchrow(
+                """
+                SELECT secret_encrypted, attempts, expires_at
+                  FROM mfa_pending_enrollments
+                 WHERE user_id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+
+            if not pending:
+                failure = {
+                    "reason": "no_pending",
+                    "http_status": 410,
+                    "detail": "No pending enrollment; start again",
+                }
+            else:
+                from datetime import datetime as _dt
+                if pending["expires_at"] and pending["expires_at"] < _dt.utcnow():
+                    await conn.execute(
+                        "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                        current["id"],
+                    )
+                    failure = {
+                        "reason": "expired",
+                        "http_status": 410,
+                        "detail": "Enrollment expired; start again",
+                    }
+                else:
+                    secret_plain = mfa_service.decrypt_secret(pending["secret_encrypted"])
+                    if not secret_plain:
+                        await conn.execute(
+                            "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                            current["id"],
+                        )
+                        failure = {
+                            "reason": "unreadable",
+                            "http_status": 500,
+                            "detail": "Pending enrollment unreadable; start again",
+                        }
+                    else:
+                        ok, _step = mfa_service.verify_totp_with_replay_guard(
+                            secret_plain, payload.code, None
+                        )
+                        if not ok:
+                            new_attempts = (pending["attempts"] or 0) + 1
+                            if new_attempts >= PENDING_ENROLLMENT_MAX_ATTEMPTS:
+                                await conn.execute(
+                                    "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "too_many_attempts",
+                                    "attempts": new_attempts,
+                                    "http_status": 410,
+                                    "detail": "Enrollment invalidated; start again",
+                                }
+                            else:
+                                await conn.execute(
+                                    "UPDATE mfa_pending_enrollments SET attempts = $1 WHERE user_id = $2",
+                                    new_attempts,
+                                    current["id"],
+                                )
+                                failure = {
+                                    "reason": "invalid_code",
+                                    "attempts": new_attempts,
+                                    "http_status": 401,
+                                    "detail": "Invalid code",
+                                }
+                        else:
+                            # Verified — finalize state inside the transaction.
+                            await conn.execute(
+                                """
+                                UPDATE users
+                                   SET mfa_enabled = TRUE,
+                                       mfa_method = 'totp',
+                                       mfa_secret_encrypted = $1,
+                                       mfa_enrolled_at = NOW(),
+                                       mfa_last_used_totp_step = NULL,
+                                       mfa_last_used_at = NULL
+                                 WHERE id = $2
+                                """,
+                                pending["secret_encrypted"],
+                                current["id"],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+                            await conn.execute(
+                                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1",
+                                current["id"],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        # Log AFTER commit so the audit row reflects what actually persisted.
+        await log_user_activity(
+            user_id=current["id"],
+            action="mfa.enrollment.failed",
+            resource_type="mfa",
+            resource_id=str(current["id"]),
+            details={
+                "reason": failure["reason"],
+                "attempts": failure.get("attempts"),
+            },
+            ip_address=ip,
+            user_agent=ua,
+        )
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.enrollment.confirmed",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"method": "totp"},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaEnrollConfirmResponse(enabled=True, backup_codes=plain_codes, method="totp")
+
+
+# ---------------------------------------------------------------------------
+# Disable + regenerate
+# ---------------------------------------------------------------------------
+
+
+async def _verify_user_code(conn, user_row: dict, code: str) -> Optional[str]:
+    """Verify a TOTP-or-backup code against the user's stored secret.
+
+    Returns the method used ('totp' / 'backup') on success, None on failure.
+
+    On TOTP success the step counter is bumped *atomically* — the UPDATE
+    only succeeds if no other request consumed the same (or a newer) step
+    in between. On backup success the consumed row's used_at is set with
+    an atomic ``WHERE used_at IS NULL RETURNING id`` pattern.
+    """
+    secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+    if secret_plain:
+        ok, step = mfa_service.verify_totp_with_replay_guard(
+            secret_plain, code, user_row["mfa_last_used_totp_step"]
+        )
+        if ok:
+            bumped = await conn.fetchval(
+                """
+                UPDATE users
+                   SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                 WHERE id = $2
+                   AND (mfa_last_used_totp_step IS NULL
+                        OR mfa_last_used_totp_step < $1)
+                 RETURNING id
+                """,
+                step,
+                user_row["id"],
+            )
+            if bumped:
+                return "totp"
+
+    rows = await conn.fetch(
+        """
+        SELECT id, code_hash FROM mfa_backup_codes
+         WHERE user_id = $1 AND used_at IS NULL
+        """,
+        user_row["id"],
+    )
+    for row in rows:
+        if await mfa_service.check_backup_code(code, row["code_hash"]):
+            consumed = await conn.fetchval(
+                """
+                UPDATE mfa_backup_codes
+                   SET used_at = NOW()
+                 WHERE id = $1 AND used_at IS NULL
+                 RETURNING id
+                """,
+                row["id"],
+            )
+            if consumed:
+                return "backup"
+    return None
+
+
+@router.post("/disable", summary="Disable MFA (requires current TOTP or backup)")
+@limiter.limit(MFA_LIMITS.disable, key_func=mfa_rate_limit_key)
+async def mfa_disable(
+    payload: MfaDisableRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    conn = await get_database_connection()
+    try:
+        user_row = await conn.fetchrow(
+            """
+            SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+              FROM users WHERE id = $1
+            """,
+            current["id"],
+        )
+        if not user_row or not user_row["mfa_enabled"]:
+            raise HTTPException(status_code=400, detail="MFA is not enabled")
+
+        method_used = await _verify_user_code(conn, dict(user_row), payload.code)
+        if not method_used:
+            await log_user_activity(
+                user_id=current["id"],
+                action="mfa.disable.failed",
+                resource_type="mfa",
+                resource_id=str(current["id"]),
+                details={"reason": "invalid_code"},
+                ip_address=str(request.client.host) if request.client else None,
+                user_agent=request.headers.get("user-agent"),
+            )
+            raise HTTPException(status_code=401, detail="Invalid code")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                current["id"],
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", current["id"]
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", current["id"]
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.disabled.self",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"verified_via": method_used},
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"enabled": False}
+
+
+@router.post(
+    "/backup-codes/regenerate",
+    summary="Issue 10 fresh backup codes (TOTP required)",
+    response_model=MfaRegenerateBackupResponse,
+)
+@limiter.limit(MFA_LIMITS.regenerate_backup_codes, key_func=mfa_rate_limit_key)
+async def mfa_regenerate_backup_codes(
+    payload: MfaRegenerateBackupRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    current = await _require_user(authorization)
+    # bcrypt-hash the new codes outside the DB transaction (~2-3s of CPU work).
+    plain_codes = mfa_service.generate_backup_codes()
+    hashes = await mfa_service.hash_backup_codes(plain_codes)
+
+    ip = str(request.client.host) if request.client else None
+    ua = request.headers.get("user-agent")
+
+    failure: Optional[dict] = None
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            user_row = await conn.fetchrow(
+                """
+                SELECT id, mfa_enabled, mfa_secret_encrypted, mfa_last_used_totp_step
+                  FROM users WHERE id = $1
+                 FOR UPDATE
+                """,
+                current["id"],
+            )
+            if not user_row or not user_row["mfa_enabled"]:
+                failure = {"http_status": 400, "detail": "MFA is not enabled"}
+            else:
+                secret_plain = mfa_service.decrypt_secret(user_row["mfa_secret_encrypted"])
+                if not secret_plain:
+                    failure = {"http_status": 500, "detail": "MFA secret unreadable"}
+                else:
+                    ok, step = mfa_service.verify_totp_with_replay_guard(
+                        secret_plain, payload.code, user_row["mfa_last_used_totp_step"]
+                    )
+                    if not ok:
+                        failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                    else:
+                        bumped = await conn.fetchval(
+                            """
+                            UPDATE users
+                               SET mfa_last_used_totp_step = $1, mfa_last_used_at = NOW()
+                             WHERE id = $2
+                               AND (mfa_last_used_totp_step IS NULL
+                                    OR mfa_last_used_totp_step < $1)
+                             RETURNING id
+                            """,
+                            step,
+                            current["id"],
+                        )
+                        if not bumped:
+                            failure = {"http_status": 401, "detail": "Invalid TOTP code"}
+                        else:
+                            await conn.execute(
+                                "DELETE FROM mfa_backup_codes WHERE user_id = $1",
+                                current["id"],
+                            )
+                            await conn.executemany(
+                                "INSERT INTO mfa_backup_codes (user_id, code_hash) VALUES ($1, $2)",
+                                [(current["id"], h) for h in hashes],
+                            )
+    finally:
+        await close_database_connection(conn)
+
+    if failure is not None:
+        raise HTTPException(status_code=failure["http_status"], detail=failure["detail"])
+
+    await log_user_activity(
+        user_id=current["id"],
+        action="mfa.backup_codes.regenerated",
+        resource_type="mfa",
+        resource_id=str(current["id"]),
+        details={"codes_count": len(plain_codes)},
+        ip_address=ip,
+        user_agent=ua,
+    )
+
+    return MfaRegenerateBackupResponse(backup_codes=plain_codes)
+
+
+# ---------------------------------------------------------------------------
+# Admin operations
+# ---------------------------------------------------------------------------
+
+
+@router.post("/admin-reset/{user_id}", summary="Admin: reset a single user's MFA")
+@limiter.limit(MFA_LIMITS.admin_reset, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset(
+    user_id: int,
+    payload: MfaAdminResetRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    if user_id == admin["id"]:
+        raise HTTPException(status_code=400, detail="Use /api/mfa/disable for self-reset")
+
+    conn = await get_database_connection()
+    try:
+        target = await conn.fetchrow(
+            "SELECT id, username, mfa_enabled FROM users WHERE id = $1", user_id
+        )
+        if not target:
+            raise HTTPException(status_code=404, detail="User not found")
+
+        async with conn.transaction():
+            await conn.execute(
+                """
+                UPDATE users
+                   SET mfa_enabled = FALSE,
+                       mfa_method = NULL,
+                       mfa_secret_encrypted = NULL,
+                       mfa_enrolled_at = NULL,
+                       mfa_last_used_at = NULL,
+                       mfa_last_used_totp_step = NULL
+                 WHERE id = $1
+                """,
+                user_id,
+            )
+            await conn.execute(
+                "DELETE FROM mfa_backup_codes WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_logins WHERE user_id = $1", user_id
+            )
+            await conn.execute(
+                "DELETE FROM mfa_pending_enrollments WHERE user_id = $1", user_id
+            )
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_reset",
+        resource_type="mfa",
+        resource_id=str(user_id),
+        details={
+            "target_user_id": user_id,
+            "target_username": target["username"],
+            "admin_user_id": admin["id"],
+            "reason": payload.reason,
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset": True, "user_id": user_id}
+
+
+@router.post(
+    "/admin-reset-all",
+    summary="Admin: emergency reset of MFA for all users (double confirm)",
+)
+@limiter.limit(MFA_LIMITS.admin_reset_all, key_func=mfa_rate_limit_key)
+async def mfa_admin_reset_all(
+    payload: MfaAdminResetAllRequest,
+    request: Request,
+    authorization: str = Header(None),
+):
+    admin = await _require_admin(authorization)
+    # Pydantic's Literal already enforces the magic string, but check defensively too.
+    if payload.confirm != "RESET ALL MFA":
+        raise HTTPException(status_code=400, detail="Invalid confirmation string")
+
+    conn = await get_database_connection()
+    try:
+        async with conn.transaction():
+            reset_count = await conn.fetchval(
+                """
+                WITH affected AS (
+                    UPDATE users
+                       SET mfa_enabled = FALSE,
+                           mfa_method = NULL,
+                           mfa_secret_encrypted = NULL,
+                           mfa_enrolled_at = NULL,
+                           mfa_last_used_at = NULL,
+                           mfa_last_used_totp_step = NULL
+                     WHERE mfa_enabled = TRUE
+                     RETURNING id
+                )
+                SELECT COUNT(*) FROM affected
+                """
+            )
+            await conn.execute("DELETE FROM mfa_backup_codes")
+            await conn.execute("DELETE FROM mfa_pending_logins")
+            await conn.execute("DELETE FROM mfa_pending_enrollments")
+    finally:
+        await close_database_connection(conn)
+
+    await log_user_activity(
+        user_id=admin["id"],
+        action="mfa.disabled.admin_bulk_reset",
+        resource_type="mfa",
+        resource_id=str(admin["id"]),
+        details={
+            "reset_count": int(reset_count or 0),
+            "reason": payload.reason,
+            "admin_user_id": admin["id"],
+        },
+        ip_address=str(request.client.host) if request.client else None,
+        user_agent=request.headers.get("user-agent"),
+    )
+    return {"reset_count": int(reset_count or 0)}

backend/routers/user.py

@@ -38,18 +38,19 @@ async def get_users(authorization: str = Header(None)):
         # Get users with their roles (only active users)
         try:
             users = await conn.fetch("""
-                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active, 
-                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at
+                SELECT u.id, u.username, u.email, u.full_name, u.phone, u.role, u.is_active,
+                       u.is_admin, u.is_verified, u.created_at, u.updated_at, u.last_login_at,
+                       COALESCE(u.mfa_enabled, FALSE) AS mfa_enabled
                 FROM users u
                 WHERE u.is_active = TRUE
                 ORDER BY u.username
             """)
         except Exception as schema_error:
             logger.warning(f"Schema error in users query, using fallback: {schema_error}")
-            # Fallback query with minimal columns
+            # Fallback query with minimal columns (pre-MFA-migration deploys)
             users = await conn.fetch("""
                 SELECT id, username, email, is_active, is_admin, created_at
-                FROM users 
+                FROM users
                 WHERE is_active = TRUE
                 ORDER BY username
             """)

backend/routers/vip.py

@@ -0,0 +1,987 @@
+"""Issue #27 — HA/VIP (Keepalived) management API (v1.7.0).
+
+Isolated router: it owns vip_instances / vip_members only. It NEVER imports or calls
+the global HAProxy apply flow (cluster.py::apply_pending_changes) or the haproxy.cfg
+generator — VIP "Apply" is a standalone verb that renders per-node keepalived.conf
+snapshots into vip_members and flips the VIP to APPLIED.
+
+For Apply-Management consistency it ALSO stages a standard `config_versions` row per
+change (version_name `vip-{id}-{action}`, status PENDING, is_active=FALSE — exactly like
+ssl-* versions) so VIP changes appear in the right-panel Pending Versions list with the
+product's standard "View Change" diff. is_active stays FALSE so a VIP version can never
+be served to an agent as haproxy.cfg (the agent config query requires is_active=TRUE);
+cluster.py keeps vip-* versions out of the generic haproxy apply/reject (NOT LIKE 'vip-%').
+
+Every DB access is wrapped so a (pathological) missing vip_* relation degrades to an
+empty/None result instead of a 500 (B-7) — preserving the fleet-wide no-op guarantee.
+"""
+import hashlib
+import json
+import logging
+import re
+import time
+import uuid
+from typing import List, Optional
+
+from fastapi import APIRouter, Header, HTTPException, Request
+
+from auth_middleware import check_user_permission, get_current_user_from_token
+from database.connection import close_database_connection, get_database_connection
+from models.vip import VIPCreate, VIPUpdate
+from services.keepalived_config import (
+    build_haproxy_check_script,
+    decrypt_vrrp_secret,
+    encrypt_vrrp_secret,
+    render_keepalived_conf,
+)
+from utils.activity_log import log_user_activity
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/vip", tags=["HA / VIP"])
+
+
+def _client_ip(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.client.host if request and request.client else None
+    except Exception:  # noqa: BLE001
+        return None
+
+
+def _user_agent(request: Optional[Request]) -> Optional[str]:
+    try:
+        return request.headers.get("user-agent") if request else None
+    except Exception:  # noqa: BLE001
+        return None
+
+
+async def _require(authorization: Optional[str], action: str):
+    """Authenticate + enforce vip.<action>; returns current_user or raises 401/403."""
+    current_user = await get_current_user_from_token(authorization)
+    ok = await check_user_permission(current_user["id"], "vip", action, current_user=current_user)
+    if not ok:
+        raise HTTPException(status_code=403, detail=f"vip.{action} permission required")
+    return current_user
+
+
+async def _alloc_free_vrid(conn, pool_id: int, requested: Optional[int]) -> int:
+    """Use the requested VRID if free in the pool, else the lowest free 1..255."""
+    used = {r["virtual_router_id"] for r in await conn.fetch(
+        "SELECT virtual_router_id FROM vip_instances WHERE pool_id=$1 AND is_active=TRUE", pool_id)}
+    if requested is not None:
+        if requested in used:
+            raise HTTPException(status_code=409, detail=f"VRID {requested} already used in this pool")
+        return requested
+    for cand in range(1, 256):
+        if cand not in used:
+            return cand
+    raise HTTPException(status_code=409, detail="No free VRID (1-255) left in this pool")
+
+
+def _md5(text: str) -> str:
+    return hashlib.md5(text.encode("utf-8")).hexdigest()
+
+
+# The VRRP secret is rendered into keepalived.conf as `auth_pass <value>`. The "secret
+# never leaves the server in cleartext" rule means any config we hand back to the UI
+# (e.g. the version diff) must mask it — only the at-rest Fernet token and the
+# agent-delivery endpoint ever see the real value. Mask the WHOLE remainder of the line
+# (not just up to the first space) so a secret containing whitespace can't partially leak.
+_AUTH_PASS_RE = re.compile(r"(auth_pass\s+).*")
+
+
+def _redact_secret(conf: Optional[str]) -> str:
+    return _AUTH_PASS_RE.sub(r"\1********", conf or "")
+
+
+def _derive_deploy_status(last_config_status: str, members, pending_delete: bool = False,
+                          is_active: bool = True) -> tuple:
+    """Display status that reflects ACTUAL agent convergence, not just the staging flag.
+
+    Other entities only read APPLIED once their agents acknowledge the new config; the
+    VIP now mirrors that so the table never claims a VIP is live before its member nodes
+    have deployed keepalived and acked the current config hash (issue #27 follow-up).
+
+        PENDING_DELETE — a deletion is STAGED and awaiting approval in Apply Management; the
+                     VIP keeps running until approved (nothing is torn down)
+        DELETING   — deletion APPROVED; member nodes are tearing keepalived down (not yet acked)
+        PENDING    — staged (created/edited); apply from Apply Management
+        SYNCING    — applied; an ONLINE member is still converging (deploying/acking)
+        AWAITING   — applied, but the un-converged members' agents are all OFFLINE, so
+                     nothing can deploy yet (bring the node's agent online) — not a hang
+        ACTIVE     — applied AND every member deployed & acked the current config hash
+        ERROR      — a member reported a deploy error
+        ATTENTION  — a member found a hand-managed keepalived (externally_managed)
+
+    Returns (status, synced_count, total_count).
+    """
+    # Approval-gated delete: staged (still running) vs approved (tearing down).
+    if pending_delete:
+        return "PENDING_DELETE", 0, len(members)
+    if not is_active:
+        total = len(members)
+        torn = sum(1 for m in members if m["last_deploy_state"] == "disabled")
+        return ("DELETED" if total and torn == total else "DELETING"), torn, total
+    if last_config_status == "PENDING":
+        return "PENDING", 0, len(members)
+    total = len(members)
+    if total == 0:
+        return "APPLIED", 0, 0
+
+    def _in_sync(m) -> bool:
+        return (m["last_deploy_state"] == "enabled"
+                and m["applied_config_hash"] is not None
+                and m["last_deploy_hash"] == m["applied_config_hash"])
+
+    synced = sum(1 for m in members if _in_sync(m))
+    if any(m["last_deploy_state"] == "error" for m in members):
+        return "ERROR", synced, total
+    if any(m["last_deploy_state"] == "externally_managed" for m in members):
+        return "ATTENTION", synced, total
+    if synced == total:
+        return "ACTIVE", synced, total
+    # Not fully converged: distinguish "actively converging" (an online agent will deploy
+    # on its next poll) from "waiting on offline agents" (nothing will happen until the
+    # operator brings the node's agent online) — the latter must NOT read as a live spinner.
+    not_synced = [m for m in members if not _in_sync(m)]
+    if not_synced and all((m["agent_status"] or "offline") != "online" for m in not_synced):
+        return "AWAITING", synced, total
+    return "SYNCING", synced, total
+
+
+async def render_vip_config_masked(conn, vip_id: int):
+    """Render the keepalived.conf each member node would deploy, as ONE masked text block
+    (VRRP secret never in cleartext). Used by the standard config-version diff for vip-*
+    versions (cluster.py) and to populate the staged version's config_content — so VIP
+    changes show the product's standard "View Change" instead of a bespoke screen.
+
+    Returns (text, {"name": ...}) or (None, None) if the VIP is gone. Best-effort: a
+    per-node render error becomes an inline comment rather than raising.
+    """
+    v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1", vip_id)
+    if not v:
+        return None, None
+    members = await conn.fetch("""
+        SELECT m.agent_id, m.network_interface, m.role, m.priority,
+               a.name AS agent_name, a.ip_address
+        FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+        WHERE m.vip_id=$1 ORDER BY m.priority DESC
+    """, vip_id)
+    auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
+    vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
+                "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                "track_haproxy": v["track_haproxy"]}
+    member_dicts = [{"role": m["role"], "priority": m["priority"],
+                     "network_interface": m["network_interface"], "agent_id": m["agent_id"],
+                     "ip_address": str(m["ip_address"]) if m["ip_address"] else ""} for m in members]
+    blocks = []
+    for m in members:
+        this_agent = next(d for d in member_dicts if d["agent_id"] == m["agent_id"])
+        peer_ips = [d["ip_address"] for d in member_dicts
+                    if d["agent_id"] != m["agent_id"] and d["ip_address"]]
+        try:
+            conf = render_keepalived_conf(vip=vip_dict, members=member_dicts,
+                                          this_agent=this_agent, peer_ips=peer_ips,
+                                          auth_pass_plain=auth_plain)
+        except Exception as exc:  # noqa: BLE001 — best-effort preview
+            conf = f"# cannot render this node yet: {exc}\n"
+        header = (f"# ===== node: {m['agent_name'] or m['agent_id']} "
+                  f"({m['ip_address'] or 'no IP'}) — {m['role']} priority {m['priority']} =====")
+        blocks.append(header + "\n" + _redact_secret(conf))
+    text = "\n\n".join(blocks) if blocks else "# (no participating nodes selected yet)\n"
+    return text, {"name": v["name"]}
+
+
+async def _stage_vip_version(conn, vip_id: int, action: str, created_by: Optional[int]):
+    """Stage a standard PENDING config_versions row for a VIP change so it shows in Apply
+    Management exactly like other entities. One row per cluster in the VIP's pool;
+    is_active=FALSE so it is NEVER delivered as haproxy.cfg. Best-effort — a staging
+    failure never fails the VIP operation (logged); the VIP still applies via its own flow.
+    """
+    try:
+        cluster_ids = [r["id"] for r in await conn.fetch(
+            "SELECT id FROM haproxy_clusters WHERE pool_id = "
+            "(SELECT pool_id FROM vip_instances WHERE id=$1)", vip_id)]
+        if not cluster_ids:
+            return
+        content, _meta = await render_vip_config_masked(conn, vip_id)
+        content = content or f"# VIP {vip_id} ({action})\n"
+        checksum = _md5(content)
+        # uuid suffix makes the name collision-proof even for sub-second same-action re-edits
+        # (UNIQUE(cluster_id, version_name) would otherwise reject a same-second retry; review LOW-1).
+        version_name = f"vip-{vip_id}-{action}-{int(time.time())}-{uuid.uuid4().hex[:6]}"
+        # Capture this change's PENDING state so undo-reject can faithfully re-stage it
+        # (restore the VIP — reactivating it if a rejected create soft-deleted it).
+        pending_state = await _capture_vip_pending_state(conn, vip_id)
+        metadata = json.dumps({"pending_state": pending_state}) if pending_state else None
+        for cid in cluster_ids:
+            # Collapse repeated pre-apply edits to a single PENDING row per VIP+cluster.
+            await conn.execute(
+                "DELETE FROM config_versions WHERE cluster_id=$1 AND status='PENDING' "
+                "AND version_name LIKE $2", cid, f"vip-{vip_id}-%")
+            await conn.execute("""
+                INSERT INTO config_versions
+                    (cluster_id, version_name, config_content, checksum, created_by, is_active, status, metadata)
+                VALUES ($1,$2,$3,$4,$5,FALSE,'PENDING',$6::jsonb)
+            """, cid, version_name, content, checksum, created_by, metadata)
+    except Exception as e:  # noqa: BLE001 — versioning is a UI convenience, never block the op
+        logger.warning(f"_stage_vip_version({vip_id},{action}) failed: {e}")
+
+
+async def _capture_vip_pending_state(conn, vip_id: int):
+    """Snapshot a VIP's current (pending) field + member state for undo-reject. The VRRP
+    secret travels only as its encrypted token (never plaintext)."""
+    v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1", vip_id)
+    if not v:
+        return None
+    members = await conn.fetch(
+        "SELECT agent_id, network_interface, role, priority FROM vip_members WHERE vip_id=$1", vip_id)
+    return {
+        "vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
+                "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                "track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
+        "members": [{"agent_id": m["agent_id"], "network_interface": m["network_interface"],
+                     "role": m["role"], "priority": m["priority"]} for m in members],
+    }
+
+
+async def restore_vip_from_rejected_version(conn, version_id: int):
+    """Undo-reject for a vip-* version: re-stage the rejected change as PENDING from the
+    version's captured `pending_state`. Reactivates the VIP if a rejected create soft-deleted
+    it, or re-applies a rejected edit's members — preserving each remaining member's
+    last-applied snapshot so a running VIP is never torn down (T-1). Returns None on success
+    or a human-readable error string (e.g. the name/address/VRID was reused since reject).
+    """
+    row = await conn.fetchrow("SELECT version_name, metadata FROM config_versions WHERE id=$1", version_id)
+    if not row:
+        return "version not found"
+    m = re.match(r"vip-(\d+)-", row["version_name"] or "")
+    if not m:
+        return "not a VIP version"
+    vip_id = int(m.group(1))
+    # Undo the reject of a DELETE request → re-arm the staged deletion (the VIP keeps running
+    # until re-approved; nothing on the node changes). Requires the VIP to still be active.
+    if "-delete-" in (row["version_name"] or ""):
+        if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id):
+            return "the VIP is no longer active — nothing to re-stage for deletion"
+        await conn.execute(
+            "UPDATE vip_instances SET pending_delete=TRUE, last_config_status='PENDING', "
+            "updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+        await conn.execute(
+            "UPDATE config_versions SET status='PENDING', is_active=FALSE, updated_at=CURRENT_TIMESTAMP "
+            "WHERE version_name=$1 AND status='REJECTED'", row["version_name"])
+        return None
+    meta = row["metadata"]
+    meta = json.loads(meta) if isinstance(meta, str) else meta
+    ps = (meta or {}).get("pending_state")
+    if not ps:
+        return ("this VIP change predates undo support — re-create or re-edit the VIP "
+                "from the HA/VIP page")
+    if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1", vip_id):
+        return "the VIP no longer exists — re-create it from the HA/VIP page"
+    sv, sm = ps["vip"], ps.get("members", [])
+    # One-VIP-per-agent must STILL hold after reactivation: if a member was meanwhile added
+    # to another active VIP (while this one was rejected/soft-deleted), undoing here would
+    # create a double-membership — and since the delivery endpoint serves one VIP per agent,
+    # reactivating this (never-applied → not_configured) VIP would tear down the other live
+    # VIP on that node. Block it with a clear message (review round-3 FINDING 2).
+    member_ids = [m["agent_id"] for m in sm]
+    if member_ids:
+        dup = await conn.fetchrow(
+            "SELECT a.name AS agent_name, v.name AS vip_name FROM vip_members vm "
+            "JOIN vip_instances v ON v.id = vm.vip_id JOIN agents a ON a.id = vm.agent_id "
+            "WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE AND v.id <> $2 LIMIT 1",
+            member_ids, vip_id)
+        if dup:
+            return (f"node '{dup['agent_name']}' now belongs to active VIP '{dup['vip_name']}' — "
+                    f"remove it there first, then undo")
+    # Preserve the running (applied) snapshot for members that remain, so undo of an edit
+    # never momentarily flips a live node to not_configured (T-1).
+    prev = {r["agent_id"]: r for r in await conn.fetch(
+        "SELECT agent_id, applied_config_content, applied_config_hash, last_deploy_state, "
+        "last_deploy_message, last_deploy_hash, last_deploy_at FROM vip_members WHERE vip_id=$1", vip_id)}
+    try:
+        async with conn.transaction():
+            await conn.execute("""
+                UPDATE vip_instances SET name=$2, description=$3, virtual_ip=$4, prefix_length=$5,
+                    virtual_router_id=$6, advert_int=$7, use_unicast=$8, track_haproxy=$9,
+                    auth_pass_encrypted=$10, is_active=TRUE, last_config_status='PENDING',
+                    updated_at=CURRENT_TIMESTAMP
+                WHERE id=$1
+            """, vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
+                sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
+                sv.get("auth_pass_encrypted"))
+            await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+            for mm in sm:
+                o = prev.get(mm["agent_id"])
+                await conn.execute("""
+                    INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                        applied_config_content, applied_config_hash,
+                        last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at)
+                    VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
+                """, vip_id, mm["agent_id"], mm["network_interface"], mm["role"], mm["priority"],
+                    o["applied_config_content"] if o else None, o["applied_config_hash"] if o else None,
+                    o["last_deploy_state"] if o else None, o["last_deploy_message"] if o else None,
+                    o["last_deploy_hash"] if o else None, o["last_deploy_at"] if o else None)
+            # A multi-cluster pool stages one row per cluster under the SAME version_name;
+            # flip them all back to PENDING so every affected cluster's Apply Management shows
+            # the restored change (mirrors the SSL auto-undo). is_active stays FALSE (review MED-2).
+            await conn.execute(
+                "UPDATE config_versions SET status='PENDING', is_active=FALSE, "
+                "updated_at=CURRENT_TIMESTAMP WHERE version_name=$1 AND status='REJECTED'",
+                row["version_name"])
+    except Exception as e:  # noqa: BLE001
+        if "unique" in str(e).lower() or "duplicate" in str(e).lower():
+            return "the VIP's name, address or VRID was reused after it was rejected"
+        raise
+    return None
+
+
+async def _transition_vip_versions(conn, vip_id: int, new_status: str):
+    """Move this VIP's PENDING config_versions to APPLIED/REJECTED (is_active stays FALSE).
+    Called by the VIP apply/reject endpoints so the right-panel version tracks the VIP."""
+    try:
+        await conn.execute(
+            "UPDATE config_versions SET status=$2, is_active=FALSE, updated_at=CURRENT_TIMESTAMP "
+            "WHERE status='PENDING' AND version_name LIKE $1", f"vip-{vip_id}-%", new_status)
+    except Exception as e:  # noqa: BLE001
+        logger.warning(f"_transition_vip_versions({vip_id},{new_status}) failed: {e}")
+
+
+def _jsonb_list(v):
+    """agents.capabilities/network_interfaces are JSONB; asyncpg returns them as a raw
+    JSON string (no codec). Normalize to a Python list."""
+    if isinstance(v, list):
+        return v
+    if isinstance(v, str):
+        try:
+            parsed = json.loads(v)
+            return parsed if isinstance(parsed, list) else []
+        except Exception:  # noqa: BLE001
+            return []
+    return []
+
+
+def _capable(capabilities) -> bool:
+    return "keepalived_management" in _jsonb_list(capabilities)
+
+
+# ---------------------------------------------------------------------------
+# List / read
+# ---------------------------------------------------------------------------
+@router.get("")
+async def list_vips(cluster_id: Optional[int] = None, authorization: str = Header(None)):
+    """List VIPs with their members + live MASTER/BACKUP state (never the secret).
+
+    Optional cluster_id scopes to VIPs in that cluster's pool — used by the Apply
+    Management page (which is cluster-scoped) to surface pending VIP changes.
+    """
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        # Show active VIPs PLUS approved-but-still-tearing-down ones (is_active=FALSE with a
+        # member that hasn't acked 'disabled' yet) so the operator can TRACK a deletion through
+        # to completion; a VIP drops off only once every member has torn keepalived down.
+        # The teardown-tracking clause is gated on last_config_status='APPLIED' so it ONLY shows
+        # deletions APPROVED via the new flow — a VIP soft-deleted under the old immediate-delete
+        # (pre-1.7.2: is_active=FALSE, last_config_status='PENDING') is NOT resurfaced (backward
+        # compat). Rejected never-applied creates (also PENDING) are likewise excluded.
+        _visible = ("(v.is_active = TRUE OR (v.last_config_status = 'APPLIED' AND EXISTS ("
+                    "SELECT 1 FROM vip_members mm WHERE mm.vip_id = v.id "
+                    "AND mm.applied_config_hash IS NOT NULL "
+                    "AND mm.last_deploy_state IS DISTINCT FROM 'disabled')))")
+        if cluster_id is not None:
+            vips = await conn.fetch(f"""
+                SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                       v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                       v.is_active, v.last_config_status, v.pending_delete,
+                       (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                       v.created_at, v.updated_at, p.name AS pool_name
+                FROM vip_instances v
+                LEFT JOIN haproxy_cluster_pools p ON p.id = v.pool_id
+                WHERE {_visible}
+                  AND v.pool_id = (SELECT pool_id FROM haproxy_clusters WHERE id = $1)
+                ORDER BY v.name
+            """, cluster_id)
+        else:
+            vips = await conn.fetch(f"""
+                SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                       v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                       v.is_active, v.last_config_status, v.pending_delete,
+                       (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                       v.created_at, v.updated_at, p.name AS pool_name
+                FROM vip_instances v
+                LEFT JOIN haproxy_cluster_pools p ON p.id = v.pool_id
+                WHERE {_visible}
+                ORDER BY v.name
+            """)
+        result = []
+        for v in vips:
+            members = await conn.fetch("""
+                SELECT m.id, m.agent_id, m.network_interface, m.role, m.priority,
+                       m.applied_config_hash, m.last_deploy_state, m.last_deploy_hash, m.last_deploy_at,
+                       a.name AS agent_name, a.status AS agent_status,
+                       a.keepalive_state, a.keepalive_ip, a.capabilities,
+                       a.ip_address
+                FROM vip_members m
+                LEFT JOIN agents a ON a.id = m.agent_id
+                WHERE m.vip_id = $1
+                ORDER BY m.priority DESC
+            """, v["id"])
+            deploy_status, deploy_synced, deploy_total = _derive_deploy_status(
+                v["last_config_status"], members, v["pending_delete"], v["is_active"])
+            result.append({
+                **dict(v),
+                # Convergence-aware status for the table (issue #27 follow-up); the raw
+                # last_config_status is kept above for the PENDING gate / Apply button.
+                "deploy_status": deploy_status,
+                "deploy_synced": deploy_synced,
+                "deploy_total": deploy_total,
+                "members": [{
+                    "id": m["id"], "agent_id": m["agent_id"], "agent_name": m["agent_name"],
+                    "network_interface": m["network_interface"], "role": m["role"],
+                    "priority": m["priority"], "agent_status": m["agent_status"],
+                    "keepalive_state": m["keepalive_state"], "keepalive_ip": m["keepalive_ip"],
+                    "ip_address": str(m["ip_address"]) if m["ip_address"] else None,
+                    "keepalived_capable": _capable(m["capabilities"]),
+                    "last_deploy_state": m["last_deploy_state"],
+                    "last_deploy_at": m["last_deploy_at"].isoformat() if m["last_deploy_at"] else None,
+                } for m in members],
+            })
+        return {"vips": result}
+    except Exception as e:  # noqa: BLE001 — degrade to empty rather than 500 (B-7)
+        logger.error(f"list_vips failed: {e}")
+        return {"vips": []}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.get("/{vip_id}")
+async def get_vip(vip_id: int, authorization: str = Header(None)):
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("""
+            SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
+                   v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
+                   v.is_active, v.last_config_status,
+                   (v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
+                   v.created_at, v.updated_at
+            FROM vip_instances v WHERE v.id = $1 AND v.is_active = TRUE
+        """, vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        members = await conn.fetch("""
+            SELECT m.agent_id, m.network_interface, m.role, m.priority,
+                   m.last_deploy_state, m.last_deploy_at,
+                   a.name AS agent_name, a.keepalive_state, a.keepalive_ip
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id = $1 ORDER BY m.priority DESC
+        """, vip_id)
+        return {**dict(v), "members": [dict(m) for m in members]}
+    finally:
+        await close_database_connection(conn)
+
+
+# ---------------------------------------------------------------------------
+# Create / update / delete
+# ---------------------------------------------------------------------------
+async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_virtual_ip: str,
+                                         exclude_vip_id: Optional[int] = None):
+    """Members must belong to the VIP's pool; VIP must not collide with an agent IP
+    or another live VIP (T-4/SQL-1)."""
+    # pool membership
+    for m in members:
+        row = await conn.fetchrow("SELECT pool_id FROM agents WHERE id=$1", m.agent_id)
+        if not row:
+            raise HTTPException(status_code=400, detail=f"agent {m.agent_id} not found")
+        if row["pool_id"] != pool_id:
+            raise HTTPException(status_code=400,
+                                detail=f"agent {m.agent_id} is not in pool {pool_id}")
+    # One active VIP per agent (v1): a node deploys a single keepalived.conf, and the
+    # delivery endpoint serves one VIP per agent — a second active membership would never
+    # converge (stuck SYNCING) instead of erroring. Reject it up front (review MED-2).
+    agent_ids = [m.agent_id for m in members]
+    if agent_ids:
+        dq = ("SELECT a.name AS agent_name, v.name AS vip_name "
+              "FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
+              "JOIN agents a ON a.id = vm.agent_id "
+              "WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE")
+        dparams = [agent_ids]
+        if exclude_vip_id is not None:
+            dq += " AND v.id <> $2"
+            dparams.append(exclude_vip_id)
+        dup = await conn.fetchrow(dq + " LIMIT 1", *dparams)
+        if dup:
+            raise HTTPException(status_code=409,
+                                detail=f"node '{dup['agent_name']}' is already a member of VIP "
+                                       f"'{dup['vip_name']}' — a node can belong to only one VIP")
+    # VIP must not be an existing agent's primary IP (INET cast, SQL-1)
+    clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", vip_virtual_ip)
+    if clash:
+        raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already a node's IP")
+    # …or another live VIP's address
+    q = "SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE"
+    params = [vip_virtual_ip]
+    if exclude_vip_id is not None:
+        q += " AND id <> $2"
+        params.append(exclude_vip_id)
+    if await conn.fetchval(q, *params):
+        raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already used by another VIP")
+
+
+@router.post("")
+async def create_vip(payload: VIPCreate, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "create")
+    conn = await get_database_connection()
+    try:
+        pool = await conn.fetchrow("SELECT id FROM haproxy_cluster_pools WHERE id=$1", payload.pool_id)
+        if not pool:
+            raise HTTPException(status_code=400, detail=f"pool {payload.pool_id} not found")
+        await _validate_members_against_pool(conn, payload.pool_id, payload.members, payload.virtual_ip)
+
+        enc = encrypt_vrrp_secret(payload.auth_pass) if payload.auth_pass else None
+        # Allocate VRID + insert, retrying once on a unique-violation race (B-1).
+        last_err = None
+        for _attempt in range(2):
+            vrid = await _alloc_free_vrid(conn, payload.pool_id, payload.virtual_router_id)
+            try:
+                async with conn.transaction():
+                    vip_id = await conn.fetchval("""
+                        INSERT INTO vip_instances
+                          (name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
+                           advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
+                           is_active, last_config_status, created_by)
+                        VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11)
+                        RETURNING id
+                    """, payload.name, payload.description, payload.pool_id, payload.virtual_ip,
+                        payload.prefix_length, vrid, payload.advert_int, enc,
+                        payload.use_unicast, payload.track_haproxy, current_user["id"])
+                    for m in payload.members:
+                        await conn.execute("""
+                            INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority)
+                            VALUES ($1,$2,$3,$4,$5)
+                        """, vip_id, m.agent_id, m.network_interface, m.role, m.priority)
+                last_err = None
+                break
+            except Exception as ie:  # noqa: BLE001
+                if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                    last_err = ie
+                    if payload.virtual_router_id is not None:
+                        raise HTTPException(status_code=409, detail="VIP name/address/VRID already in use")
+                    continue  # auto-VRID race → retry allocation
+                raise
+        if last_err is not None:
+            raise HTTPException(status_code=409, detail="VIP create conflict (name/address/VRID)")
+
+        # Stage a standard PENDING config_version so the change shows in Apply Management.
+        await _stage_vip_version(conn, vip_id, "create", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="create", resource_type="vip",
+            resource_id=str(vip_id),
+            details={"name": payload.name, "virtual_ip": payload.virtual_ip,
+                     "pool_id": payload.pool_id, "vrid": vrid, "members": len(payload.members)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"id": vip_id, "message": "VIP created (PENDING — apply from Apply Management)"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.put("/{vip_id}")
+async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "update")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        new_ip = payload.virtual_ip or v["virtual_ip"]
+        members = payload.members if payload.members is not None else None
+        if members is not None:
+            await _validate_members_against_pool(conn, v["pool_id"], members, new_ip, exclude_vip_id=vip_id)
+        elif payload.virtual_ip:
+            # Address-only change: re-check the VIP isn't a node IP or another live VIP.
+            clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address=$1::inet LIMIT 1", new_ip)
+            if clash:
+                raise HTTPException(status_code=409, detail=f"{new_ip} is already a node's IP")
+            dup = await conn.fetchval(
+                "SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE AND id<>$2", new_ip, vip_id)
+            if dup:
+                raise HTTPException(status_code=409, detail=f"{new_ip} is already used by another VIP")
+
+        enc_set = payload.auth_pass is not None and payload.auth_pass != ""
+        try:
+            async with conn.transaction():
+                await conn.execute("""
+                    UPDATE vip_instances SET
+                        name = COALESCE($2, name),
+                        description = COALESCE($3, description),
+                        virtual_ip = COALESCE($4, virtual_ip),
+                        prefix_length = COALESCE($5, prefix_length),
+                        virtual_router_id = COALESCE($6, virtual_router_id),
+                        advert_int = COALESCE($7, advert_int),
+                        use_unicast = COALESCE($8, use_unicast),
+                        track_haproxy = COALESCE($9, track_haproxy),
+                        auth_pass_encrypted = CASE WHEN $10 THEN $11 ELSE auth_pass_encrypted END,
+                        last_config_status = 'PENDING',
+                        -- Editing a VIP means you are KEEPING and changing it, so it cancels any
+                        -- staged deletion (otherwise a later Apply would delete instead of applying
+                        -- the edit). The edit then becomes the pending change to approve.
+                        pending_delete = FALSE,
+                        purge_on_teardown = FALSE,
+                        updated_at = CURRENT_TIMESTAMP
+                    WHERE id = $1
+                """, vip_id, payload.name, payload.description, payload.virtual_ip,
+                    payload.prefix_length, payload.virtual_router_id, payload.advert_int,
+                    payload.use_unicast, payload.track_haproxy,
+                    enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None)
+                if members is not None:
+                    # T-1: preserve the APPLIED snapshot + deploy ack for members that REMAIN,
+                    # so a pending member edit never momentarily flips a running node to
+                    # not_configured (which would self-heal-teardown a live VIP). The new
+                    # config only goes live on the next Apply. A REMOVED member loses its row
+                    # → delivery returns not_configured → marker self-heal teardown (correct:
+                    # it's no longer part of the VIP). A NEW member starts with a NULL snapshot.
+                    prev = {r["agent_id"]: r for r in await conn.fetch(
+                        "SELECT agent_id, applied_config_content, applied_config_hash, "
+                        "last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at "
+                        "FROM vip_members WHERE vip_id=$1", vip_id)}
+                    await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+                    for m in members:
+                        o = prev.get(m.agent_id)
+                        await conn.execute("""
+                            INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                                applied_config_content, applied_config_hash,
+                                last_deploy_state, last_deploy_message, last_deploy_hash, last_deploy_at)
+                            VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
+                        """, vip_id, m.agent_id, m.network_interface, m.role, m.priority,
+                            o["applied_config_content"] if o else None,
+                            o["applied_config_hash"] if o else None,
+                            o["last_deploy_state"] if o else None,
+                            o["last_deploy_message"] if o else None,
+                            o["last_deploy_hash"] if o else None,
+                            o["last_deploy_at"] if o else None)
+        except HTTPException:
+            raise
+        except Exception as ie:  # noqa: BLE001
+            if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                raise HTTPException(status_code=409, detail="VIP name/address/VRID already in use")
+            raise
+        # Re-stage the standard PENDING config_version reflecting the edited config.
+        await _stage_vip_version(conn, vip_id, "update", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="update", resource_type="vip",
+            resource_id=str(vip_id), details={"vip_id": vip_id},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": "VIP updated (PENDING — apply from Apply Management)"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.delete("/{vip_id}")
+async def delete_vip(vip_id: int, request: Request, purge_package: bool = False,
+                     authorization: str = Header(None)):
+    """Request VIP deletion — APPROVAL-GATED for safety.
+
+    Deleting a *running* (already-applied) VIP does NOT take effect immediately: it is STAGED
+    for Apply Management (pending_delete=TRUE + a vip-*-delete version) and the VIP keeps
+    running — is_active stays TRUE, agents keep serving it, NOTHING is torn down — until the
+    operator APPROVES the deletion. Rejecting it leaves the VIP running, untouched. Only the
+    approval flips is_active=FALSE and lets the agents tear keepalived down. So a misclick can
+    never tear down a production VIP, and an agent never deletes without an explicit approval.
+
+    A VIP that was NEVER applied (not deployed to any node) is removed immediately — there is
+    nothing running to tear down. purge_package opts into uninstalling the keepalived package
+    on teardown (default keeps it) and is honoured only once the deletion is approved, and only
+    on nodes where WE installed it (the agent's install marker), never an admin's package.
+    """
+    current_user = await _require(authorization, "delete")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow(
+            "SELECT id, name, applied_snapshot FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+
+        if v["applied_snapshot"] is None:
+            # Never deployed to any node — removing it affects nothing, so do it at once.
+            await conn.execute(
+                "UPDATE vip_instances SET is_active=FALSE, last_config_status='PENDING', "
+                "pending_delete=FALSE, purge_on_teardown=$2, updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+                vip_id, bool(purge_package))
+            await conn.execute(
+                "DELETE FROM config_versions WHERE status='PENDING' AND version_name LIKE $1",
+                f"vip-{vip_id}-%")
+            await log_user_activity(
+                user_id=current_user["id"], action="delete", resource_type="vip",
+                resource_id=str(vip_id), details={"name": v["name"], "never_applied": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "VIP removed — it was never applied, so no node was affected.",
+                    "staged": False}
+
+        # Running VIP → STAGE the deletion for approval. is_active stays TRUE (no teardown yet);
+        # the agent keeps serving the VIP until the operator approves in Apply Management.
+        await conn.execute(
+            "UPDATE vip_instances SET pending_delete=TRUE, purge_on_teardown=$2, "
+            "last_config_status='PENDING', updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+            vip_id, bool(purge_package))
+        await _stage_vip_version(conn, vip_id, "delete", current_user["id"])
+        await log_user_activity(
+            user_id=current_user["id"], action="delete-requested", resource_type="vip",
+            resource_id=str(vip_id), details={"name": v["name"], "purge_package": bool(purge_package)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": ("Deletion staged for approval — the VIP keeps running until you APPROVE it "
+                            "in Apply Management; reject to keep it. Nothing changes on the node until "
+                            "you approve."),
+                "staged": True, "purge_package": bool(purge_package)}
+    finally:
+        await close_database_connection(conn)
+
+
+# ---------------------------------------------------------------------------
+# Apply (isolated) + status
+# ---------------------------------------------------------------------------
+@router.post("/{vip_id}/apply")
+async def apply_vip(vip_id: int, request: Request, authorization: str = Header(None)):
+    current_user = await _require(authorization, "apply")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        # APPROVED DELETION: if a deletion was staged for this VIP, approving it here performs the
+        # actual delete — flip is_active=FALSE so the agents tear keepalived down on their next
+        # poll (honouring purge_on_teardown). Until this moment the VIP kept running untouched, so
+        # the teardown happens ONLY after this explicit human approval.
+        if v["pending_delete"]:
+            async with conn.transaction():
+                await conn.execute(
+                    "UPDATE vip_instances SET is_active=FALSE, pending_delete=FALSE, "
+                    "last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await _transition_vip_versions(conn, vip_id, "APPLIED")
+            await log_user_activity(
+                user_id=current_user["id"], action="delete", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "approved_delete": True,
+                         "purge_package": bool(v["purge_on_teardown"])},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            msg = "VIP deletion approved — member nodes will stop keepalived and release the VIP on their next poll"
+            if v["purge_on_teardown"]:
+                msg += "; the keepalived package will be uninstalled on nodes where we installed it"
+            return {"message": msg, "deleted": True}
+        members = await conn.fetch("""
+            SELECT m.id, m.agent_id, m.network_interface, m.role, m.priority, a.ip_address
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id = $1
+        """, vip_id)
+        if len(members) < 1:
+            raise HTTPException(status_code=400, detail="VIP needs at least 1 member")
+        masters = [m for m in members if m["role"] == "MASTER"]
+        if len(masters) != 1:
+            raise HTTPException(status_code=400, detail="exactly one member must be MASTER")
+        if any(m["ip_address"] is None for m in members):
+            raise HTTPException(status_code=400,
+                                detail="every member must have a reported IP before apply (unicast peers)")
+        master_prio = masters[0]["priority"]
+        if any(m["role"] == "BACKUP" and m["priority"] >= master_prio for m in members):
+            raise HTTPException(status_code=400, detail="MASTER priority must exceed every BACKUP")
+
+        auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
+        # If a secret is set but can't be decrypted (SECRET_KEY/VIP_ENCRYPTION_KEY rotated or
+        # drifted between pods), FAIL the apply rather than silently rendering a config with NO
+        # VRRP authentication — that would be a silent security downgrade and a guaranteed
+        # MASTER/BACKUP auth mismatch with any node still holding the old config (review MED-1).
+        if v["auth_pass_encrypted"] and not auth_plain:
+            raise HTTPException(status_code=409,
+                                detail="VRRP secret could not be decrypted (encryption key changed?) — "
+                                       "re-enter the VRRP secret on the VIP, then apply again")
+        check_script = build_haproxy_check_script() if v["track_haproxy"] else ""
+        member_dicts = [{"role": m["role"], "priority": m["priority"],
+                         "network_interface": m["network_interface"],
+                         "agent_id": m["agent_id"],
+                         "ip_address": str(m["ip_address"])} for m in members]
+        vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
+                    "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                    "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                    "track_haproxy": v["track_haproxy"]}
+
+        snapshot_members = []
+        async with conn.transaction():
+            for m in members:
+                this_agent = next(d for d in member_dicts if d["agent_id"] == m["agent_id"])
+                peer_ips = [d["ip_address"] for d in member_dicts if d["agent_id"] != m["agent_id"]]
+                conf = render_keepalived_conf(vip=vip_dict, members=member_dicts,
+                                              this_agent=this_agent, peer_ips=peer_ips,
+                                              auth_pass_plain=auth_plain)
+                chash = _md5(conf)
+                await conn.execute("""
+                    UPDATE vip_members
+                    SET applied_config_content=$2, applied_config_hash=$3, updated_at=CURRENT_TIMESTAMP
+                    WHERE id=$1
+                """, m["id"], conf, chash)
+                snapshot_members.append({
+                    "agent_id": m["agent_id"], "network_interface": m["network_interface"],
+                    "role": m["role"], "priority": m["priority"],
+                    "applied_config_content": conf, "applied_config_hash": chash})
+            # Capture the field-level applied state so a later pending edit can be REJECTED
+            # and fully reverted to exactly this state (auth secret stored encrypted, never plain).
+            applied_snapshot = {
+                "vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
+                        "prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
+                        "advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
+                        "track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
+                "members": snapshot_members}
+            await conn.execute(
+                "UPDATE vip_instances SET last_config_status='APPLIED', "
+                "applied_snapshot=$2::jsonb, updated_at=CURRENT_TIMESTAMP WHERE id=$1",
+                vip_id, json.dumps(applied_snapshot))
+
+        # Move the standard config_version PENDING → APPLIED (stays is_active=FALSE so it is
+        # never served as haproxy.cfg). Keeps the right-panel version in lockstep with the VIP.
+        await _transition_vip_versions(conn, vip_id, "APPLIED")
+        await log_user_activity(
+            user_id=current_user["id"], action="apply", resource_type="vip",
+            resource_id=str(vip_id),
+            details={"name": v["name"], "virtual_ip": v["virtual_ip"], "members": len(members)},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        # check_script is rendered but not stored on the vip row; the agent gets it via
+        # the delivery endpoint (which rebuilds it). Returned here only for visibility.
+        return {"message": "VIP applied — agents will converge on next poll",
+                "members_rendered": len(members), "tracks_haproxy": bool(check_script)}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.post("/{vip_id}/reject")
+async def reject_vip(vip_id: int, request: Request, authorization: str = Header(None)):
+    """Discard a VIP's PENDING changes and fully restore the last-APPLIED state — the
+    isolated equivalent of the product's reject -> restore-to-previous. Restores both the
+    vip_instances fields and the exact member set (with their delivered config snapshots)
+    from `applied_snapshot`, so the agents keep running what they already have (no churn).
+    A never-applied PENDING VIP (no snapshot) is SOFT-deleted (is_active=FALSE) and its
+    staged version marked REJECTED — so the change is reversible from the Rejected tab
+    (undo-reject reactivates it). Never touches the global haproxy.cfg apply flow.
+    """
+    current_user = await _require(authorization, "update")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT * FROM vip_instances WHERE id=$1 AND is_active=TRUE", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        # REJECT A STAGED DELETION: cancel it — the VIP keeps running exactly as before (it was
+        # never touched; is_active was never flipped). Clear the delete + purge intent and mark
+        # the staged version REJECTED. This is the "nothing happened" path the operator expects.
+        if v["pending_delete"]:
+            await conn.execute(
+                "UPDATE vip_instances SET pending_delete=FALSE, purge_on_teardown=FALSE, "
+                "last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await _transition_vip_versions(conn, vip_id, "REJECTED")
+            await log_user_activity(
+                user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "delete_cancelled": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "Deletion rejected — the VIP keeps running unchanged."}
+        if v["last_config_status"] != "PENDING":
+            return {"message": "Nothing to reject — no pending changes"}
+
+        snap_raw = v["applied_snapshot"]
+        snap = json.loads(snap_raw) if isinstance(snap_raw, str) else snap_raw
+        if not snap:
+            # Created but never applied -> reject SOFT-deletes the VIP (is_active=FALSE) and
+            # marks its staged version REJECTED. The row + members are kept so undo-reject can
+            # reactivate the exact VIP (no orphan). The partial unique indexes free its
+            # name/address/VRID for reuse while it's inactive.
+            await _transition_vip_versions(conn, vip_id, "REJECTED")
+            await conn.execute(
+                "UPDATE vip_instances SET is_active=FALSE, last_config_status='PENDING', "
+                "updated_at=CURRENT_TIMESTAMP WHERE id=$1", vip_id)
+            await log_user_activity(
+                user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+                details={"name": v["name"], "discarded": True},
+                ip_address=_client_ip(request), user_agent=_user_agent(request))
+            return {"message": "Pending VIP rejected (undo from the Rejected tab to restore it)"}
+
+        sv = snap["vip"]
+        sm = snap.get("members", [])
+        try:
+            async with conn.transaction():
+                await conn.execute("""
+                    UPDATE vip_instances SET
+                        name=$2, description=$3, virtual_ip=$4, prefix_length=$5, virtual_router_id=$6,
+                        advert_int=$7, use_unicast=$8, track_haproxy=$9, auth_pass_encrypted=$10,
+                        last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP
+                    WHERE id=$1
+                """, vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
+                    sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
+                    sv.get("auth_pass_encrypted"))
+                await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
+                for m in sm:
+                    await conn.execute("""
+                        INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority,
+                            applied_config_content, applied_config_hash)
+                        VALUES ($1,$2,$3,$4,$5,$6,$7)
+                    """, vip_id, m["agent_id"], m["network_interface"], m["role"], m["priority"],
+                        m.get("applied_config_content"), m.get("applied_config_hash"))
+        except HTTPException:
+            raise
+        except Exception as ie:  # noqa: BLE001
+            if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
+                raise HTTPException(status_code=409,
+                                    detail="Cannot restore — the previous address/VRID was taken in the meantime")
+            raise
+        # Mark the standard config_version REJECTED (history); the VIP is back to APPLIED.
+        await _transition_vip_versions(conn, vip_id, "REJECTED")
+        await log_user_activity(
+            user_id=current_user["id"], action="reject", resource_type="vip", resource_id=str(vip_id),
+            details={"name": v["name"], "restored": True},
+            ip_address=_client_ip(request), user_agent=_user_agent(request))
+        return {"message": "Pending changes rejected — VIP restored to its last applied state"}
+    finally:
+        await close_database_connection(conn)
+
+
+@router.get("/{vip_id}/status")
+async def vip_status(vip_id: int, authorization: str = Header(None)):
+    await _require(authorization, "read")
+    conn = await get_database_connection()
+    try:
+        v = await conn.fetchrow("SELECT id, name, is_active, last_config_status FROM vip_instances WHERE id=$1", vip_id)
+        if not v:
+            raise HTTPException(status_code=404, detail="VIP not found")
+        members = await conn.fetch("""
+            SELECT m.agent_id, m.role, m.priority, m.network_interface,
+                   m.last_deploy_state, m.last_deploy_message, m.last_deploy_at, m.applied_config_hash,
+                   a.name AS agent_name, a.keepalive_state, a.keepalive_ip, a.status AS agent_status,
+                   a.capabilities
+            FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
+            WHERE m.vip_id=$1 ORDER BY m.priority DESC
+        """, vip_id)
+        out = []
+        for m in members:
+            applied = m["applied_config_hash"]
+            deploy = m["last_deploy_state"]
+            capable = _capable(m["capabilities"])
+            if applied and not deploy:
+                converge = "awaiting agent (upgrade may be required)" if not capable else "converging"
+            else:
+                converge = deploy or "pending"
+            out.append({
+                "agent_name": m["agent_name"], "role": m["role"], "priority": m["priority"],
+                "network_interface": m["network_interface"],
+                "agent_status": m["agent_status"], "keepalive_state": m["keepalive_state"],
+                "keepalive_ip": m["keepalive_ip"], "keepalived_capable": capable,
+                "deploy_state": deploy, "deploy_message": m["last_deploy_message"],
+                "deploy_at": m["last_deploy_at"].isoformat() if m["last_deploy_at"] else None,
+                "convergence": converge,
+            })
+        return {"id": v["id"], "name": v["name"], "is_active": v["is_active"],
+                "last_config_status": v["last_config_status"], "members": out}
+    finally:
+        await close_database_connection(conn)
+
+
+# NOTE: the keepalived config preview/diff is served by the STANDARD config-version diff
+# endpoint (cluster.py get_config_version_diff, vip-* branch) via render_vip_config_masked
+# above — there is no bespoke VIP preview endpoint, so VIP changes use the product's
+# standard "View Change" like every other entity (issue #27 follow-up).

backend/services/acme_diagnostics.py

@@ -116,6 +116,10 @@ _PROBLEM_HUMANIZED: Dict[str, Dict[str, str]] = {
         "title": "HTTP-01 challenge response mismatch",
         "hint": "The CA fetched the challenge URL but received the wrong key authorization. Confirm the challenge was served from the right backend.",
     },
+    "urn:ietf:params:acme:error:externalAccountRequired": {
+        "title": "External Account Binding (EAB) required",
+        "hint": "This CA (e.g. ZeroSSL, Google) requires EAB. Enter the EAB Key ID and HMAC Key from your CA account when registering.",
+    },
     "urn:ietf:params:acme:error:invalidContact": {
         "title": "Invalid contact email",
         "hint": "The ACME account email is malformed. Update the LE account email.",
@@ -199,6 +203,23 @@ def humanize_error_detail(error_detail: Any) -> Dict[str, Any]:
     status = parsed.get("status")
     subproblems = parsed.get("subproblems") or []
 
+    # Issue #35: DNS-01 failures are recorded as {stage, reason, timestamp} (no RFC8555 "type"),
+    # so without this fallback the humanized alert would show a bare "ACME error" with no message.
+    # Surface the reason and a targeted hint so the operator knows exactly what to fix.
+    if not problem_type and parsed.get("reason"):
+        reason = str(parsed.get("reason"))
+        message = message or reason
+        title = "DNS-01 validation failed"
+        rlow = reason.lower()
+        if "decrypt" in rlow or "credential" in rlow:
+            hint = hint or "Re-enter the DNS provider credentials for this account in ACME Automation."
+        elif "zone" in rlow:
+            hint = hint or "Confirm the domain's DNS zone is managed by the configured provider and the token has access to it."
+        elif "deadline" in rlow or "expired" in rlow or "confirm" in rlow:
+            hint = hint or "The manual confirmation window passed. Create a new certificate request and publish the TXT record promptly."
+        else:
+            hint = hint or "Check the DNS TXT record and provider credentials, then retry."
+
     out = {
         "title": title,
         "message": message,
@@ -694,6 +715,7 @@ async def run_checks(
     cluster_ids: List[int],
     account_id: Optional[int],
     only: Optional[List[str]] = None,
+    challenge_type: str = "http-01",
 ) -> List[Dict[str, Any]]:
     """Execute the full pre-flight check suite. `only` lets callers re-run a
     subset (per-check rerun in the UI).
@@ -714,12 +736,29 @@ async def run_checks(
     except (TypeError, ValueError):
         safe_account_id = None
 
+    # Issue #35: DNS-01 validates via a TXT record, so the HTTP-01 reachability checks
+    # (public A record, inbound port 80, ACME Challenge Routing) do not apply — report them
+    # as `skipped` rather than failing an internal/isolated host that is actually fine.
+    is_dns01 = (challenge_type == "dns-01")
     if "dns" in selected:
-        results.append(await _safe_check("dns", "DNS resolution", check_dns(safe_domains)))
+        if is_dns01:
+            results.append(_check_result("dns", "DNS resolution", "skipped",
+                                         "DNS-01: a public A record is not required (validation is via a TXT record).",
+                                         severity="info"))
+        else:
+            results.append(await _safe_check("dns", "DNS resolution", check_dns(safe_domains)))
     if "port80" in selected:
-        results.append(await _safe_check("port80", "Port 80 reachability", check_port80(safe_domains)))
+        if is_dns01:
+            results.append(_check_result("port80", "Port 80 reachability", "skipped",
+                                         "DNS-01: inbound port 80 is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("port80", "Port 80 reachability", check_port80(safe_domains)))
     if "routing" in selected:
-        results.append(await _safe_check("routing", "HAProxy routing", check_routing(conn, safe_domains, safe_cluster_ids)))
+        if is_dns01:
+            results.append(_check_result("routing", "HAProxy routing", "skipped",
+                                         "DNS-01: ACME Challenge Routing is not required.", severity="info"))
+        else:
+            results.append(await _safe_check("routing", "HAProxy routing", check_routing(conn, safe_domains, safe_cluster_ids)))
     if "account" in selected:
         results.append(await _safe_check("account", "ACME account", check_account(conn, safe_account_id)))
     if "agents" in selected:

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -71,17 +75,21 @@ class ACMEService:
                     raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
                 data = await resp.json()
                 if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
+                    self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
                 data['_fetched_at'] = time.time()
                 self._directory_cache[directory_url] = data
                 return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
-            nonce = self._nonce
-            self._nonce = None
-            return nonce
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         directory = await self.get_directory(directory_url)
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         async with aiohttp.ClientSession() as session:
             async with session.head(directory['newNonce']) as resp:
                 return resp.headers['Replay-Nonce']
@@ -128,6 +136,20 @@ class ACMEService:
         digest = hashlib.sha256(ordered.encode('utf-8')).digest()
         return _b64url(digest)
 
+    @staticmethod
+    def _dns_txt_value(key_authorization: str) -> str:
+        """RFC 8555 §8.4: the DNS-01 TXT value is base64url(SHA256(key_authorization)) over the
+        RAW 32-byte digest (NOT the hexdigest)."""
+        return _b64url(hashlib.sha256(key_authorization.encode('utf-8')).digest())
+
+    @staticmethod
+    def _challenge_dns_name(identifier: str) -> str:
+        """The `_acme-challenge.<base>` record name for an ACME identifier. A leading wildcard
+        `*.` is stripped, so both `*.example.com` and bare `example.com` map to the SAME name
+        `_acme-challenge.example.com` (which is why apex+wildcard need two coexisting TXT values)."""
+        base = identifier[2:] if identifier.startswith('*.') else identifier
+        return f"_acme-challenge.{base}"
+
     def _sign_jws(self, private_key, protected: dict, payload: Any) -> dict:
         protected_b64 = _b64url(json.dumps(protected).encode('utf-8'))
         if payload == "":
@@ -174,11 +196,17 @@ class ACMEService:
                     timeout=aiohttp.ClientTimeout(total=30),
                 ) as resp:
                     if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
+                    if resp.status == 400 and attempt < 2:
                         err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
+                        etype = (err.get('type') or '')
+                        edetail = (err.get('detail') or '').lower()
+                        # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                        # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                        # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                        # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                        if etype.endswith('badNonce') or 'nonce' in edetail:
                             nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
                             protected['nonce'] = nonce
                             body = self._sign_jws(private_key, protected, payload)
@@ -208,6 +236,8 @@ class ACMEService:
         tos_agreed: bool = True,
         eab_kid: Optional[str] = None,
         eab_hmac_key: Optional[str] = None,
+        challenge_type: str = 'http-01',
+        dns_provider: Optional[str] = None,
     ) -> dict:
         directory = await self.get_directory(directory_url)
         pem, jwk = self._generate_account_key()
@@ -250,13 +280,14 @@ class ACMEService:
         conn = await get_database_connection()
         try:
             row = await conn.fetchrow("""
-                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                INSERT INTO letsencrypt_accounts (email, directory_url, account_url, jwk_private_key, status, tos_agreed, eab_kid, challenge_type, dns_provider)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 ON CONFLICT (email, directory_url) DO UPDATE SET
-                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6, updated_at = NOW()
-                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at
+                    account_url = $3, jwk_private_key = $4, status = $5, tos_agreed = $6,
+                    challenge_type = $8, dns_provider = $9, updated_at = NOW()
+                RETURNING id, email, directory_url, account_url, status, tos_agreed, created_at, challenge_type, dns_provider
             """, email, directory_url, account_url, pem,
-                data.get('status') or 'valid', tos_agreed, eab_kid)
+                data.get('status') or 'valid', tos_agreed, eab_kid, challenge_type, dns_provider)
             return dict(row)
         finally:
             await close_database_connection(conn)
@@ -302,8 +333,10 @@ class ACMEService:
         account_id: int,
         domains: List[str],
         cluster_ids: Optional[List[int]] = None,
+        challenge_type: str = 'http-01',
+        created_by: Optional[int] = None,
     ) -> dict:
-        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}")
+        logger.info(f"ACME: Creating order for domains={domains}, account_id={account_id}, challenge_type={challenge_type}")
         conn = await get_database_connection()
         try:
             account = await conn.fetchrow(
@@ -341,12 +374,12 @@ class ACMEService:
 
             order_row = await conn.fetchrow("""
                 INSERT INTO letsencrypt_orders
-                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids)
-                VALUES ($1, $2, $3, $4, $5, $6, $7)
+                    (account_id, order_url, status, domains, finalize_url, expires_at, cluster_ids, challenge_type, created_by)
+                VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
                 RETURNING id
             """, account_id, order_url, data.get('status') or 'pending',
                 json.dumps(domains), data.get('finalize') or '', expires_at,
-                json.dumps(cluster_ids or []))
+                json.dumps(cluster_ids or []), challenge_type, created_by)
 
             order_id = order_row['id']
 
@@ -383,18 +416,24 @@ class ACMEService:
                 domain = (auth_data.get('identifier') or {}).get('value', '')
                 http01_for_domain = False
                 for challenge in (auth_data.get('challenges') or []):
-                    if challenge.get('type') == 'http-01':
+                    # Store only the challenge of the CHOSEN method (default 'http-01' keeps the
+                    # existing behaviour byte-identical; 'dns-01' selects the TXT challenge instead).
+                    if challenge.get('type') == challenge_type:
                         token = challenge['token']
                         jwk = self._get_jwk(private_key)
                         thumbprint = self._jwk_thumbprint(jwk)
                         key_auth = f"{token}.{thumbprint}"
+                        dns_txt = self._dns_txt_value(key_auth) if challenge_type == 'dns-01' else None
 
                         await conn.execute("""
-                            INSERT INTO acme_challenges (order_id, domain, token, key_authorization, challenge_url, status)
-                            VALUES ($1, $2, $3, $4, $5, $6)
+                            INSERT INTO acme_challenges
+                                (order_id, domain, token, key_authorization, challenge_url, status,
+                                 challenge_type, dns_txt_value)
+                            VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
                         """, order_id, domain, token, key_auth,
-                            challenge.get('url') or '', challenge.get('status') or 'pending')
-                        logger.info(f"ACME: Challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
+                            challenge.get('url') or '', challenge.get('status') or 'pending',
+                            challenge_type, dns_txt)
+                        logger.info(f"ACME: {challenge_type} challenge stored for domain={domain}, token={token[:20]}..., challenge_url={(challenge.get('url') or '')[:60]}")
                         http01_for_domain = True
                 if http01_for_domain and domain:
                     domains_with_http01.add(domain)
@@ -413,7 +452,7 @@ class ACMEService:
                     error_payload, order_id
                 )
                 raise Exception(
-                    f"ACME order {order_id} created but no http-01 challenges available "
+                    f"ACME order {order_id} created but no {challenge_type} challenges available "
                     f"(auth fetch failures: {len(auth_fetch_failures)}). See order.error_detail for diagnostics."
                 )
             elif auth_fetch_failures:
@@ -449,10 +488,15 @@ class ACMEService:
         try:
             # Issue #12 / Commit 5a: include 'failed' challenges so they can be retried,
             # but rate-limit per challenge: max 5 attempts in last 5 minutes.
+            # DNS-01 skip-gate (the single safe choke point): never POST a challenge response for a
+            # dns-01 row whose TXT record has not been published yet — that would make the CA validate
+            # against a missing record and burn the order. http-01 rows (challenge_type 'http-01'/NULL)
+            # are never excluded, so the existing flow is byte-identical.
             challenges = await conn.fetch(
                 """SELECT * FROM acme_challenges
                    WHERE order_id = $1
-                     AND (status IN ('pending', 'failed') OR status IS NULL)""",
+                     AND (status IN ('pending', 'failed') OR status IS NULL)
+                     AND NOT (COALESCE(challenge_type, 'http-01') = 'dns-01' AND COALESCE(dns_record_published, FALSE) = FALSE)""",
                 order_id
             )
             order = await conn.fetchrow(

backend/services/dns01_orchestrator.py

@@ -0,0 +1,321 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): non-blocking per-cycle orchestration.
+
+Driven by the existing `complete_pending_acme_orders` background task (which already claims
+in-progress orders with `FOR UPDATE SKIP LOCKED`). For a dns-01 order this module advances AT MOST
+ONE step per 60s cycle — publish TXT, then (after a short min-age) respond — so the serial claim
+loop is never blocked by a multi-minute wait, and NO DNS library is needed (the CA is the source of
+truth; a propagation-lag `invalid` is recovered by a bounded fresh-order chain).
+
+Design invariants (from the hardening review):
+- Additive at the RRset level: publish/cleanup operate on a single (name, value), so wildcard+apex
+  (two values at one name) coexist.
+- No new order-status value: a failed order stays `invalid`; a boolean `dns01_retry_claimed` does the
+  winner-only CAS + claim exclusion, so existing `status` consumers are untouched.
+- Secrets (provider API tokens) are NEVER logged or written to error_detail/events.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta, timezone
+from typing import Dict, List, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+from services.acme_service import acme_service as acme_svc, ACMEService
+from services.dns_providers import get_provider, is_supported, DnsProviderError
+from utils.dns_credentials import decrypt_dns_credentials
+from utils.activity_log import record_event
+
+logger = logging.getLogger(__name__)
+
+# Tunables (kept conservative vs Let's Encrypt rate limits: 5 failed-validations/host/hour,
+# 300 new-orders/account/3h).
+PROPAGATION_GRACE_SECONDS = 25          # min age before we tell the CA to validate
+MANUAL_CONFIRM_TTL = timedelta(hours=48)
+MAX_RETRIES = 3                          # bounded fresh-order chain (1 original + 3 retries = 4 orders)
+# Retry backoff floor (minutes) indexed by the order's current dns01_attempts: [15, 30, 60].
+# The AUTHORITATIVE implementation is the SQL CASE in main.py's claim query
+# (complete_pending_acme_orders), so the backoff is evaluated atomically with the
+# FOR UPDATE SKIP LOCKED claim. Documented here only — do not reintroduce a second copy.
+
+
+def _now() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _aware(dt) -> Optional[datetime]:
+    if dt is None:
+        return None
+    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+
+
+async def _load_credentials(conn, account_id: int) -> Tuple[Optional[Dict[str, str]], bool]:
+    """Return (credentials_dict_or_None, row_exists). credentials None + row_exists True means the
+    stored token could not be decrypted (e.g. SECRET_KEY rotated)."""
+    row = await conn.fetchrow(
+        "SELECT credentials_encrypted FROM letsencrypt_account_dns_credentials WHERE account_id = $1",
+        account_id,
+    )
+    if not row:
+        return None, False
+    return decrypt_dns_credentials(row["credentials_encrypted"]), True
+
+
+async def _fail_order(conn, order_id: int, reason: str) -> None:
+    """Mark an order invalid with a sanitized reason (no secrets) + event."""
+    import json
+    payload = json.dumps({"stage": "dns01", "reason": reason, "timestamp": _now().isoformat()})
+    await conn.execute(
+        "UPDATE letsencrypt_orders SET status = 'invalid', error_detail = $1, updated_at = NOW() WHERE id = $2",
+        payload, order_id,
+    )
+    await record_event(order_id, "acme.dns01.validation", severity="ERROR", message=reason, conn=conn)
+
+
+async def advance_dns01_order(order_id: int) -> None:
+    """One non-blocking step for a claimed pending/processing dns-01 order. No-op for http-01 or
+    orders not in a publishable state. Safe to call every cycle (idempotent via CAS flags)."""
+    conn = await get_database_connection()
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.id, o.status, o.challenge_type, o.account_id, a.dns_provider
+               FROM letsencrypt_orders o JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if order["status"] not in ("pending", "processing"):
+            return
+
+        challenges = await conn.fetch(
+            "SELECT * FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'", order_id
+        )
+        if not challenges:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+
+        # --- Manual provider: the user publishes + confirms; we only enforce the deadline. ---
+        if provider_name == "manual" or not is_supported(provider_name):
+            deadline = _aware(challenges[0]["manual_confirm_deadline"])
+            if deadline is None:
+                new_deadline = _now() + MANUAL_CONFIRM_TTL
+                await conn.execute(
+                    "UPDATE acme_challenges SET manual_confirm_deadline = $1 WHERE order_id = $2 AND manual_confirm_deadline IS NULL",
+                    new_deadline, order_id,
+                )
+            elif _now() > deadline:
+                await _fail_order(conn, order_id,
+                                  "Manual DNS-01 confirmation deadline passed without confirmation.")
+            return  # respond happens via the dns-confirm endpoint
+
+        # --- Automated provider (e.g. Cloudflare). ---
+        creds, row_exists = await _load_credentials(conn, order["account_id"])
+        if not row_exists:
+            await _fail_order(conn, order_id,
+                              f"No DNS provider credentials configured for provider '{provider_name}'.")
+            return
+        if creds is None:
+            await _fail_order(conn, order_id,
+                              "DNS provider credentials could not be decrypted; re-enter them in Settings.")
+            return
+
+        provider = get_provider(provider_name, creds)
+
+        # Publish any not-yet-published challenge (CAS so two replicas can't double-publish).
+        for ch in challenges:
+            if ch["dns_record_published"]:
+                continue
+            flipped = await conn.fetchval(
+                """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = NOW()
+                   WHERE id = $1 AND dns_record_published = FALSE RETURNING id""",
+                ch["id"],
+            )
+            if not flipped:
+                continue
+            name = ACMEService._challenge_dns_name(ch["domain"])
+            try:
+                await provider.add_txt_record(name, ch["dns_txt_value"])
+                await record_event(order_id, "acme.dns01.publish",
+                                   message=f"Published TXT {name}; waiting for DNS propagation before asking the CA to validate.",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+            except DnsProviderError as exc:
+                # Revert so the next cycle retries the publish; keep the order pending. exc is sanitized.
+                await conn.execute(
+                    "UPDATE acme_challenges SET dns_record_published = FALSE, dns_published_at = NULL WHERE id = $1",
+                    ch["id"],
+                )
+                await record_event(order_id, "acme.dns01.publish", severity="WARNING",
+                                   message=f"Publish failed for {name}: {exc}",
+                                   details={"name": name, "provider": provider_name}, conn=conn)
+                return
+
+        # All published? Then respond once the min-age gate has elapsed (across cycles, no sleep).
+        rows = await conn.fetch(
+            "SELECT dns_record_published, dns_published_at FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+            order_id,
+        )
+        if any(not r["dns_record_published"] for r in rows):
+            return
+        published_ats = [_aware(r["dns_published_at"]) for r in rows if r["dns_published_at"]]
+        if not published_ats:
+            return
+        if (_now() - min(published_ats)).total_seconds() < PROPAGATION_GRACE_SECONDS:
+            return  # wait one more cycle
+
+        # Only POST the challenge response if something still needs validating — avoids re-POSTing
+        # every cycle (and bumping dns01_last_attempt_at) once the CA already has them processing.
+        still_pending = await conn.fetchval(
+            """SELECT 1 FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'
+               AND (status IN ('pending', 'failed') OR status IS NULL) LIMIT 1""",
+            order_id,
+        )
+        if not still_pending:
+            return
+
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Told the CA to validate the DNS-01 challenge(s).", conn=conn)
+    finally:
+        await close_database_connection(conn)
+
+
+async def confirm_manual_dns01(order_id: int) -> Dict:
+    """Called by POST /orders/{id}/dns-confirm for the manual provider: mark the TXT published and
+    tell the CA to validate. Returns a small status dict."""
+    conn = await get_database_connection()
+    try:
+        await conn.execute(
+            """UPDATE acme_challenges SET dns_record_published = TRUE, dns_published_at = COALESCE(dns_published_at, NOW())
+               WHERE order_id = $1 AND challenge_type = 'dns-01'""",
+            order_id,
+        )
+        await acme_svc.respond_to_challenges(order_id)
+        await conn.execute("UPDATE letsencrypt_orders SET dns01_last_attempt_at = NOW() WHERE id = $1", order_id)
+        await record_event(order_id, "acme.dns01.responded",
+                           message="Manual DNS-01 confirmed; told the CA to validate.", conn=conn)
+        return {"ok": True}
+    finally:
+        await close_database_connection(conn)
+
+
+async def retry_invalid_dns01(order_id: int) -> None:
+    """Bounded fresh-order recovery for a dns-01 order that went `invalid` (e.g. propagation lag).
+    Winner-only CAS on `dns01_retry_claimed`; cleans the old TXT, mints a child order. No-op for
+    http-01 or when the budget is exhausted."""
+    conn = await get_database_connection()
+    child_created = False
+    try:
+        order = await conn.fetchrow(
+            """SELECT o.*, a.dns_provider FROM letsencrypt_orders o
+               JOIN letsencrypt_accounts a ON o.account_id = a.id WHERE o.id = $1""",
+            order_id,
+        )
+        if not order or order["challenge_type"] != "dns-01":
+            return
+        if (order["dns01_attempts"] or 0) >= MAX_RETRIES:
+            return  # budget exhausted; stays terminal invalid
+
+        # Winner-only claim (closes the cross-replica double-mint race).
+        claimed = await conn.fetchval(
+            """UPDATE letsencrypt_orders SET dns01_retry_claimed = TRUE, updated_at = NOW()
+               WHERE id = $1 AND status = 'invalid' AND dns01_retry_claimed = FALSE RETURNING id""",
+            order_id,
+        )
+        if not claimed:
+            return
+
+        provider_name = (order["dns_provider"] or "manual").strip()
+        # Best-effort cleanup of this order's TXT before minting the replacement.
+        if provider_name != "manual" and is_supported(provider_name):
+            creds, _exists = await _load_credentials(conn, order["account_id"])
+            if creds:
+                provider = get_provider(provider_name, creds)
+                chs = await conn.fetch(
+                    "SELECT domain, dns_txt_value FROM acme_challenges WHERE order_id = $1 AND challenge_type = 'dns-01'",
+                    order_id,
+                )
+                for ch in chs:
+                    try:
+                        await provider.remove_txt_record(ACMEService._challenge_dns_name(ch["domain"]), ch["dns_txt_value"])
+                    except DnsProviderError:
+                        pass  # tolerate; the reconcile sweep will retry
+        await conn.execute(
+            "UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE order_id = $1 AND dns_record_published = TRUE",
+            order_id,
+        )
+
+        import json
+        domains = json.loads(order["domains"]) if isinstance(order["domains"], str) else (order["domains"] or [])
+        cluster_ids = json.loads(order["cluster_ids"]) if isinstance(order["cluster_ids"], str) else (order["cluster_ids"] or [])
+        next_attempts = (order["dns01_attempts"] or 0) + 1
+        child = await acme_svc.create_order(
+            order["account_id"], domains, cluster_ids, challenge_type="dns-01",
+            created_by=order["created_by"],
+        )
+        child_created = True
+        await conn.execute(
+            """UPDATE letsencrypt_orders
+               SET dns01_attempts = $1, dns01_parent_order_id = $2, dns01_last_attempt_at = NOW()
+               WHERE id = $3""",
+            next_attempts, order_id, child["order_id"],
+        )
+        await record_event(order_id, "acme.dns01.validation", severity="WARNING",
+                           message=f"DNS-01 order invalid; minted retry #{next_attempts} (order {child['order_id']}).",
+                           details={"child_order_id": child["order_id"], "attempt": next_attempts}, conn=conn)
+    except Exception as exc:  # noqa: BLE001
+        logger.error(f"[DNS01-RETRY] order {order_id}: {exc}")
+        # A transient failure (e.g. CA rate limit) BEFORE the child was minted must NOT permanently
+        # burn the retry slot — reset the claim so the next cycle can retry. If the child was already
+        # created, leave the claim set (resetting would double-mint).
+        if not child_created:
+            try:
+                await conn.execute(
+                    "UPDATE letsencrypt_orders SET dns01_retry_claimed = FALSE WHERE id = $1 AND status = 'invalid'",
+                    order_id,
+                )
+            except Exception:
+                pass
+    finally:
+        await close_database_connection(conn)
+
+
+async def reconcile_dns01_cleanup() -> None:
+    """Best-effort sweep that removes any TXT records left published for terminal orders (covers a
+    cleanup that failed, or the kill-switch being flipped off mid-flight). NOT gated by the
+    kill-switch. Runs once per completion cycle."""
+    conn = await get_database_connection()
+    try:
+        rows = await conn.fetch(
+            """SELECT c.id AS chal_id, c.order_id, c.domain, c.dns_txt_value, o.account_id, a.dns_provider
+               FROM acme_challenges c
+               JOIN letsencrypt_orders o ON c.order_id = o.id
+               JOIN letsencrypt_accounts a ON o.account_id = a.id
+               WHERE c.challenge_type = 'dns-01'
+                 AND c.dns_record_published = TRUE
+                 AND COALESCE(c.dns_record_cleaned, FALSE) = FALSE
+                 AND o.status IN ('valid', 'invalid', 'cancelled')
+               LIMIT 50""",
+        )
+        for r in rows:
+            provider_name = (r["dns_provider"] or "manual").strip()
+            if provider_name == "manual" or not is_supported(provider_name):
+                # Manual: nothing to call; mark cleaned so we stop revisiting.
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                continue
+            creds, _exists = await _load_credentials(conn, r["account_id"])
+            if creds is None:
+                continue  # can't clean without creds; leave for a later pass
+            provider = get_provider(provider_name, creds)
+            try:
+                await provider.remove_txt_record(ACMEService._challenge_dns_name(r["domain"]), r["dns_txt_value"])
+                await conn.execute("UPDATE acme_challenges SET dns_record_cleaned = TRUE WHERE id = $1", r["chal_id"])
+                await record_event(r["order_id"], "acme.dns01.cleanup",
+                                   message=f"Cleaned up TXT for {r['domain']}", conn=conn)
+            except DnsProviderError:
+                pass  # retry next sweep
+    except Exception as exc:  # noqa: BLE001
+        logger.debug(f"[DNS01-RECONCILE] skipped: {exc}")
+    finally:
+        await close_database_connection(conn)

backend/services/dns_providers/__init__.py

@@ -0,0 +1,14 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): pluggable DNS provider package.
+
+A small adapter layer so DNS-01 challenges can publish/clean up the
+`_acme-challenge.<domain>` TXT record via different DNS providers. The interface is
+additive at the RRset level (add/remove a single value by name+content, never
+overwrite-by-name) so multiple coexisting values at one name (wildcard + apex) work.
+
+MVP providers: manual (user publishes the TXT themselves) and Cloudflare. New providers
+plug in via the registry without touching the orchestration.
+"""
+from .base import DnsProvider, DnsProviderError
+from .registry import get_provider, list_providers, is_supported
+
+__all__ = ["DnsProvider", "DnsProviderError", "get_provider", "list_providers", "is_supported"]

backend/services/dns_providers/base.py

@@ -0,0 +1,53 @@
+"""Abstract DNS provider interface for ACME DNS-01 (Issue #35)."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Dict, List
+
+
+class DnsProviderError(Exception):
+    """A DNS provider failure with a SANITIZED, user-safe message.
+
+    The message must NEVER contain API tokens, request headers, or other secrets — it is
+    persisted to acme_order_events / order error_detail and shown in the UI. Raise this (not a
+    raw aiohttp/json error) so credentials can't leak into logs or the order timeline.
+    """
+
+
+class DnsProvider(ABC):
+    """Base class for a pluggable DNS provider.
+
+    RRset semantics are ADDITIVE: ``add_txt_record`` ensures a (name, value) TXT exists WITHOUT
+    removing other values at the same name, and ``remove_txt_record`` deletes ONLY the record
+    matching (name, value). This is required because a cert for ``example.com`` + ``*.example.com``
+    publishes two distinct values at the SAME name ``_acme-challenge.example.com``.
+    """
+
+    # Stable machine name (used in DB + API); human label; whether the provider automates publishing.
+    name: str = "base"
+    label: str = "Base"
+    automated: bool = True
+
+    # Declarative schema the UI renders to collect credentials. Each field:
+    #   {"key", "label", "type" ("text"|"password"), "required" (bool), "max_length" (int), "help" (str)}
+    credential_fields: List[Dict] = []
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        self.credentials = credentials or {}
+
+    @abstractmethod
+    async def verify_credentials(self) -> Dict:
+        """Validate the stored credentials against the provider. Returns
+        ``{"ok": bool, "detail": str}`` (detail is user-safe). Must not raise on auth failure —
+        return ``ok=False`` with a sanitized detail; may raise DnsProviderError on transport errors.
+        """
+
+    @abstractmethod
+    async def add_txt_record(self, name: str, value: str) -> None:
+        """Ensure a TXT record (name, value) exists. Idempotent; must not remove other values
+        at the same name. Raise DnsProviderError (sanitized) on failure."""
+
+    @abstractmethod
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        """Remove ONLY the TXT record matching (name, value). Tolerate 'already gone'.
+        Raise DnsProviderError (sanitized) on a real failure."""

backend/services/dns_providers/cloudflare.py

@@ -0,0 +1,179 @@
+"""Cloudflare DNS provider for ACME DNS-01 (Issue #35).
+
+Uses the Cloudflare API v4 over aiohttp (no new dependency). The base URL is a hardcoded
+constant and redirects are not followed (no user-controlled URL — only the already-validated
+domain name influences which zone is used). Errors are wrapped in DnsProviderError with a
+sanitized message so the API token never reaches logs / order events.
+
+Token scope required: Zone:DNS:Edit + Zone:Read.
+"""
+from __future__ import annotations
+
+import logging
+import re
+from typing import Dict, List, Optional, Tuple
+from urllib.parse import quote
+
+import aiohttp
+
+from .base import DnsProvider, DnsProviderError
+
+logger = logging.getLogger(__name__)
+
+CLOUDFLARE_API_BASE = "https://api.cloudflare.com/client/v4"
+_TIMEOUT = aiohttp.ClientTimeout(total=20)
+
+# Characters NOT valid in an HTTP bearer credential (RFC 6750 token68: A-Za-z0-9-._~+/=).
+# Cloudflare API tokens are a strict subset of this set, so removing anything outside it can
+# never corrupt a valid token, but it does strip the paste artifacts that make Cloudflare
+# reject the Authorization header with HTTP 400 "Invalid request headers" (CF code 6003):
+# surrounding/embedded quotes, interior spaces/tabs, zero-width/unicode chars, and CR/LF
+# (the latter would otherwise make aiohttp raise client-side before the request is even sent).
+_NON_TOKEN68 = re.compile(r"[^A-Za-z0-9._~+/=-]")
+
+
+def _strip_quotes(s: str) -> str:
+    s = (s or "").strip()
+    if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
+        return s[1:-1]
+    return s
+
+
+def _sanitize_token(s: str) -> str:
+    """Strip surrounding quotes/whitespace, then drop every character outside the token68 set."""
+    return _NON_TOKEN68.sub("", _strip_quotes(s))
+
+
+class CloudflareDNSProvider(DnsProvider):
+    name = "cloudflare"
+    label = "Cloudflare"
+    automated = True
+    credential_fields: List[Dict] = [
+        {
+            "key": "api_token",
+            "label": "API Token",
+            "type": "password",
+            "required": True,
+            "max_length": 200,
+            "help": "Scoped API token with Zone:DNS:Edit and Zone:Read permissions.",
+        }
+    ]
+
+    def __init__(self, credentials: Dict[str, str] | None = None):
+        super().__init__(credentials)
+        self._raw_token = (self.credentials.get("api_token") or "").strip()
+        # Sanitize to the token68 set so a pasted token with quotes/spaces/control/unicode chars
+        # cannot produce an invalid Authorization header (CF 6003 "Invalid request headers").
+        self._token = _sanitize_token(self._raw_token)
+
+    def _headers(self) -> Dict[str, str]:
+        return {"Authorization": f"Bearer {self._token}", "Content-Type": "application/json"}
+
+    async def _request(self, session: aiohttp.ClientSession, method: str, path: str, **kwargs) -> dict:
+        """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
+        DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
+        url = f"{CLOUDFLARE_API_BASE}{path}"
+        try:
+            async with session.request(
+                method, url, headers=self._headers(), allow_redirects=False, **kwargs
+            ) as resp:
+                try:
+                    body = await resp.json()
+                except Exception:  # noqa: BLE001
+                    body = {}
+                if resp.status in (401, 403):
+                    raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                if resp.status >= 400 or not body.get("success", False):
+                    # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                    # human message text, never the request (which carries the token header).
+                    msgs = "; ".join(
+                        str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                    )
+                    raise DnsProviderError(
+                        f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                    )
+                return body
+        except DnsProviderError:
+            raise
+        except aiohttp.ClientError as exc:
+            # Do NOT include exc verbatim everywhere; aiohttp client errors are URL/transport only
+            # (no token), but keep the message generic and stable.
+            raise DnsProviderError(f"Could not reach the Cloudflare API ({type(exc).__name__}).")
+        except Exception as exc:  # noqa: BLE001
+            raise DnsProviderError(f"Unexpected Cloudflare API failure ({type(exc).__name__}).")
+
+    async def verify_credentials(self) -> Dict:
+        if not self._token:
+            return {"ok": False, "detail": "No Cloudflare API token provided."}
+        try:
+            async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+                body = await self._request(session, "GET", "/zones?per_page=1")
+            total = ((body.get("result_info") or {}).get("total_count"))
+            detail = "Cloudflare token valid."
+            if isinstance(total, int):
+                detail = f"Cloudflare token valid; {total} zone(s) visible."
+            return {"ok": True, "detail": detail}
+        except DnsProviderError as exc:
+            # Always surface the real Cloudflare reason (e.g. token scope). If sanitizing also changed
+            # the token, append a hint that stray characters were stripped (never echo the token).
+            detail = str(exc)
+            if self._raw_token != self._token:
+                detail += (" Note: the token contained characters that were stripped; if it still "
+                           "fails, re-copy it from Cloudflare without quotes or spaces.")
+            return {"ok": False, "detail": detail}
+        except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
+            return {"ok": False, "detail": "Could not verify the Cloudflare token."}
+
+    async def _resolve_zone(self, session: aiohttp.ClientSession, record_name: str) -> Tuple[str, str]:
+        """Find the most-specific (longest-suffix) managed zone for a record name.
+        Returns (zone_id, zone_name). Raises DnsProviderError if no zone matches."""
+        labels = record_name.split(".")
+        # Walk suffixes from longest to shortest; a zone needs at least 2 labels.
+        for i in range(len(labels) - 1):
+            candidate = ".".join(labels[i:])
+            if candidate.count(".") < 1:
+                break
+            body = await self._request(
+                session, "GET", f"/zones?name={quote(candidate)}&status=active&per_page=50"
+            )
+            results = body.get("result") or []
+            if results:
+                return results[0]["id"], candidate
+        raise DnsProviderError(f"No managed Cloudflare zone found for {record_name}.")
+
+    async def _find_record_id(
+        self, session: aiohttp.ClientSession, zone_id: str, name: str, value: str
+    ) -> Optional[str]:
+        body = await self._request(
+            session, "GET", f"/zones/{zone_id}/dns_records?type=TXT&name={quote(name)}&per_page=100"
+        )
+        for rec in body.get("result") or []:
+            if _strip_quotes(rec.get("content", "")) == value:
+                return rec.get("id")
+        return None
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            zone_id, _zone_name = await self._resolve_zone(session, name)
+            # Idempotent: only create if (name, value) is not already present (preserves coexisting values).
+            existing = await self._find_record_id(session, zone_id, name, value)
+            if existing:
+                return
+            await self._request(
+                session,
+                "POST",
+                f"/zones/{zone_id}/dns_records",
+                json={"type": "TXT", "name": name, "content": value, "ttl": 120},
+            )
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        async with aiohttp.ClientSession(timeout=_TIMEOUT) as session:
+            try:
+                zone_id, _zone_name = await self._resolve_zone(session, name)
+            except DnsProviderError:
+                # Zone gone / not resolvable — nothing we can clean up.
+                return
+            record_id = await self._find_record_id(session, zone_id, name, value)
+            if not record_id:
+                return  # already gone — tolerate
+            await self._request(session, "DELETE", f"/zones/{zone_id}/dns_records/{record_id}")

backend/services/dns_providers/manual.py

@@ -0,0 +1,31 @@
+"""Manual DNS provider for ACME DNS-01 (Issue #35).
+
+The user publishes the `_acme-challenge` TXT record in their own DNS (any provider, including
+fully internal/isolated DNS that no API can reach) and then confirms via the UI. There is no API
+to call, so add/remove are no-ops and the orchestration waits for an explicit `dns-confirm`.
+CNAME delegation works implicitly here: the CA follows a CNAME, so a user who delegates
+`_acme-challenge` elsewhere just publishes the value there and confirms.
+"""
+from __future__ import annotations
+
+from typing import Dict, List
+
+from .base import DnsProvider
+
+
+class ManualDNSProvider(DnsProvider):
+    name = "manual"
+    label = "Manual (publish the TXT record yourself)"
+    automated = False
+    credential_fields: List[Dict] = []  # no credentials needed
+
+    async def verify_credentials(self) -> Dict:
+        return {"ok": True, "detail": "Manual mode needs no credentials. You will publish the TXT record yourself."}
+
+    async def add_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user publishes the record and confirms via the UI.
+        return None
+
+    async def remove_txt_record(self, name: str, value: str) -> None:
+        # No-op: the user may remove the record manually after issuance.
+        return None

backend/services/dns_providers/registry.py

@@ -0,0 +1,44 @@
+"""DNS provider registry for ACME DNS-01 (Issue #35).
+
+Single source of truth mapping a provider name -> class. The API serves the credential-field
+schema from here (so the UI has no hardcoded provider fields) and validates inbound provider
+names against this allow-list. Adding a provider = add it here; nothing else changes.
+"""
+from __future__ import annotations
+
+from typing import Dict, List, Type
+
+from .base import DnsProvider
+from .cloudflare import CloudflareDNSProvider
+from .manual import ManualDNSProvider
+
+_PROVIDERS: Dict[str, Type[DnsProvider]] = {
+    ManualDNSProvider.name: ManualDNSProvider,
+    CloudflareDNSProvider.name: CloudflareDNSProvider,
+}
+
+
+def is_supported(name: str) -> bool:
+    return name in _PROVIDERS
+
+
+def get_provider(name: str, credentials: Dict[str, str] | None = None) -> DnsProvider:
+    cls = _PROVIDERS.get(name)
+    if cls is None:
+        raise ValueError(f"Unsupported DNS provider: {name}")
+    return cls(credentials or {})
+
+
+def list_providers() -> List[Dict]:
+    """Return the UI-facing provider catalog: name, label, automated flag, and credential schema."""
+    out: List[Dict] = []
+    for name, cls in _PROVIDERS.items():
+        out.append(
+            {
+                "name": cls.name,
+                "label": cls.label,
+                "automated": cls.automated,
+                "credential_fields": cls.credential_fields,
+            }
+        )
+    return out

backend/services/haproxy_config.py

@@ -908,6 +908,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 "redirect": [], "use_be": [], "default_be": [],
             }
             _stick_table_emitted = False
+            # Whether this frontend emits ANY stick-counter usage (`track-sc<N>` or an
+            # `sc_*_rate(...)` fetch). If it does but no `stick-table` is declared, HAProxy
+            # fatally rejects the WHOLE cluster config with "table '<frontend>' used but not
+            # configured". This happens with rate-limit directives baked into a frontend's
+            # stored fields (request_headers/options) by an older version or a config import.
+            # We track it here and inject a default stick-table before flushing if needed.
+            _sc_counter_used = False
             # Phase K Phase D follow-up (Bulgu #13) — same dedup
             # contract for `http-request track-sc<N> <fetch>` lines.
             # HAProxy only NEEDS one tracking call per
@@ -931,9 +938,13 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 correct frontend-block bucket. Idempotent for stick-table
                 lines (R3.3 dedup) AND http-request track-sc<N> lines
                 (Bulgu #13 dedup)."""
-                nonlocal _stick_table_emitted
+                nonlocal _stick_table_emitted, _sc_counter_used
                 cat = _categorize_haproxy_directive(line)
                 stripped = line.strip()
+                # Any stick-counter usage (track-sc<N> write, or an sc_*_rate(...) fetch like
+                # sc_http_req_rate(0)) requires a stick-table in this frontend.
+                if "track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped):
+                    _sc_counter_used = True
                 if cat == "stick":
                     if _stick_table_emitted and stripped.startswith("stick-table"):
                         logger.debug(
@@ -1188,6 +1199,22 @@ async def generate_haproxy_config_for_cluster(cluster_id: int, conn: Optional[An
                 else:
                     logger.warning(f"Config Generation: No config lines generated for WAF rule '{waf_rule['name']}' (ID: {waf_rule['id']}, Type: {waf_rule['rule_type']})")
 
+            # Robustness fix: if this frontend uses a stick counter (track-sc<N> or an
+            # sc_*_rate(...) fetch) but declared NO stick-table, inject a default one so HAProxy
+            # doesn't fatally reject the whole cluster config with "table '<frontend>' used but
+            # not configured". This rescues rate-limit directives baked into a frontend's stored
+            # request_headers/options by an older version or import. Purely additive — it only
+            # fires when a counter is used AND no table exists (a config that is invalid today),
+            # so it never changes a frontend that already has a stick-table or doesn't rate-limit.
+            if _sc_counter_used and not _stick_table_emitted:
+                _fe_buckets["stick"].insert(
+                    0, "    stick-table type ip size 100k expire 30s store http_req_rate(10s)")
+                _stick_table_emitted = True
+                logger.info(
+                    f"STICK-TABLE AUTO-INJECT: frontend '{frontend['name']}' uses a stick "
+                    f"counter (track-sc/sc_*_rate) but declared no stick-table; injected a "
+                    f"default so the config stays valid.")
+
             # ─────────────────────────────────────────────────────────────
             # Flush the per-frontend buckets in canonical HAProxy order.
             # The order below is the single source of truth for emit

backend/services/keepalived_config.py

@@ -0,0 +1,192 @@
+"""Issue #27 — HA/VIP (Keepalived) management (v1.7.0).
+
+Standalone, DB-free renderer for a node's /etc/keepalived/keepalived.conf and the
+HAProxy health-check script, plus Fernet at-rest encryption for the VRRP secret.
+
+The router fetches DB rows and calls these pure functions; nothing here touches the
+database or logs secrets. Trivially unit-testable (see tests/test_keepalived_config.py).
+"""
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import re
+from typing import List, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# Written into every file we manage so the agent can tell "ours" from a
+# hand-maintained keepalived setup (ownership guard, B-3/T-2). Must match the
+# string the agent greps for in linux_install.sh.
+OWNERSHIP_MARKER = "# Managed by HAProxy OpenManager"
+CHECK_SCRIPT_PATH = "/etc/keepalived/check_haproxy.sh"
+
+
+# ---------------------------------------------------------------------------
+# VRRP secret at rest (mirrors backend/services/mfa_service.py)
+# ---------------------------------------------------------------------------
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit VIP_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so the secret survives restarts, like MFA)."""
+    explicit = os.getenv("VIP_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("VIP_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"vip-vrrp-secret-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_vrrp_secret(secret_plain: str) -> str:
+    return _get_fernet().encrypt(secret_plain.encode("utf-8")).decode("utf-8")
+
+
+def decrypt_vrrp_secret(secret_encrypted: str) -> Optional[str]:
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt VRRP secret (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting VRRP secret: %s", exc)
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Renderers
+# ---------------------------------------------------------------------------
+def build_haproxy_check_script(*, bin_path: Optional[str] = None,
+                               config_path: Optional[str] = None) -> str:
+    """Render the health-check the agent writes to CHECK_SCRIPT_PATH.
+
+    Derives the process name from the HAProxy binary basename (B-4) rather than a
+    blind hardcoded 'haproxy'. Returns non-zero when HAProxy isn't running so the
+    VRRP track_script lowers this node's priority and the VIP fails over.
+    """
+    proc = "haproxy"
+    if bin_path:
+        base = os.path.basename(bin_path.strip())
+        if re.match(r'^[A-Za-z0-9._-]{1,64}$', base):
+            proc = base
+    return (
+        "#!/bin/sh\n"
+        f"{OWNERSHIP_MARKER} — DO NOT EDIT\n"
+        "# Exits 0 while HAProxy is up; non-zero triggers VRRP failover.\n"
+        f"pidof {proc} >/dev/null 2>&1 || exit 1\n"
+        "exit 0\n"
+    )
+
+
+def _vrrp_instance_name(vip_id: int) -> str:
+    return f"VI_{int(vip_id)}"
+
+
+def _failover_weight(members: List[dict]) -> int:
+    """Negative weight so a failed MASTER drops strictly below every healthy BACKUP
+    (B-6). master_priority + weight < min(backup_priority)."""
+    master = next((m for m in members if str(m.get("role", "")).upper() == "MASTER"), None)
+    backups = [int(m["priority"]) for m in members if str(m.get("role", "")).upper() != "MASTER"]
+    if not master or not backups:
+        return -20
+    return -((int(master["priority"]) - min(backups)) + 1)
+
+
+def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
+                           peer_ips: List[str], auth_pass_plain: Optional[str]) -> str:
+    """Render one node's keepalived.conf from the VIP + member rows.
+
+    `vip` keys: id, name, virtual_ip, prefix_length, virtual_router_id, advert_int,
+                use_unicast, track_haproxy.
+    `this_agent` keys: role, priority, network_interface, ip_address (str).
+    `peer_ips`: the OTHER members' ip_address strings (already str()'d by the caller).
+    Caller must never log the returned string (it may contain auth_pass).
+    """
+    role = str(this_agent["role"]).upper()
+    iface = this_agent["network_interface"]
+    prio = int(this_agent["priority"])
+    track = bool(vip.get("track_haproxy", True))
+    use_unicast = bool(vip.get("use_unicast", True))
+    vrid = int(vip["virtual_router_id"])
+    advert = int(vip.get("advert_int", 1))
+    name = str(vip.get("name", ""))
+    inst = _vrrp_instance_name(vip["id"])
+
+    lines: List[str] = []
+    lines.append(f"{OWNERSHIP_MARKER} — DO NOT EDIT")
+    lines.append(f'# VIP "{name}" (id={vip["id"]}) — role {role}')
+    lines.append("global_defs {")
+    lines.append("    enable_script_security")
+    lines.append("    script_user root")
+    lines.append("}")
+    lines.append("")
+
+    if track:
+        weight = _failover_weight(members)
+        lines.append("vrrp_script chk_haproxy {")
+        lines.append(f'    script "{CHECK_SCRIPT_PATH}"')
+        lines.append("    interval 2")
+        lines.append("    fall 2")
+        lines.append("    rise 2")
+        lines.append(f"    weight {weight}")
+        lines.append("}")
+        lines.append("")
+
+    lines.append(f"vrrp_instance {inst} {{")
+    lines.append(f"    state {role}")
+    lines.append(f"    interface {iface}")
+    lines.append(f"    virtual_router_id {vrid}")
+    lines.append(f"    priority {prio}")
+    lines.append(f"    advert_int {advert}")
+    if auth_pass_plain:
+        lines.append("    authentication {")
+        lines.append("        auth_type PASS")
+        lines.append(f"        auth_pass {auth_pass_plain}")
+        lines.append("    }")
+    # Unicast only makes sense with at least one peer. For a single-node VIP (no peers)
+    # we deliberately omit the unicast block: keepalived treats a bare `unicast_src_ip`
+    # with no `unicast_peer` as deprecated, warns, and silently falls back to multicast —
+    # and `keepalived -t` flags it. Omitting it yields a clean multicast config that holds
+    # the VIP with no peer to talk to. Multi-node behaviour (peers present) is unchanged.
+    if use_unicast and peer_ips:
+        src = this_agent.get("ip_address")
+        if src:
+            lines.append(f"    unicast_src_ip {src}")
+        lines.append("    unicast_peer {")
+        for p in peer_ips:
+            lines.append(f"        {p}")
+        lines.append("    }")
+    lines.append("    virtual_ipaddress {")
+    lines.append(f'        {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
+    lines.append("    }")
+    if track:
+        lines.append("    track_script {")
+        lines.append("        chk_haproxy")
+        lines.append("    }")
+    lines.append("}")
+    return "\n".join(lines) + "\n"

backend/services/mfa_service.py

@@ -0,0 +1,238 @@
+"""MFA (TOTP + backup codes) service layer — Issue #18, v1.6.0.
+
+Owns the cryptographic and persistence-shape concerns of multi-factor auth:
+  - TOTP secret generation / verification with replay protection (RFC 6238)
+  - Backup code generation, hashing (bcrypt) and atomic single-use consumption
+  - Fernet-based encryption of TOTP secrets at rest
+
+Strictly no logging of secrets — only metadata (lengths, counts) is logged.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import logging
+import os
+import secrets as _secrets
+import time
+from typing import List, Optional, Tuple
+from urllib.parse import quote
+
+import bcrypt
+import pyotp
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+# RFC 6238 parameters — kept conservative for widest authenticator app compatibility.
+TOTP_DIGITS = 6
+TOTP_PERIOD = 30
+TOTP_DIGEST = "sha1"
+TOTP_VALID_WINDOW_STEPS = 1  # ±1 step (±30s) tolerance
+
+# Backup code spec (Plan section 5).
+# Alphabet drops the confusing pairs: 0/O, 1/I, L. Resulting size is 31, which
+# still yields 31**8 ≈ 8.5×10^11 combinations per half — far beyond brute-force.
+BACKUP_CODE_COUNT = 10
+BACKUP_CODE_ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"
+BACKUP_CODE_HALF_LEN = 4  # XXXX-YYYY
+
+# OTP URI defaults.
+DEFAULT_ISSUER = "HAProxy OpenManager"
+ACCOUNT_LABEL_DOMAIN_FALLBACK = "haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Fernet key resolution
+# ---------------------------------------------------------------------------
+
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Resolve the Fernet key, preferring the explicit env var.
+
+    Falls back to HKDF over SECRET_KEY with a versioned info string so a future
+    rotation can be expressed by bumping the version suffix.
+    """
+    explicit = os.getenv("MFA_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:
+            logger.error("MFA_ENCRYPTION_KEY env var present but invalid: %s", exc)
+            # fall through to HKDF derivation rather than crashing the app
+
+    logger.warning(
+        "MFA_ENCRYPTION_KEY env var not set or invalid; deriving from SECRET_KEY (v1). "
+        "Set an explicit MFA_ENCRYPTION_KEY in production to enable key rotation."
+    )
+    hkdf = HKDF(
+        algorithm=hashes.SHA256(),
+        length=32,
+        salt=None,
+        info=b"mfa-totp-secret-v1",
+    )
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution of the Fernet key after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+# ---------------------------------------------------------------------------
+# TOTP secrets
+# ---------------------------------------------------------------------------
+
+
+def generate_totp_secret() -> str:
+    """Return a fresh base32 TOTP secret (32 chars)."""
+    return pyotp.random_base32()
+
+
+def encrypt_secret(secret_plain: str) -> str:
+    """Fernet-encrypt the base32 secret. Returns str for direct DB storage."""
+    token = _get_fernet().encrypt(secret_plain.encode("utf-8"))
+    return token.decode("utf-8")
+
+
+def decrypt_secret(secret_encrypted: str) -> Optional[str]:
+    """Decrypt a previously stored secret. Returns None when the token can't be
+    decrypted (e.g. key rotated without re-enroll). Never raises to the caller.
+    """
+    try:
+        return _get_fernet().decrypt(secret_encrypted.encode("utf-8")).decode("utf-8")
+    except InvalidToken:
+        logger.warning("Failed to decrypt MFA secret (invalid Fernet token)")
+        return None
+    except Exception as exc:
+        logger.error("Unexpected error decrypting MFA secret: %s", exc)
+        return None
+
+
+def build_otpauth_uri(account_label: str, secret_plain: str, issuer: str = DEFAULT_ISSUER) -> str:
+    """Build an otpauth:// URI that all major authenticator apps accept.
+
+    Format: otpauth://totp/<issuer>:<account>?secret=<b32>&issuer=<issuer>&algorithm=SHA1&digits=6&period=30
+    """
+    issuer_q = quote(issuer, safe="")
+    label = f"{issuer}:{account_label}"
+    label_q = quote(label, safe=":@")
+    return (
+        f"otpauth://totp/{label_q}?secret={secret_plain}"
+        f"&issuer={issuer_q}&algorithm=SHA1&digits={TOTP_DIGITS}&period={TOTP_PERIOD}"
+    )
+
+
+def build_account_label(username: str, hostname_hint: Optional[str] = None) -> str:
+    """Compose the per-user otpauth label, respecting env > hostname > fallback."""
+    domain = (
+        os.getenv("MFA_ACCOUNT_LABEL_DOMAIN", "").strip()
+        or (hostname_hint or "").strip()
+        or ACCOUNT_LABEL_DOMAIN_FALLBACK
+    )
+    return f"{username}@{domain}"
+
+
+def verify_totp_with_replay_guard(
+    secret_plain: str,
+    code: str,
+    last_used_step: Optional[int],
+) -> Tuple[bool, Optional[int]]:
+    """Verify a 6-digit TOTP code with explicit per-step replay protection.
+
+    Returns (success, step_consumed). Caller persists the consumed step on success.
+
+    Implementation notes:
+      - pyotp.TOTP.at(seconds_since_epoch) — to target step N we pass step*PERIOD.
+      - secrets.compare_digest is used for constant-time comparison.
+      - Replay guard rejects codes whose step is <= the previously consumed step.
+    """
+    if not secret_plain or not code:
+        return (False, None)
+    code = code.strip()
+    if len(code) != TOTP_DIGITS or not code.isdigit():
+        return (False, None)
+
+    totp = pyotp.TOTP(secret_plain, digits=TOTP_DIGITS, interval=TOTP_PERIOD, digest=TOTP_DIGEST)
+    now = int(time.time())
+    current_step = now // TOTP_PERIOD
+
+    for offset in (0, -1, 1):
+        step = current_step + offset
+        expected = totp.at(step * TOTP_PERIOD)
+        if len(expected) == len(code) and _secrets.compare_digest(expected, code):
+            if last_used_step is not None and step <= last_used_step:
+                return (False, None)
+            return (True, step)
+    return (False, None)
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+def generate_backup_codes(count: int = BACKUP_CODE_COUNT) -> List[str]:
+    """Return ``count`` plain-text backup codes formatted as ``XXXX-YYYY``."""
+    codes: List[str] = []
+    for _ in range(count):
+        left = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        right = "".join(_secrets.choice(BACKUP_CODE_ALPHABET) for _ in range(BACKUP_CODE_HALF_LEN))
+        codes.append(f"{left}-{right}")
+    return codes
+
+
+def normalize_backup_code(user_input: str) -> str:
+    """Canonical form for comparison: uppercase, strip dashes/spaces."""
+    if not user_input:
+        return ""
+    return user_input.strip().upper().replace("-", "").replace(" ", "")
+
+
+async def _hash_one_backup_code(code_plain: str) -> str:
+    """Bcrypt-hash a single backup code on a worker thread."""
+    normalized = normalize_backup_code(code_plain)
+    hashed = await asyncio.to_thread(bcrypt.hashpw, normalized.encode("utf-8"), bcrypt.gensalt())
+    return hashed.decode("utf-8")
+
+
+async def hash_backup_codes(codes_plain: List[str]) -> List[str]:
+    """Hash backup codes in parallel (each bcrypt op runs in its own thread)."""
+    return await asyncio.gather(*(_hash_one_backup_code(c) for c in codes_plain))
+
+
+async def check_backup_code(user_input: str, code_hash: str) -> bool:
+    """Run a single bcrypt verify on the worker pool."""
+    normalized = normalize_backup_code(user_input)
+    if not normalized:
+        return False
+    return await asyncio.to_thread(
+        bcrypt.checkpw, normalized.encode("utf-8"), code_hash.encode("utf-8")
+    )
+
+
+# ---------------------------------------------------------------------------
+# Misc helpers
+# ---------------------------------------------------------------------------
+
+
+def generate_challenge_token() -> str:
+    """64-char hex challenge token for /api/auth/login → /mfa-verify hand-off."""
+    return _secrets.token_hex(32)

backend/tests/test_acme_humanizer.py

@@ -81,6 +81,7 @@ def test_legacy_plain_string_with_brace_but_invalid_json_falls_back():
     ("urn:ietf:params:acme:error:rejectedIdentifier", "blacklisted",    "rejected"),
     ("urn:ietf:params:acme:error:serverInternal", "internal err",       "ACME server"),
     ("urn:ietf:params:acme:error:userActionRequired", "agree to ToS",   "User action"),
+    ("urn:ietf:params:acme:error:externalAccountRequired", "EAB required", "External Account Binding"),
 ])
 def test_known_problem_types_are_humanized(problem_type, detail_text, expected_title_contains):
     payload = json.dumps({"type": problem_type, "detail": detail_text, "status": 400})

backend/tests/test_acme_pydantic_validation.py

@@ -8,7 +8,31 @@ from pydantic import ValidationError
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from routers.letsencrypt import CertificateRequest
+from routers.letsencrypt import CertificateRequest, AccountCreate
+
+
+class TestAccountCreateEAB:
+    """Issue #35 follow-up: EAB HMAC key must be valid base64url; empty/None passes through
+    (falls back to global Settings) so non-EAB accounts (HTTP-01 / LE / Cloudflare) are unaffected."""
+
+    def test_no_eab_is_allowed(self):
+        acc = AccountCreate(email="a@b.com")
+        assert acc.eab_hmac_key is None and acc.eab_kid is None
+
+    def test_valid_base64url_hmac_accepted(self):
+        # urlsafe base64, unpadded and padded — both accepted.
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZGVmZ2g")
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZA==")
+
+    def test_invalid_base64_hmac_rejected(self):
+        # 5 base64 chars (count ≡ 1 mod 4) is undecodable — the exact shape that would otherwise
+        # make register_account's _b64url_decode raise a cryptic binascii error.
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="AAAAA")
+
+    def test_oversized_hmac_rejected(self):
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="A" * 600)
 
 
 class TestCertificateRequestDomains:

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_dns01.py

@@ -0,0 +1,220 @@
+"""Issue #35 — ACME DNS-01: focused unit tests for the pure logic (no DB/network).
+
+Covers the TXT-value math (RFC 8555 §8.4 — raw SHA-256 digest, base64url, NOT hex),
+the _acme-challenge record-name derivation (wildcard stripping), credential encryption
+round-trip + tamper handling, and the DNS provider registry/allow-list.
+"""
+import base64
+import hashlib
+import os
+
+os.environ.setdefault("SECRET_KEY", "test-secret-key-for-dns01-unit-tests")
+
+from services.acme_service import ACMEService
+from services.dns_providers import list_providers, is_supported, get_provider, DnsProviderError
+from utils.dns_credentials import (
+    encrypt_dns_credentials, decrypt_dns_credentials, reset_fernet_for_tests,
+)
+
+
+def _b64url(b: bytes) -> str:
+    return base64.urlsafe_b64encode(b).rstrip(b"=").decode("ascii")
+
+
+def test_dns_txt_value_is_raw_sha256_base64url():
+    key_auth = "token123.thumbprintABC"
+    expected = _b64url(hashlib.sha256(key_auth.encode("utf-8")).digest())
+    assert ACMEService._dns_txt_value(key_auth) == expected
+    # Must NOT be the (classic-mistake) base64url of the HEX digest.
+    hex_based = _b64url(hashlib.sha256(key_auth.encode("utf-8")).hexdigest().encode("utf-8"))
+    assert ACMEService._dns_txt_value(key_auth) != hex_based
+
+
+def test_challenge_dns_name_derivation():
+    assert ACMEService._challenge_dns_name("example.com") == "_acme-challenge.example.com"
+    # Wildcard: the '*.' is stripped, so apex + wildcard share the SAME record name.
+    assert ACMEService._challenge_dns_name("*.example.com") == "_acme-challenge.example.com"
+    assert ACMEService._challenge_dns_name("foo.bar.example.com") == "_acme-challenge.foo.bar.example.com"
+
+
+def test_credential_encryption_roundtrip():
+    reset_fernet_for_tests()
+    creds = {"api_token": "super-secret-token-value"}
+    token = encrypt_dns_credentials(creds)
+    assert token != "super-secret-token-value"
+    assert "super-secret-token-value" not in token  # ciphertext, not plaintext
+    assert decrypt_dns_credentials(token) == creds
+
+
+def test_decrypt_invalid_token_returns_none():
+    reset_fernet_for_tests()
+    assert decrypt_dns_credentials("not-a-valid-fernet-token") is None
+
+
+def test_provider_registry_and_allow_list():
+    names = {p["name"] for p in list_providers()}
+    assert {"manual", "cloudflare"} <= names
+    assert is_supported("manual") and is_supported("cloudflare")
+    assert not is_supported("route53")  # not in MVP allow-list
+
+    assert get_provider("manual").automated is False
+    cf = get_provider("cloudflare", {"api_token": "x"})
+    assert cf.automated is True
+    assert any(f["key"] == "api_token" for f in cf.credential_fields)
+
+    raised = False
+    try:
+        get_provider("definitely-not-a-provider")
+    except ValueError:
+        raised = True
+    assert raised
+
+
+def test_cloudflare_token_sanitize():
+    # Issue #35 follow-up: a pasted token with quotes/spaces/control/unicode chars produced an
+    # invalid Authorization header (CF 6003 "Invalid request headers"). The sanitizer strips them.
+    from services.dns_providers.cloudflare import _sanitize_token, CloudflareDNSProvider
+
+    # Surrounding double quotes stripped.
+    assert _sanitize_token('"abc123-_def"') == 'abc123-_def'
+    # Interior spaces / tabs / newlines removed.
+    assert _sanitize_token('abc 123\tdef\n') == 'abc123def'
+    # A clean token68 string is unchanged (cannot corrupt a valid Cloudflare token).
+    clean = 'A1b2-_C3.d4~e5+f6/g7=='
+    assert _sanitize_token(clean) == clean
+    # Single quotes and a zero-width char removed.
+    assert _sanitize_token("'tok" + chr(0x200b) + "en'") == 'token'
+
+    # The provider constructor sanitizes into _token and keeps the raw input for diagnostics.
+    p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
+    assert p._token == 'my-token_123'
+    assert p._raw_token == '"my-token_123"'
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/tests/test_keepalive_fault_state.py

@@ -0,0 +1,21 @@
+"""v1.7.6 — guard: the agent must surface keepalived FAULT state.
+
+A VIP whose interface has no usable IPv4 (or whose track-script fails) puts keepalived into
+FAULT — the virtual IP is NOT held. Previously get_keepalive_state only grepped (MASTER|BACKUP),
+so a FAULT'd VIP reported as BACKUP — misleading (looks healthy-ish). It must report FAULT so the
+UI shows it red. Both get_keepalive_state copies (installer + SKIP_TO_DAEMON daemon) must include
+FAULT. Pure source guard."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def test_keepalive_state_detects_fault_in_both_copies():
+    with open(os.path.join(ROOT, "utils", "agent_scripts", "linux_install.sh"), encoding="utf-8") as f:
+        s = f.read()
+    # FAULT added to the state grep in both copies (installer + daemon), both detection methods.
+    assert s.count("MASTER|BACKUP|FAULT") >= 2
+    # and the misleading MASTER|BACKUP-only grep is gone.
+    assert 'grep -oE "(MASTER|BACKUP)"' not in s

backend/tests/test_keepalived_config.py

@@ -0,0 +1,129 @@
+"""Issue #27 (v1.7.0) — unit tests for the keepalived config generator + secret crypto.
+
+Pure-function tests; no DB. Validates the rendered keepalived.conf for MASTER/BACKUP,
+unicast peers, the failover weight arithmetic, the script-security requirements, the
+ownership marker, and Fernet round-trip.
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from services import keepalived_config as kc  # noqa: E402
+
+
+VIP = {
+    "id": 3, "name": "web-vip", "virtual_ip": "10.0.0.100", "prefix_length": 24,
+    "virtual_router_id": 51, "advert_int": 1, "use_unicast": True, "track_haproxy": True,
+}
+MEMBERS = [
+    {"role": "MASTER", "priority": 150, "network_interface": "eth0", "agent_id": 1, "ip_address": "10.0.0.11"},
+    {"role": "BACKUP", "priority": 100, "network_interface": "eth0", "agent_id": 2, "ip_address": "10.0.0.12"},
+]
+
+
+def _render(this_idx, auth="s3cr3t"):
+    this_agent = MEMBERS[this_idx]
+    peers = [m["ip_address"] for m in MEMBERS if m["agent_id"] != this_agent["agent_id"]]
+    return kc.render_keepalived_conf(vip=VIP, members=MEMBERS, this_agent=this_agent,
+                                     peer_ips=peers, auth_pass_plain=auth)
+
+
+class TestRender:
+    def test_master_state_priority_iface_vrid(self):
+        conf = _render(0)
+        assert "state MASTER" in conf
+        assert "priority 150" in conf
+        assert "interface eth0" in conf
+        assert "virtual_router_id 51" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+    def test_backup_state(self):
+        conf = _render(1)
+        assert "state BACKUP" in conf
+        assert "priority 100" in conf
+
+    def test_unicast_peers(self):
+        # MASTER's config lists the BACKUP as its unicast peer (and its own src ip).
+        conf = _render(0)
+        assert "unicast_src_ip 10.0.0.11" in conf
+        assert "unicast_peer" in conf
+        assert "10.0.0.12" in conf
+
+    def test_script_security_block(self):
+        conf = _render(0)
+        assert "enable_script_security" in conf
+        assert "script_user root" in conf
+
+    def test_ownership_marker(self):
+        assert kc.OWNERSHIP_MARKER in _render(0)
+
+    def test_weight_makes_failed_master_lose(self):
+        # On HAProxy failure the master's effective priority must drop below the backup.
+        conf = _render(0)
+        weight_line = [l for l in conf.splitlines() if l.strip().startswith("weight ")][0]
+        weight = int(weight_line.strip().split()[1])
+        assert 150 + weight < 100, "failed master must fall below every backup"
+        assert "track_script" in conf and "chk_haproxy" in conf
+
+    def test_track_disabled_omits_script(self):
+        vip = {**VIP, "track_haproxy": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain=None)
+        assert "vrrp_script" not in conf
+        assert "track_script" not in conf
+
+    def test_no_auth_when_secret_absent(self):
+        conf = _render(0, auth=None)
+        assert "auth_pass" not in conf
+
+    def test_multicast_omits_unicast(self):
+        vip = {**VIP, "use_unicast": False}
+        conf = kc.render_keepalived_conf(vip=vip, members=MEMBERS, this_agent=MEMBERS[0],
+                                         peer_ips=["10.0.0.12"], auth_pass_plain="x")
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+
+    def test_single_node_omits_unicast_block(self):
+        # Single-node VIP (no peers): even with use_unicast=True we must NOT emit a bare
+        # `unicast_src_ip`/`unicast_peer` — keepalived treats a unicast keyword with no peers
+        # as deprecated, warns, and falls back to multicast (and `keepalived -t` flags it).
+        # Omitting the block yields a clean multicast config that holds the VIP solo.
+        only = [{"role": "MASTER", "priority": 150, "network_interface": "eth0",
+                 "agent_id": 1, "ip_address": "10.0.0.11"}]
+        conf = kc.render_keepalived_conf(vip=VIP, members=only, this_agent=only[0],
+                                         peer_ips=[], auth_pass_plain=None)
+        assert "unicast_src_ip" not in conf
+        assert "unicast_peer" not in conf
+        assert "state MASTER" in conf
+        assert "10.0.0.100/24 dev eth0" in conf
+
+
+class TestCheckScript:
+    def test_default_process_name(self):
+        s = kc.build_haproxy_check_script()
+        assert "pidof haproxy" in s
+        assert kc.OWNERSHIP_MARKER in s
+
+    def test_process_name_from_bin_path(self):
+        s = kc.build_haproxy_check_script(bin_path="/opt/hap/sbin/haproxy-ent")
+        assert "pidof haproxy-ent" in s
+
+    def test_malicious_bin_path_falls_back(self):
+        s = kc.build_haproxy_check_script(bin_path="/x/haproxy; rm -rf /")
+        assert "rm -rf" not in s
+        assert "pidof haproxy" in s
+
+
+class TestSecretCrypto:
+    def test_roundtrip(self):
+        kc.reset_fernet_for_tests()
+        token = kc.encrypt_vrrp_secret("s3cr3t")
+        assert token != "s3cr3t"
+        assert kc.decrypt_vrrp_secret(token) == "s3cr3t"
+
+    def test_decrypt_garbage_returns_none(self):
+        kc.reset_fernet_for_tests()
+        assert kc.decrypt_vrrp_secret("not-a-fernet-token") is None

backend/tests/test_mfa_backwards_compat.py

@@ -0,0 +1,150 @@
+"""Backwards-compatibility regression tests for the MFA rollout (Issue #18).
+
+These tests don't hit a real database — they exercise the authoritative
+contract surfaces (login response shape, auth_middleware behaviour) using
+mocks where needed so the suite stays fast and deterministic.
+"""
+from __future__ import annotations
+
+import os
+import sys
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+class TestAuthMiddlewareUnchanged:
+    """auth_middleware MUST NOT look for MFA claims — Madde 2 of the plan."""
+
+    def test_decoder_imports_without_mfa_dependencies(self):
+        import auth_middleware
+        # The middleware's verification function exists and is callable.
+        assert callable(getattr(auth_middleware, "get_current_user_from_token", None))
+
+    def test_middleware_source_has_no_mfa_claim_check(self):
+        """The middleware source must not reference ``mfa`` claims directly."""
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "auth_middleware.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        # Allow incidental occurrences (e.g. comments); but never a claim lookup.
+        assert "payload.get('mfa'" not in source
+        assert 'payload.get("mfa"' not in source
+        assert "claims['mfa'" not in source
+        assert 'claims["mfa"' not in source
+
+
+class TestMfaModelsCoexistWithUserModels:
+    def test_models_user_module_unchanged_pydantic_shape(self):
+        from models import user as user_mod
+        # Ensure that User / UserUpdate / LoginRequest still load and still
+        # don't expose mfa-related fields (kept in models.mfa).
+        for cls in (user_mod.User, user_mod.UserUpdate, user_mod.LoginRequest):
+            fields = set(cls.model_fields.keys())
+            assert not {"mfa_enabled", "mfa_required", "mfa_token"} & fields, (
+                f"{cls.__name__} unexpectedly exposes MFA field; should stay byte-identical."
+            )
+
+    def test_models_mfa_module_exposes_expected_models(self):
+        from models import mfa as mfa_mod
+        for name in (
+            "MfaVerifyRequest",
+            "MfaEnrollStartResponse",
+            "MfaEnrollConfirmRequest",
+            "MfaEnrollConfirmResponse",
+            "MfaDisableRequest",
+            "MfaRegenerateBackupRequest",
+            "MfaRegenerateBackupResponse",
+            "MfaAdminResetRequest",
+            "MfaAdminResetAllRequest",
+            "MfaStatusResponse",
+        ):
+            assert hasattr(mfa_mod, name), f"Missing model: {name}"
+
+
+class TestRouterIncluded:
+    def test_main_includes_mfa_router(self):
+        with open(
+            os.path.join(
+                os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                "main.py",
+            ),
+            "r",
+            encoding="utf-8",
+        ) as fh:
+            source = fh.read()
+        assert "from routers.mfa import router as mfa_router" in source
+        assert "app.include_router(mfa_router)" in source
+
+
+class TestLoginResponseShapeForNonMfaUser:
+    """When MFA columns are missing or mfa_enabled=FALSE, /login returns the
+    pre-MFA response shape — no ``mfa_required`` / ``mfa_token`` keys leak through.
+    """
+
+    def test_login_without_mfa_returns_legacy_shape(self):
+        from fastapi.testclient import TestClient
+        from main import app
+
+        client = TestClient(app)
+
+        async def _fetch_mfa_state_none(conn, user_id):
+            return None
+
+        async def _no_log(*args, **kwargs):
+            return None
+
+        fake_user = {
+            "id": 1,
+            "username": "admin",
+            "email": "admin@example.com",
+            "password_hash": "$2b$12$placeholder",
+            "is_active": True,
+            "role": "admin",
+            "created_at": None,
+            "updated_at": None,
+            "last_login_at": None,
+        }
+
+        mock_conn = MagicMock()
+        mock_conn.fetchrow = AsyncMock(return_value=fake_user)
+        mock_conn.fetch = AsyncMock(return_value=[])
+        mock_conn.execute = AsyncMock(return_value=None)
+
+        async def _get_conn():
+            return mock_conn
+
+        async def _close(conn):
+            return None
+
+        with patch(
+            "routers.auth.get_database_connection", _get_conn
+        ), patch("routers.auth.close_database_connection", _close), patch(
+            "routers.auth._fetch_mfa_state", _fetch_mfa_state_none
+        ), patch("routers.auth.log_user_activity", _no_log), patch(
+            "bcrypt.checkpw", return_value=True
+        ):
+            resp = client.post(
+                "/api/auth/login",
+                json={"username": "admin", "password": "anything"},
+            )
+
+        assert resp.status_code == 200, resp.text
+        body = resp.json()
+        assert "access_token" in body
+        assert "token_type" in body
+        assert "expires_in" in body
+        assert "user" in body
+        assert "roles" in body
+        assert "permissions" in body
+        # CRITICAL — pre-MFA contract must not be polluted with MFA fields.
+        assert "mfa_required" not in body
+        assert "mfa_token" not in body
+        assert "methods" not in body

backend/tests/test_mfa_rate_limit_key.py

@@ -0,0 +1,200 @@
+"""Tests for middleware.mfa_rate_limit_key — user-aware + ingress-aware key."""
+import importlib
+from datetime import datetime, timedelta
+from typing import Dict, Optional
+
+import pytest
+from fastapi import Request
+from jose import jwt
+
+
+def _make_request(
+    headers: Optional[Dict[str, str]] = None,
+    peer: str = "127.0.0.1",
+) -> Request:
+    """Tiny ASGI scope shim — enough for the key_func surface."""
+    raw_headers = []
+    if headers:
+        raw_headers = [
+            (k.encode("latin-1"), v.encode("latin-1")) for k, v in headers.items()
+        ]
+    scope = {
+        "type": "http",
+        "headers": raw_headers,
+        "client": (peer, 12345),
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    return Request(scope)
+
+
+@pytest.fixture
+def reload_key(monkeypatch):
+    """Reload the key module so MFA_TRUSTED_PROXY_CIDRS is re-parsed."""
+
+    def _reload(**env):
+        for var in ("MFA_TRUSTED_PROXY_CIDRS",):
+            monkeypatch.delenv(var, raising=False)
+        for k, v in env.items():
+            monkeypatch.setenv(k, v)
+        from middleware import mfa_rate_limit_key as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+# ---------------------------------------------------------------------------
+# User-aware key extraction
+# ---------------------------------------------------------------------------
+
+
+def _mint_jwt(user_id, claim: str = "user_id") -> str:
+    """Mint a test JWT. Note: RFC 7519 says ``sub`` is a StringOrURI,
+    and python-jose validates that type when decoding, so callers that use
+    ``claim='sub'`` must pass a string user_id (matches production behavior
+    where auth_middleware also accepts string ``sub``)."""
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        claim: user_id,
+        "exp": datetime.utcnow() + timedelta(minutes=10),
+    }
+    return jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+
+
+def test_user_aware_via_user_id_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt(42, claim="user_id")
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:42"
+
+
+def test_user_aware_via_sub_claim(reload_key):
+    m = reload_key()
+    token = _mint_jwt("7", claim="sub")  # JWT spec: sub is a string
+    req = _make_request(headers={"authorization": f"Bearer {token}"})
+    assert m.mfa_rate_limit_key(req) == "user:7"
+
+
+def test_no_auth_header_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_missing_bearer_prefix_falls_back_to_ip(reload_key):
+    m = reload_key()
+    req = _make_request(headers={"authorization": "abc.def.ghi"}, peer="203.0.113.5")
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_bearer_null_or_undefined_falls_back_to_ip(reload_key):
+    m = reload_key()
+    for bogus in ("null", "undefined", "", "   "):
+        req = _make_request(
+            headers={"authorization": f"Bearer {bogus}"}, peer="198.51.100.9"
+        )
+        assert m.mfa_rate_limit_key(req) == "ip:198.51.100.9"
+
+
+def test_tampered_jwt_falls_back_to_ip(reload_key):
+    """A token with a forged signature must NOT be honored — fallback to IP."""
+    m = reload_key()
+    bad = "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjogMTIzfQ.NOT_A_VALID_SIGNATURE"
+    req = _make_request(headers={"authorization": f"Bearer {bad}"}, peer="10.1.2.3")
+    assert m.mfa_rate_limit_key(req) == "ip:10.1.2.3"
+
+
+def test_expired_jwt_falls_back_to_ip(reload_key):
+    m = reload_key()
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+    payload = {
+        "user_id": 9,
+        "exp": datetime.utcnow() - timedelta(minutes=5),
+    }
+    expired = jwt.encode(payload, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
+    req = _make_request(headers={"authorization": f"Bearer {expired}"}, peer="10.0.0.7")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.7"
+
+
+# ---------------------------------------------------------------------------
+# Trusted-proxy X-Forwarded-For handling
+# ---------------------------------------------------------------------------
+
+
+def test_untrusted_peer_xff_is_ignored(reload_key):
+    """X-Forwarded-For from an untrusted client cannot move buckets."""
+    m = reload_key()  # no trusted CIDRs
+    req = _make_request(
+        headers={"x-forwarded-for": "1.2.3.4"},
+        peer="203.0.113.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.5"
+
+
+def test_trusted_peer_xff_is_honored(reload_key):
+    """Peer in trusted CIDR → first XFF hop becomes the bucket."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(
+        headers={"x-forwarded-for": "203.0.113.42, 10.0.0.99"},
+        peer="10.0.0.99",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:203.0.113.42"
+
+
+def test_trusted_cidr_multiple_ranges(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8, 172.16.0.0/12")
+    req = _make_request(
+        headers={"x-forwarded-for": "198.51.100.4"},
+        peer="172.16.5.5",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:198.51.100.4"
+
+
+def test_trusted_peer_no_xff_falls_back_to_peer(reload_key):
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    req = _make_request(peer="10.0.0.99")
+    assert m.mfa_rate_limit_key(req) == "ip:10.0.0.99"
+
+
+def test_invalid_cidr_in_env_is_logged_and_ignored(reload_key, caplog):
+    import logging
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limit_key"):
+        m = reload_key(MFA_TRUSTED_PROXY_CIDRS="not-a-cidr, 10.0.0.0/8")
+    assert any("ignoring invalid CIDR" in r.message for r in caplog.records)
+    # The valid one is still effective.
+    req = _make_request(
+        headers={"x-forwarded-for": "9.9.9.9"},
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "ip:9.9.9.9"
+
+
+def test_user_bucket_wins_over_xff(reload_key):
+    """Auth always wins, even from a trusted proxy."""
+    m = reload_key(MFA_TRUSTED_PROXY_CIDRS="10.0.0.0/8")
+    token = _mint_jwt(99)  # integer user_id claim
+    req = _make_request(
+        headers={
+            "authorization": f"Bearer {token}",
+            "x-forwarded-for": "1.1.1.1",
+        },
+        peer="10.0.0.1",
+    )
+    assert m.mfa_rate_limit_key(req) == "user:99"
+
+
+def test_no_client_in_scope_does_not_crash(reload_key):
+    m = reload_key()
+    scope = {
+        "type": "http",
+        "headers": [],
+        "client": None,
+        "method": "POST",
+        "path": "/api/mfa/enroll/start",
+        "query_string": b"",
+    }
+    req = Request(scope)
+    # Whatever it returns, it must be deterministic and not raise.
+    out = m.mfa_rate_limit_key(req)
+    assert out.startswith("ip:")

backend/tests/test_mfa_rate_limits.py

@@ -0,0 +1,93 @@
+"""Tests for middleware.mfa_rate_limits — env-driven MFA rate-limit config."""
+import importlib
+import logging
+
+import pytest
+
+
+@pytest.fixture
+def reload_module(monkeypatch):
+    """Helper: reload the module after env mutation so dataclass defaults
+    pick up the new values."""
+
+    def _reload(**env):
+        for key in list(globals().get('_OVERRIDDEN_ENVS', set())):
+            monkeypatch.delenv(key, raising=False)
+        for key, value in env.items():
+            monkeypatch.setenv(key, value)
+        from middleware import mfa_rate_limits as m
+        return importlib.reload(m)
+
+    return _reload
+
+
+def test_defaults_when_no_env(reload_module, monkeypatch):
+    """No env var set → secure defaults applied (user-aware key assumption)."""
+    for key in (
+        "MFA_RATE_LIMIT_ENROLL_START",
+        "MFA_RATE_LIMIT_ENROLL_CONFIRM",
+        "MFA_RATE_LIMIT_DISABLE",
+        "MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES",
+        "MFA_RATE_LIMIT_ADMIN_RESET",
+        "MFA_RATE_LIMIT_ADMIN_RESET_ALL",
+    ):
+        monkeypatch.delenv(key, raising=False)
+    m = reload_module()
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert m.MFA_LIMITS.enroll_confirm == "10/minute"
+    assert m.MFA_LIMITS.disable == "10/minute"
+    assert m.MFA_LIMITS.regenerate_backup_codes == "5/hour"
+    assert m.MFA_LIMITS.admin_reset == "60/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "1/day"
+
+
+def test_env_override_per_endpoint(reload_module):
+    m = reload_module(
+        MFA_RATE_LIMIT_ENROLL_START="100/hour",
+        MFA_RATE_LIMIT_ADMIN_RESET_ALL="3/day",
+    )
+    assert m.MFA_LIMITS.enroll_start == "100/hour"
+    assert m.MFA_LIMITS.admin_reset_all == "3/day"
+    # Untouched values still default.
+    assert m.MFA_LIMITS.disable == "10/minute"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "totally-bogus",
+        "5/lightyear",
+        "abc/minute",
+        "5",
+        "/minute",
+        "5//minute",
+        "",
+    ],
+)
+def test_invalid_format_falls_back_to_default(reload_module, caplog, bad):
+    with caplog.at_level(logging.WARNING, logger="middleware.mfa_rate_limits"):
+        m = reload_module(MFA_RATE_LIMIT_ENROLL_START=bad)
+    # Falls back to the secure default for enroll_start.
+    assert m.MFA_LIMITS.enroll_start == "10/minute"
+    assert any("not a valid slowapi limit string" in r.message for r in caplog.records)
+
+
+def test_whitespace_around_value_is_tolerated(reload_module):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE="  30/minute  ")
+    assert m.MFA_LIMITS.disable == "30/minute"
+
+
+@pytest.mark.parametrize(
+    "valid",
+    ["1/second", "100/minute", "1000/hour", "10/day"],
+)
+def test_all_valid_periods_accepted(reload_module, valid):
+    m = reload_module(MFA_RATE_LIMIT_DISABLE=valid)
+    assert m.MFA_LIMITS.disable == valid
+
+
+def test_dataclass_is_frozen(reload_module):
+    """Frozen dataclass guards against accidental mutation after import."""
+    m = reload_module()
+    with pytest.raises((AttributeError, Exception)):
+        m.MFA_LIMITS.disable = "999/second"  # type: ignore[misc]

backend/tests/test_mfa_service.py

@@ -0,0 +1,223 @@
+"""Unit tests for the MFA service layer (Issue #18, v1.6.0).
+
+These tests cover the pure-Python side of MFA — no DB, no FastAPI.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+import re
+import sys
+import time
+
+import pytest
+
+# Repo path setup (mirrors other tests in this folder).
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import pyotp  # noqa: E402
+
+from services import mfa_service  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# TOTP
+# ---------------------------------------------------------------------------
+
+
+class TestTotpSecret:
+    def test_secret_is_base32(self):
+        secret = mfa_service.generate_totp_secret()
+        # pyotp.random_base32() returns 32-character base32 strings.
+        assert len(secret) == 32
+        assert re.fullmatch(r"[A-Z2-7]+", secret), "secret must be valid base32"
+
+    def test_secrets_are_unique(self):
+        secrets = {mfa_service.generate_totp_secret() for _ in range(50)}
+        assert len(secrets) == 50
+
+
+class TestVerifyTotp:
+    def setup_method(self):
+        self.secret = mfa_service.generate_totp_secret()
+        self.totp = pyotp.TOTP(self.secret, digits=6, interval=30, digest="sha1")
+
+    def test_happy_path(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        assert step == int(time.time()) // 30
+
+    def test_invalid_code_format_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "abc", None)
+        assert ok is False and step is None
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "12345", None)
+        assert ok is False and step is None
+
+    def test_tolerance_minus_30s(self):
+        now = int(time.time())
+        previous_step = (now // 30) - 1
+        prev_code = self.totp.at(previous_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, prev_code, None)
+        assert ok is True
+        assert step == previous_step
+
+    def test_tolerance_plus_30s(self):
+        now = int(time.time())
+        next_step = (now // 30) + 1
+        next_code = self.totp.at(next_step * 30)
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, next_code, None)
+        assert ok is True
+        assert step == next_step
+
+    def test_replay_rejected(self):
+        code = self.totp.now()
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, code, None)
+        assert ok is True
+        # Submit again with the previously-consumed step: must be rejected.
+        ok2, step2 = mfa_service.verify_totp_with_replay_guard(self.secret, code, step)
+        assert ok2 is False
+        assert step2 is None
+
+    def test_wrong_code_rejected(self):
+        ok, step = mfa_service.verify_totp_with_replay_guard(self.secret, "000000", None)
+        assert ok is False
+        assert step is None
+
+
+# ---------------------------------------------------------------------------
+# Fernet + key resolution
+# ---------------------------------------------------------------------------
+
+
+class TestFernet:
+    def setup_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def teardown_method(self):
+        mfa_service.reset_fernet_for_tests()
+
+    def test_encrypt_decrypt_roundtrip_with_env_key(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        key = Fernet.generate_key().decode()
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", key)
+        mfa_service.reset_fernet_for_tests()
+
+        secret = "JBSWY3DPEHPK3PXP" * 2
+        token = mfa_service.encrypt_secret(secret)
+        assert token and token != secret
+        recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+
+    def test_decrypt_invalid_token_returns_none(self, monkeypatch):
+        from cryptography.fernet import Fernet
+        monkeypatch.setenv("MFA_ENCRYPTION_KEY", Fernet.generate_key().decode())
+        mfa_service.reset_fernet_for_tests()
+
+        assert mfa_service.decrypt_secret("not-a-valid-fernet-token") is None
+
+    def test_hkdf_fallback_when_env_unset(self, monkeypatch, caplog):
+        monkeypatch.delenv("MFA_ENCRYPTION_KEY", raising=False)
+        mfa_service.reset_fernet_for_tests()
+
+        with caplog.at_level("WARNING"):
+            secret = "JBSWY3DPEHPK3PXPJBSWY3DPEHPK3PXP"
+            token = mfa_service.encrypt_secret(secret)
+            recovered = mfa_service.decrypt_secret(token)
+        assert recovered == secret
+        assert any("MFA_ENCRYPTION_KEY" in r.message for r in caplog.records), (
+            "expected a WARN log when falling back to SECRET_KEY derivation"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Backup codes
+# ---------------------------------------------------------------------------
+
+
+class TestBackupCodes:
+    def test_generate_count_and_format(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(codes) == 10
+        # 31-char alphabet: A-H J K M N P-Z 2-9 (excludes I, L, O, 0, 1).
+        for code in codes:
+            assert re.fullmatch(r"[A-HJKM-NP-Z2-9]{4}-[A-HJKM-NP-Z2-9]{4}", code), code
+
+    def test_alphabet_excludes_confusing_characters(self):
+        # Generate enough codes to virtually guarantee any forbidden char would surface.
+        for _ in range(20):
+            codes = mfa_service.generate_backup_codes()
+            for code in codes:
+                for ch in code.replace("-", ""):
+                    assert ch not in "0O1IL", f"forbidden char {ch!r} in {code!r}"
+
+    def test_codes_are_unique(self):
+        codes = mfa_service.generate_backup_codes()
+        assert len(set(codes)) == len(codes)
+
+    def test_normalize_strips_case_dash_space(self):
+        assert mfa_service.normalize_backup_code("abcd-efgh") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code(" ab cd-ef gh ") == "ABCDEFGH"
+        assert mfa_service.normalize_backup_code("") == ""
+        assert mfa_service.normalize_backup_code(None) == ""  # type: ignore[arg-type]
+
+    def test_hash_and_check_async(self):
+        async def _run():
+            plain = mfa_service.generate_backup_codes()[:1]
+            hashes = await mfa_service.hash_backup_codes(plain)
+            assert len(hashes) == 1
+            assert await mfa_service.check_backup_code(plain[0], hashes[0]) is True
+            assert await mfa_service.check_backup_code("WRONG-CODE", hashes[0]) is False
+            # Case + dash normalization
+            assert await mfa_service.check_backup_code(plain[0].lower(), hashes[0]) is True
+            assert (
+                await mfa_service.check_backup_code(plain[0].replace("-", ""), hashes[0])
+                is True
+            )
+
+        asyncio.run(_run())
+
+
+# ---------------------------------------------------------------------------
+# otpauth URI
+# ---------------------------------------------------------------------------
+
+
+class TestOtpAuthUri:
+    def test_uri_shape(self):
+        uri = mfa_service.build_otpauth_uri("alice@example.com", "JBSWY3DPEHPK3PXP")
+        assert uri.startswith("otpauth://totp/")
+        assert "secret=JBSWY3DPEHPK3PXP" in uri
+        assert "issuer=" in uri
+        assert "algorithm=SHA1" in uri
+        assert "digits=6" in uri
+        assert "period=30" in uri
+
+    def test_account_label_env_override(self, monkeypatch):
+        monkeypatch.setenv("MFA_ACCOUNT_LABEL_DOMAIN", "ops.example.com")
+        label = mfa_service.build_account_label("alice", hostname_hint="ignored.com")
+        assert label == "alice@ops.example.com"
+
+    def test_account_label_hostname_hint(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint="api.local")
+        assert label == "alice@api.local"
+
+    def test_account_label_fallback(self, monkeypatch):
+        monkeypatch.delenv("MFA_ACCOUNT_LABEL_DOMAIN", raising=False)
+        label = mfa_service.build_account_label("alice", hostname_hint=None)
+        assert label == "alice@haproxy-openmanager"
+
+
+# ---------------------------------------------------------------------------
+# Challenge token
+# ---------------------------------------------------------------------------
+
+
+class TestChallengeToken:
+    def test_length_and_uniqueness(self):
+        tokens = {mfa_service.generate_challenge_token() for _ in range(50)}
+        assert len(tokens) == 50
+        for t in tokens:
+            assert len(t) == 64
+            assert re.fullmatch(r"[0-9a-f]{64}", t)

backend/tests/test_vip_delete_approval.py

@@ -0,0 +1,69 @@
+"""Issue #27 safety (v1.7.2) — source-level guards for APPROVAL-GATED VIP deletion.
+
+The critical invariant: an agent must NEVER tear a VIP down without an explicit human approval.
+We enforce that by keeping the VIP is_active=TRUE (so the agent keeps serving it) when a delete
+is merely *requested*; only an APPROVE (apply) flips is_active=FALSE. These guards lock in that
+wiring so a future edit can't silently make delete immediate again. Pure (no DB)."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_delete_stages_for_approval_keeps_vip_running():
+    s = _read("routers/vip.py")
+    # A running (applied) VIP's delete sets pending_delete=TRUE and stages a delete version —
+    # it must NOT flip is_active=FALSE in delete_vip (that only happens on approval in apply_vip).
+    assert "pending_delete=TRUE" in s
+    assert '_stage_vip_version(conn, vip_id, "delete"' in s
+    # The staged-delete branch (running VIP) must not contain an is_active=FALSE soft-delete;
+    # only the "never applied" branch may remove immediately (guarded by applied_snapshot IS NULL).
+    assert 'v["applied_snapshot"] is None' in s
+
+
+def test_apply_performs_delete_only_on_approval():
+    s = _read("routers/vip.py")
+    # apply_vip short-circuits on pending_delete and only THEN flips is_active=FALSE.
+    assert 'if v["pending_delete"]:' in s
+    assert "is_active=FALSE, pending_delete=FALSE" in s
+
+
+def test_reject_cancels_delete_as_noop():
+    s = _read("routers/vip.py")
+    # reject_vip clears the staged delete + purge intent; the VIP keeps running (is_active never
+    # touched here) — the "nothing happened" path.
+    assert "pending_delete=FALSE, purge_on_teardown=FALSE" in s
+    assert "Deletion rejected" in s
+
+
+def test_agent_delivery_teardown_only_on_inactive():
+    # The agent is told to tear down ONLY when is_active=FALSE; a pending-delete VIP is still
+    # is_active=TRUE, so the delivery returns 'available' and the agent keeps serving it.
+    s = _read("routers/agent.py")
+    assert "if not row['is_active']:" in s
+    assert '"status": "teardown"' in s
+
+
+def test_migration_adds_pending_delete_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "pending_delete BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7
+
+
+def test_list_visibility_backward_compat_gates_on_applied():
+    # A VIP soft-deleted under the OLD immediate-delete (pre-1.7.2: is_active=FALSE,
+    # last_config_status='PENDING') must NOT reappear in the list as DELETING — the
+    # teardown-tracking clause is gated on last_config_status='APPLIED' (new-flow approved
+    # deletes only). Locks the backward-compat fix.
+    s = _read("routers/vip.py")
+    assert "v.last_config_status = 'APPLIED' AND EXISTS" in s

backend/tests/test_vip_diff_minimal.py

@@ -0,0 +1,34 @@
+"""v1.7.5 — guard for the HA/VIP "View Change" diff.
+
+Editing a VIP (e.g. priority + virtual IP) must show a REAL line diff (only the changed lines)
+against the previous applied vip-* config — not the whole keepalived.conf marked as "added", and
+without the doubled "+ +" prefix (the line must be stored WITHOUT a +/- prefix; the UI adds it
+from `type`, exactly like the standard haproxy diff). Pure source guard (the behavioural diff
+test needs a DB); locks the wiring so it can't regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _cluster_src() -> str:
+    with open(os.path.join(ROOT, "routers", "cluster.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_vip_diff_uses_real_difflib_against_previous_applied():
+    s = _cluster_src()
+    # The previous APPLIED vip-* config for this VIP is fetched as the diff baseline.
+    assert "AND status='APPLIED' AND cluster_id=$2 AND id < $3 ORDER BY id DESC LIMIT 1" in s
+    # And the diff is a real unified_diff of old vs new (not "everything added").
+    assert "difflib.unified_diff(old_content.split('\\n'), new_content.split('\\n')" in s
+
+
+def test_vip_diff_stores_lines_without_prefix():
+    s = _cluster_src()
+    # The old vip bug prepended "+ {line}", doubling the UI prefix ("+ +"). The vip diff now
+    # stores the stripped diff line (dl[1:]) — the UI adds the +/- from `type`. `dl` is unique
+    # to the vip branch (the standard haproxy diff uses `line`), so this targets the vip fix
+    # only and does not touch the (separate, out-of-scope) SSL diff branch.
+    assert '"line": dl[1:], "line_number": line_number' in s

backend/tests/test_vip_models.py

@@ -0,0 +1,91 @@
+"""Issue #27 (v1.7.0) — unit tests for VIP request-model validation (pure, no DB)."""
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from models.vip import VIPCreate, VIPMemberIn  # noqa: E402
+
+
+def _members(master_prio=150, backup_prio=100, n_backup=1):
+    members = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": master_prio}]
+    for i in range(n_backup):
+        members.append({"agent_id": 2 + i, "network_interface": "eth0", "role": "BACKUP", "priority": backup_prio})
+    return members
+
+
+def _create(**over):
+    base = dict(name="web-vip", pool_id=1, virtual_ip="10.0.0.100", members=_members())
+    base.update(over)
+    return VIPCreate(**base)
+
+
+class TestField:
+    def test_ok(self):
+        v = _create()
+        assert v.virtual_ip == "10.0.0.100"
+        assert v.use_unicast is True  # cloud-safe default
+
+    def test_ipv6_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="fd00::1")
+
+    def test_bad_ip_rejected(self):
+        with pytest.raises(ValueError):
+            _create(virtual_ip="not-an-ip")
+
+    def test_auth_pass_max_8(self):
+        _create(auth_pass="12345678")
+        with pytest.raises(ValueError):
+            _create(auth_pass="123456789")
+
+    def test_interface_forbidden_char(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0; rm -rf /", role="BACKUP", priority=100)
+
+    def test_priority_range(self):
+        with pytest.raises(ValueError):
+            VIPMemberIn(agent_id=1, network_interface="eth0", role="BACKUP", priority=255)
+
+    def test_role_normalized(self):
+        m = VIPMemberIn(agent_id=1, network_interface="eth0", role="master", priority=150)
+        assert m.role == "MASTER"
+
+    def test_vrid_range(self):
+        with pytest.raises(ValueError):
+            _create(virtual_router_id=300)
+
+
+class TestMembers:
+    def test_single_node_allowed(self):
+        # A single-node VIP (one MASTER, no BACKUP) is valid: a keepalived-managed floating
+        # IP without failover (Issue #27 follow-up — relaxed from >=2 members to >=1, e.g. a
+        # one-box cluster that wants a stable VIP, or before a 2nd node is added for real HA).
+        v = VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5",
+                      members=[{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150}])
+        assert len(v.members) == 1 and v.members[0].role == "MASTER"
+
+    def test_needs_at_least_one(self):
+        # An empty membership is still rejected — a VIP must have at least one node.
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=[])
+
+    def test_exactly_one_master(self):
+        bad = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 2, "network_interface": "eth0", "role": "MASTER", "priority": 140}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=bad)
+
+    def test_master_must_be_highest(self):
+        with pytest.raises(ValueError):
+            _create(members=_members(master_prio=100, backup_prio=120))
+
+    def test_no_duplicate_agent(self):
+        dup = [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150},
+               {"agent_id": 1, "network_interface": "eth1", "role": "BACKUP", "priority": 100}]
+        with pytest.raises(ValueError):
+            VIPCreate(name="x", pool_id=1, virtual_ip="10.0.0.5", members=dup)

backend/tests/test_vip_purge.py

@@ -0,0 +1,51 @@
+"""Issue #27 follow-up (v1.7.2) — source-level guards for the opt-in keepalived package
+uninstall + the "we installed it" marker. Pure (no DB), mirroring the project's other
+source-assertion tests: they lock in the *safe defaults* so a future edit can't silently
+turn routine VIP deletion into a package purge, or purge a package we didn't install."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(rel: str) -> str:
+    with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_agent_script_purge_is_optin_and_marker_guarded():
+    s = _read("utils/agent_scripts/linux_install.sh")
+    # The install marker is written only when WE install keepalived, and read by the purge
+    # guard — present in BOTH function copies (installer-mode + SKIP_TO_DAEMON).
+    assert s.count(".hom_installed") >= 4
+    # Multi-distro purge cascade exists, but ONLY inside the opt-in branch.
+    assert "apt-get purge -y -qq keepalived" in s
+    assert "apk del keepalived" in s
+    # Orphan self-heal (not_configured) must NEVER purge — graceful teardown only, both copies.
+    assert s.count('_kp_teardown "false"') >= 2
+    # Purge is honored only on an explicit teardown, parsed from the delivery response.
+    assert s.count(".purge // false") >= 2
+
+
+def test_agent_delivery_signals_purge_on_teardown():
+    s = _read("routers/agent.py")
+    assert "v.purge_on_teardown" in s
+    assert '"purge"' in s  # teardown response carries the opt-in flag
+
+
+def test_delete_endpoint_accepts_purge_package_default_off():
+    s = _read("routers/vip.py")
+    # Query param defaults to False (safe), and it sets the persisted teardown flag.
+    assert "purge_package: bool = False" in s
+    assert "purge_on_teardown=$2" in s
+
+
+def test_migration_adds_purge_column_and_bumps_schema():
+    import re
+    s = _read("database/migrations.py")
+    assert "purge_on_teardown BOOLEAN NOT NULL DEFAULT FALSE" in s
+    # Schema was bumped to accommodate this column. Assert >= 7 (the version it landed in)
+    # rather than pinning an exact value, so later schema bumps don't re-break this test.
+    m = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", s, re.MULTILINE)
+    assert m is not None and int(m.group(1)) >= 7

backend/tests/test_waf_stick_table_autoinject.py

@@ -0,0 +1,34 @@
+"""v1.7.4 — guard for the stick-table auto-inject in the HAProxy config renderer.
+
+A frontend that uses a stick counter (`track-sc<N>` or an `sc_*_rate(...)` fetch) but declares
+no `stick-table` makes HAProxy fatally reject the WHOLE cluster config with "table '<frontend>'
+used but not configured". This happens with rate-limit directives baked into a frontend's stored
+request_headers/options by an older version or a config import. The renderer now injects a default
+stick-table in that case. Pure source guard (the behavioural renderer test needs a DB and is
+disabled); this locks the wiring so it can't silently regress."""
+from __future__ import annotations
+
+import os
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src() -> str:
+    with open(os.path.join(ROOT, "services", "haproxy_config.py"), encoding="utf-8") as f:
+        return f.read()
+
+
+def test_renderer_detects_stick_counter_usage():
+    s = _src()
+    # Detection: any track-sc<N> write OR an sc_*_rate(...) fetch flips _sc_counter_used.
+    assert "_sc_counter_used" in s
+    assert '"track-sc" in stripped or ("sc_" in stripped and "_rate(" in stripped)' in s
+
+
+def test_renderer_injects_stick_table_when_missing():
+    s = _src()
+    # Inject ONLY when a counter is used AND no stick-table was emitted (purely additive).
+    assert "if _sc_counter_used and not _stick_table_emitted:" in s
+    assert 'stick-table type ip size 100k expire 30s store http_req_rate(10s)' in s
+    # The inject must register the table so it is not added twice.
+    assert "_stick_table_emitted = True" in s

backend/utils/agent_scripts/linux_install.sh

@@ -153,7 +153,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -470,7 +470,12 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "haproxy-agent.service"; do
+# issue #31: match ONLY the installed agent (binary path + service), never the bare string
+# "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command line
+# or a sudo/PAM ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself ("Killed", install aborts). The systemd service is also stopped below; the
+# "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "haproxy-agent.service"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -635,7 +640,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -1055,7 +1060,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1289,7 +1299,7 @@ collect_system_info() {
     "disk_space": $disk_bytes,
     "ip_address": "$ip_address",
     "network_interfaces": ["${network_interfaces//,/\",\"}"],
-    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service"]
+    "capabilities": ["haproxy_management", "ssl_deployment", "config_reload", "systemd_service", "keepalived_management"]
 SYSTEM_INFO_EOF
 }
 
@@ -1309,7 +1319,7 @@ get_keepalive_state() {
     # Method 1: journalctl (most reliable on RHEL/CentOS/Ubuntu with systemd)
     if command -v journalctl &>/dev/null; then
         state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-            | grep -oE "(MASTER|BACKUP)" | tail -1)
+            | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
     fi
 
     # Method 2: Fallback to log files (for non-systemd or restricted journalctl)
@@ -1317,22 +1327,27 @@ get_keepalive_state() {
         for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
             if [[ -r "$logfile" ]]; then
                 state=$(tail -200 "$logfile" 2>/dev/null \
-                    | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                    | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                 [[ -n "$state" ]] && break
             fi
         done
     fi
 
-    # Method 3: Check VIP presence on network interfaces (confirms MASTER)
+    # Method 3: VIP presence on local interfaces — the most portable signal, independent
+    # of logging/journald (works on any distro / init system / keepalived install type).
+    # MASTER iff ANY configured VIP is actually held locally; keepalived up but holding no
+    # VIP => BACKUP. Exact whole-line IP match (grep -Fxq) so 10.0.0.1 can't falsely match
+    # 10.0.0.10/100, and ALL configured VIPs are checked (not just the first).
     if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-        local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-            | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
-        if [[ -n "$conf_vip" ]]; then
-            if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
-                state="MASTER"
-            else
-                state="BACKUP"
-            fi
+        local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+            | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
+        if [[ -n "$conf_vips" ]]; then
+            local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
+            state="BACKUP"
+            local _cvip
+            for _cvip in $conf_vips; do
+                if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+            done
         fi
     fi
 
@@ -1352,7 +1367,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -1665,6 +1685,163 @@ check_ssl_updates() {
 }
 
 # Reload HAProxy service (Linux specific)
+# Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence. OPT-IN and inert by default:
+# a node with no APPLIED VIP gets status=not_configured and (lacking our ownership
+# marker) does NOTHING. Never clobbers a hand-managed keepalived. Uses $AGENT_TOKEN
+# (kept in sync with the live token in both daemon loops).
+fetch_and_deploy_keepalived_config() {
+    local marker="# Managed by HAProxy OpenManager"
+    local resp status http_code we_own="false" conf chk
+
+    # Timeouts so a hung management server can never stall the daemon loop.
+    resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+        "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    http_code="${resp##*$'\n'}"   # last line = HTTP status
+    resp="${resp%$'\n'*}"          # everything before = JSON body
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+    # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+    # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+    [[ "$http_code" == "404" ]] && status="teardown"
+    # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+    conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+    [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+    chk="$(dirname "$conf")/check_haproxy.sh"
+    if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+    _kp_report() {  # $1=state  $2=vip_id(or empty)  $3=hash  $4=message
+        local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+        curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+            >/dev/null 2>&1 || true
+    }
+    _kp_teardown() {
+        # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+        # keepalived and remove OUR config (the VIP is released) but KEEP the package — the safe
+        # enterprise default. Purge only when opted in AND we are the ones who installed it
+        # (the .hom_installed marker); never remove a package the admin pre-installed.
+        local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+        log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+        systemctl stop keepalived >/dev/null 2>&1
+        systemctl disable keepalived >/dev/null 2>&1
+        rm -f "$conf" "$chk"
+        if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+            log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+            fi
+            rm -f "$marker_inst"
+            if command -v keepalived >/dev/null 2>&1; then
+                _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+            else
+                _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+            fi
+        elif [[ "$purge" == "true" ]]; then
+            log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+            _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+        else
+            _kp_report "disabled" "" "" "torn down"
+        fi
+    }
+
+    case "$status" in
+        available) : ;;  # fall through to converge
+        teardown)
+            # Explicit server-side delete. Honor the operator's opt-in package purge (.purge);
+            # marker-guarded inside _kp_teardown, and a node we don't own (no marker conf) is a no-op.
+            local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+            [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"
+            return 0 ;;
+        not_configured)
+            [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only, never purge
+            return 0 ;;
+        *) return 0 ;;   # unknown / auth error → no-op
+    esac
+
+    # --- status == available: converge ---
+    local vip_id new_conf check_script install_if new_hash
+    vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+    new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+    new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+    check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+    install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+    [[ -z "$new_conf" ]] && return 0
+
+    # Ownership guard: never overwrite a keepalived.conf we don't own.
+    if [[ -f "$conf" && "$we_own" != "true" ]]; then
+        log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+        _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+        return 0
+    fi
+
+    # Hybrid install: install keepalived only if missing.
+    if ! command -v keepalived >/dev/null 2>&1; then
+        if [[ "$install_if" == "true" ]]; then
+            log "INFO" "KEEPALIVED: installing package..."
+            if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+            elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+            elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+            elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+            elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+            fi
+        fi
+        if ! command -v keepalived >/dev/null 2>&1; then
+            log "ERROR" "KEEPALIVED: not installed/available"
+            _kp_report "error" "$vip_id" "" "keepalived not installed"
+            return 0
+        fi
+        # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+        # delete removes only OUR install, never an admin's pre-existing keepalived package.
+        mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+    fi
+
+    # Idempotency: skip write+reload when the on-disk content already matches.
+    if [[ -f "$conf" ]]; then
+        local cur_hash would_hash
+        cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+        would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+        if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            return 0
+        fi
+    fi
+
+    mkdir -p "$(dirname "$conf")"
+    if [[ -n "$check_script" ]]; then
+        printf '%s' "$check_script" > "$chk"
+        chown root:root "$chk" 2>/dev/null
+        chmod 0755 "$chk"   # root-owned, not world-writable — required by enable_script_security
+    fi
+    # Validate on a TEMP file and swap in only on success: a bad render must never land on
+    # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+    local tmp_conf="${conf}.hom.tmp"
+    printf '%s' "$new_conf" > "$tmp_conf"
+    chmod 0644 "$tmp_conf"
+    if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+        rm -f "$tmp_conf"
+        log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+        _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+        return 0
+    fi
+    mv -f "$tmp_conf" "$conf"
+    chown root:root "$conf" 2>/dev/null
+    chmod 0644 "$conf"
+    systemctl enable keepalived >/dev/null 2>&1
+    if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+        log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+        _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+    else
+        log "ERROR" "KEEPALIVED: reload/restart failed"
+        _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+    fi
+    return 0
+}
+
 reload_haproxy_service() {
     log "INFO" "Performing zero-downtime HAProxy configuration reload..."
     
@@ -2356,8 +2533,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence at the SSL cadence (inert for non-VIP nodes)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2789,28 +2970,31 @@ SYSTEM_INFO_EOF
 
         if command -v journalctl &>/dev/null; then
             state=$(journalctl -u keepalived -n 50 --no-pager 2>/dev/null \
-                | grep -oE "(MASTER|BACKUP)" | tail -1)
+                | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
         fi
 
         if [[ -z "$state" ]]; then
             for logfile in /var/log/messages /var/log/syslog /var/log/keepalived.log; do
                 if [[ -r "$logfile" ]]; then
                     state=$(tail -200 "$logfile" 2>/dev/null \
-                        | grep -i keepalived | grep -oE "(MASTER|BACKUP)" | tail -1)
+                        | grep -i keepalived | grep -oE "(MASTER|BACKUP|FAULT)" | tail -1)
                     [[ -n "$state" ]] && break
                 fi
             done
         fi
 
+        # Method 3: VIP presence on local interfaces — portable, logging-independent.
+        # MASTER iff ANY configured VIP is held locally (exact whole-line match, all VIPs).
         if [[ -z "$state" ]] && [[ -r /etc/keepalived/keepalived.conf ]]; then
-            local conf_vip=$(grep -A10 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
-                | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -1)
-            if [[ -n "$conf_vip" ]]; then
-                if ip addr show 2>/dev/null | grep -q "$conf_vip"; then
-                    state="MASTER"
-                else
-                    state="BACKUP"
-                fi
+            local conf_vips=$(grep -A20 'virtual_ipaddress' /etc/keepalived/keepalived.conf 2>/dev/null \
+                | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}')
+            if [[ -n "$conf_vips" ]]; then
+                local local_ips=$(ip -4 -o addr show 2>/dev/null | grep -oE 'inet ([0-9]{1,3}\.){3}[0-9]{1,3}' | awk '{print $2}')
+                state="BACKUP"
+                local _cvip
+                for _cvip in $conf_vips; do
+                    if printf '%s\n' "$local_ips" | grep -Fxq "$_cvip"; then state="MASTER"; break; fi
+                done
             fi
         fi
 
@@ -2849,7 +3033,15 @@ SYSTEM_INFO_EOF
         local keepalive_info=$(get_keepalive_state)
         local keepalive_state=$(echo "$keepalive_info" | cut -d'|' -f1)
         local keepalive_ip=$(echo "$keepalive_info" | cut -d'|' -f2)
-        
+
+        # Network interfaces (DAEMON) — reported so the HA/VIP create form can offer this
+        # node's real NICs. /sys/class/net is universal across distros (no iproute2 needed);
+        # exclude loopback here, and the UI further hides docker/veth/bridge interfaces. Empty
+        # -> "[]" (never "[\"\"]") so a node with no NICs doesn't surface a blank option.
+        local net_ifaces ni_json
+        net_ifaces=$(ls /sys/class/net 2>/dev/null | grep -vE '^lo$' | tr '\n' ',' | sed 's/,$//')
+        if [ -n "$net_ifaces" ]; then ni_json="[\"${net_ifaces//,/\",\"}\"]"; else ni_json="[]"; fi
+
         # Prepare heartbeat payload with all data
         local heartbeat_payload="{
             \"name\": \"$AGENT_NAME\",
@@ -2859,6 +3051,8 @@ SYSTEM_INFO_EOF
             \"platform\": \"$platform\",
             \"architecture\": \"$(uname -m)\",
             \"version\": \"{{AGENT_VERSION}}\",
+            \"capabilities\": [\"haproxy_management\", \"ssl_deployment\", \"config_reload\", \"systemd_service\", \"keepalived_management\"],
+            \"network_interfaces\": $ni_json,
             \"haproxy_status\": \"$haproxy_status\",
             \"haproxy_version\": \"$haproxy_version\",
             \"cluster_id\": $CLUSTER_ID,
@@ -3041,9 +3235,159 @@ CONFIG_RESPONSE_EOF
     HAPROXY_BIN="${HAPROXY_BIN_PATH}"
     HAPROXY_CONFIG="${HAPROXY_CONFIG_PATH}"
     log "INFO" "DAEMON: Initialized HAProxy paths - bin: $HAPROXY_BIN, config: $HAPROXY_CONFIG"
-    
+
+    # Issue #27 (v1.7.0) — HA/VIP (Keepalived) convergence (DAEMON copy). Inert by
+    # default: a node with no APPLIED VIP gets not_configured and (lacking our marker)
+    # does nothing. Never clobbers a hand-managed keepalived. $AGENT_TOKEN is synced to
+    # the live token at the top of each loop iteration below.
+    fetch_and_deploy_keepalived_config() {
+        local marker="# Managed by HAProxy OpenManager"
+        local resp status http_code we_own="false" conf chk
+
+        # Timeouts so a hung management server can never stall the daemon loop.
+        resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
+            "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        http_code="${resp##*$'\n'}"   # last line = HTTP status
+        resp="${resp%$'\n'*}"          # everything before = JSON body
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        # A 404 means this agent was deleted server-side (node decommissioned / pool removed):
+        # self-heal teardown OUR keepalived so a removed node stops advertising the VIP. The
+        # teardown branch below is marker-guarded, so a node we don't manage stays untouched.
+        [[ "$http_code" == "404" ]] && status="teardown"
+        # Cluster-driven keepalived.conf path (delivered in every response); default is universal.
+        conf=$(echo "$resp" | jq -r '.config_path // empty' 2>/dev/null)
+        [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
+        chk="$(dirname "$conf")/check_haproxy.sh"
+        if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+
+        _kp_report() {
+            local vid="${2:-null}"; [[ -z "$2" ]] && vid="null"
+            curl -k -s --connect-timeout 10 --max-time 30 -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vid},\"state\":\"$1\",\"config_hash\":\"${3:-}\",\"message\":\"${4:-}\"}" \
+                >/dev/null 2>&1 || true
+        }
+        _kp_teardown() {
+            # $1=purge ("true" only on an explicit operator opt-in delete). Default: stop+disable
+            # keepalived and remove OUR config (VIP released) but KEEP the package. Purge only when
+            # opted in AND we installed it (.hom_installed marker) — never an admin's package.
+            local purge="${1:-false}" marker_inst; marker_inst="$(dirname "$conf")/.hom_installed"
+            log "INFO" "KEEPALIVED: tearing down our managed VIP config (purge=$purge)"
+            systemctl stop keepalived >/dev/null 2>&1
+            systemctl disable keepalived >/dev/null 2>&1
+            rm -f "$conf" "$chk"
+            if [[ "$purge" == "true" && -f "$marker_inst" ]]; then
+                log "INFO" "KEEPALIVED: uninstalling keepalived package (operator opt-in)"
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get purge -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf remove -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum remove -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive remove -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk del keepalived >/dev/null 2>&1
+                fi
+                rm -f "$marker_inst"
+                if command -v keepalived >/dev/null 2>&1; then
+                    _kp_report "disabled" "" "" "config removed; package uninstall attempted (still present)"
+                else
+                    _kp_report "disabled" "" "" "torn down + keepalived package uninstalled"
+                fi
+            elif [[ "$purge" == "true" ]]; then
+                log "INFO" "KEEPALIVED: package was pre-existing (not installed by us) — left in place; our config removed"
+                _kp_report "disabled" "" "" "torn down (package left: pre-existing, not installed by us)"
+            else
+                _kp_report "disabled" "" "" "torn down"
+            fi
+        }
+
+        case "$status" in
+            available) : ;;
+            teardown)
+                local _kp_purge; _kp_purge=$(echo "$resp" | jq -r '.purge // false' 2>/dev/null)
+                [[ "$we_own" == "true" ]] && _kp_teardown "$_kp_purge"   # explicit delete: honor opt-in purge
+                return 0 ;;
+            not_configured)
+                [[ "$we_own" == "true" ]] && _kp_teardown "false"   # T-2 orphan self-heal: graceful only
+                return 0 ;;
+            *) return 0 ;;
+        esac
+
+        local vip_id new_conf check_script install_if new_hash
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        new_conf=$(echo "$resp" | jq -r '.keepalived.config_content // empty' 2>/dev/null)
+        new_hash=$(echo "$resp" | jq -r '.keepalived.config_hash // empty' 2>/dev/null)
+        check_script=$(echo "$resp" | jq -r '.keepalived.check_script // empty' 2>/dev/null)
+        install_if=$(echo "$resp" | jq -r '.keepalived.install_if_missing // false' 2>/dev/null)
+        [[ -z "$new_conf" ]] && return 0
+
+        if [[ -f "$conf" && "$we_own" != "true" ]]; then
+            log "WARN" "KEEPALIVED: $conf is externally managed — refusing to overwrite"
+            _kp_report "externally_managed" "$vip_id" "" "pre-existing unmanaged keepalived.conf"
+            return 0
+        fi
+
+        if ! command -v keepalived >/dev/null 2>&1; then
+            if [[ "$install_if" == "true" ]]; then
+                log "INFO" "KEEPALIVED: installing package..."
+                if command -v apt-get >/dev/null 2>&1; then timeout 300 apt-get install -y -qq keepalived >/dev/null 2>&1
+                elif command -v dnf >/dev/null 2>&1; then timeout 300 dnf install -y -q keepalived >/dev/null 2>&1
+                elif command -v yum >/dev/null 2>&1; then timeout 300 yum install -y -q keepalived >/dev/null 2>&1
+                elif command -v zypper >/dev/null 2>&1; then timeout 300 zypper --non-interactive install -y keepalived >/dev/null 2>&1
+                elif command -v apk >/dev/null 2>&1; then timeout 300 apk add --no-cache keepalived >/dev/null 2>&1
+                fi
+            fi
+            if ! command -v keepalived >/dev/null 2>&1; then
+                log "ERROR" "KEEPALIVED: not installed/available"
+                _kp_report "error" "$vip_id" "" "keepalived not installed"
+                return 0
+            fi
+            # We installed keepalived (it was missing) — drop a marker so an opt-in uninstall on
+            # delete removes only OUR install, never an admin's pre-existing keepalived package.
+            mkdir -p "$(dirname "$conf")" 2>/dev/null; : > "$(dirname "$conf")/.hom_installed" 2>/dev/null
+        fi
+
+        if [[ -f "$conf" ]]; then
+            local cur_hash would_hash
+            cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
+            would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
+            if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                return 0
+            fi
+        fi
+
+        mkdir -p "$(dirname "$conf")"
+        if [[ -n "$check_script" ]]; then
+            printf '%s' "$check_script" > "$chk"
+            chown root:root "$chk" 2>/dev/null
+            chmod 0755 "$chk"
+        fi
+        # Validate on a TEMP file and swap in only on success: a bad render must never land on
+        # $conf (it would also make the md5 idempotency guard above suppress retries forever).
+        local tmp_conf="${conf}.hom.tmp"
+        printf '%s' "$new_conf" > "$tmp_conf"
+        chmod 0644 "$tmp_conf"
+        if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
+            rm -f "$tmp_conf"
+            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
+            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
+            return 0
+        fi
+        mv -f "$tmp_conf" "$conf"
+        chown root:root "$conf" 2>/dev/null
+        chmod 0644 "$conf"
+        systemctl enable keepalived >/dev/null 2>&1
+        if systemctl reload keepalived >/dev/null 2>&1 || systemctl restart keepalived >/dev/null 2>&1; then
+            log "INFO" "KEEPALIVED: applied config for VIP ${vip_id}"
+            _kp_report "enabled" "$vip_id" "$new_hash" "applied"
+        else
+            log "ERROR" "KEEPALIVED: reload/restart failed"
+            _kp_report "error" "$vip_id" "$new_hash" "reload/restart failed"
+        fi
+        return 0
+    }
+
     log "DEBUG" "DAEMON: Starting agent monitoring loop for $AGENT_NAME"
-    
+
     # Enhanced daemon loop with upgrade capability - EXACT MacOS COPY
     while true; do
         sleep 30
@@ -3208,6 +3552,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP (Keepalived) convergence at the SSL cadence (~2.5 min);
+        # inert for non-VIP nodes (status=not_configured + no ownership marker).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/agent_scripts/macos_install.sh

@@ -318,7 +318,11 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "com.haproxy.agent"; do
+# issue #31: match ONLY the installed agent (binary path + LaunchDaemon label), never the bare
+# string "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command
+# line or a sudo ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself. The "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "com.haproxy.agent"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -522,7 +526,7 @@ if [[ "$SKIP_TO_DAEMON" != "true" ]]; then
 
     # Validate cluster exists by checking management API
     log "DEBUG" "Validating HAProxy Cluster..."
-    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
+    CLUSTER_CHECK=$("$CURL_BIN" -k -s -f "$MANAGEMENT_URL/api/clusters" -H "X-API-Key: $AGENT_TOKEN" -H "User-Agent: haproxy-agent-installer" || echo "FAILED")
     if [[ "$CLUSTER_CHECK" == "FAILED" ]]; then
         log "ERROR" "Failed to connect to management API!"
         echo "   Please check your network connection and management URL."
@@ -867,7 +871,7 @@ SSL_SYNC_TIMESTAMP_FILE="/tmp/haproxy-agent-ssl-sync-${AGENT_NAME}"
 get_cluster_paths() {
     log "DEBUG" "Fetching cluster paths from management API"
     local cluster_response=$("$CURL_BIN" -k -s -X GET "$MANAGEMENT_URL/api/clusters" \
-        -H "Authorization: Bearer $AGENT_TOKEN")
+        -H "X-API-Key: $AGENT_TOKEN")
     
     if [[ $? -eq 0 ]]; then
         local _cfg _bin _sock
@@ -920,7 +924,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1164,13 +1173,39 @@ get_keepalive_state() {
     return 0
 }
 
+# Issue #27 — keepalived/VRRP is Linux-only. macOS agents cannot run keepalived, so they
+# can never be VIP members (the backend also withholds the keepalived_management capability,
+# so the UI warns at assign time). This is a safe no-op that, if a VIP is somehow mis-assigned
+# to a macOS node, reports a clear "unsupported on macOS" status instead of silently never
+# converging. It never installs/writes anything.
+fetch_and_deploy_keepalived_config() {
+    local resp status vip_id
+    resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+        -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+    [[ -z "$resp" ]] && return 0
+    status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+    if [[ "$status" == "available" ]]; then
+        vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+        log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+        curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+            -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+            -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+    fi
+    return 0
+}
+
 # Send heartbeat
 send_heartbeat() {
     local haproxy_status=$(get_haproxy_status)
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then
@@ -2197,8 +2232,12 @@ run_daemon() {
         _loop_count=$((_loop_count + 1))
         if (( _loop_count % 5 == 0 )); then
             check_ssl_updates
+            # Issue #27 — HA/VIP convergence (no-op on macOS; reports unsupported if assigned)
+            if type fetch_and_deploy_keepalived_config &>/dev/null; then
+                fetch_and_deploy_keepalived_config
+            fi
         fi
-        
+
         check_agent_upgrade
     done
     
@@ -2629,7 +2668,25 @@ SYSTEM_INFO_EOF
         echo "|"
         return 0
     }
-    
+
+    # Issue #27 — keepalived/VRRP is Linux-only (DAEMON version). Safe no-op that reports a
+    # clear "unsupported on macOS" status if a VIP is ever mis-assigned to a macOS node.
+    fetch_and_deploy_keepalived_config() {
+        local resp status vip_id
+        resp=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-config" \
+            -H "X-API-Key: $AGENT_TOKEN" 2>/dev/null) || return 0
+        [[ -z "$resp" ]] && return 0
+        status=$(echo "$resp" | jq -r '.status // "unknown"' 2>/dev/null)
+        if [[ "$status" == "available" ]]; then
+            vip_id=$(echo "$resp" | jq -r '.keepalived.vip_id // empty' 2>/dev/null)
+            log "WARN" "KEEPALIVED: a VIP is assigned to this node, but keepalived/VRRP is not supported on macOS — ignoring"
+            curl -k -s -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-status" \
+                -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
+                -d "{\"vip_id\":${vip_id:-null},\"state\":\"error\",\"message\":\"keepalived/VRRP not supported on macOS\"}" >/dev/null 2>&1 || true
+        fi
+        return 0
+    }
+
     # Send heartbeat (DAEMON version - includes all stats)
     send_heartbeat() {
         local haproxy_status=$(get_haproxy_status)
@@ -3021,6 +3078,13 @@ CONFIG_RESPONSE_EOF
             fi
         fi
         
+        # Issue #27 — HA/VIP convergence at the SSL cadence (no-op on macOS; reports
+        # unsupported if a VIP is mis-assigned to this node).
+        _kp_loop_count=$(( ${_kp_loop_count:-0} + 1 ))
+        if (( _kp_loop_count % 5 == 0 )) && type fetch_and_deploy_keepalived_config &>/dev/null; then
+            fetch_and_deploy_keepalived_config
+        fi
+
         # Check for configuration updates with proper error handling
         config_response=$(curl -k -s -X GET "$MANAGEMENT_URL/api/agents/$AGENT_NAME/config" \
             -H "X-API-Key: $CURRENT_AGENT_TOKEN" 2>/dev/null)

backend/utils/dns_credentials.py

@@ -0,0 +1,88 @@
+"""Issue #35 — ACME DNS-01 (v1.8.0): at-rest encryption for per-account DNS provider credentials.
+
+Mirrors the established Fernet + HKDF(SECRET_KEY) pattern used for the VRRP secret
+(backend/services/keepalived_config.py) and TOTP secrets (backend/services/mfa_service.py):
+prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY env var (enables key rotation), else derive a
+stable key from SECRET_KEY via HKDF with a versioned info string.
+
+DNS provider credentials are a small dict (e.g. {"api_token": "..."}). They are JSON-serialized,
+encrypted to a Fernet token string for storage, and only ever decrypted in-process when a DNS-01
+order needs to talk to the provider. Plaintext credentials are NEVER logged or returned by the API.
+
+NOTE on key rotation: if SECRET_KEY rotates and DNS_PROVIDER_ENCRYPTION_KEY is not set, previously
+stored credentials become undecryptable (decrypt returns None). Callers MUST treat a None result as
+"credentials unavailable — re-enter in Settings" and surface a clear error, never a silent hang.
+"""
+from __future__ import annotations
+
+import base64
+import json
+import logging
+import os
+from typing import Dict, Optional
+
+from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+
+from config import SECRET_KEY
+
+logger = logging.getLogger(__name__)
+
+_fernet_instance: Optional[Fernet] = None
+
+
+def _resolve_fernet_key() -> bytes:
+    """Prefer an explicit DNS_PROVIDER_ENCRYPTION_KEY; else derive from SECRET_KEY via HKDF
+    with a versioned info string (so credentials survive restarts)."""
+    explicit = os.getenv("DNS_PROVIDER_ENCRYPTION_KEY", "").strip()
+    if explicit:
+        try:
+            Fernet(explicit.encode())
+            return explicit.encode()
+        except Exception as exc:  # noqa: BLE001
+            logger.error("DNS_PROVIDER_ENCRYPTION_KEY env var present but invalid: %s", exc)
+    logger.warning(
+        "DNS_PROVIDER_ENCRYPTION_KEY not set; deriving the DNS-credentials encryption key from "
+        "SECRET_KEY. Set DNS_PROVIDER_ENCRYPTION_KEY to a Fernet key to enable key rotation."
+    )
+    hkdf = HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=b"dns-provider-creds-v1")
+    derived = hkdf.derive(SECRET_KEY.encode("utf-8"))
+    return base64.urlsafe_b64encode(derived)
+
+
+def _get_fernet() -> Fernet:
+    global _fernet_instance
+    if _fernet_instance is None:
+        _fernet_instance = Fernet(_resolve_fernet_key())
+    return _fernet_instance
+
+
+def reset_fernet_for_tests() -> None:
+    """Test-only hook to force re-resolution after env mutation."""
+    global _fernet_instance
+    _fernet_instance = None
+
+
+def encrypt_dns_credentials(credentials: Dict[str, str]) -> str:
+    """JSON-serialize and Fernet-encrypt a credentials dict to a storable token string."""
+    payload = json.dumps(credentials, separators=(",", ":")).encode("utf-8")
+    return _get_fernet().encrypt(payload).decode("utf-8")
+
+
+def decrypt_dns_credentials(token: str) -> Optional[Dict[str, str]]:
+    """Decrypt a stored token back to the credentials dict. Returns None if the token can't be
+    decrypted (e.g. key rotated) — callers must surface a clear 're-enter credentials' error."""
+    try:
+        plain = _get_fernet().decrypt(token.encode("utf-8")).decode("utf-8")
+        data = json.loads(plain)
+        if not isinstance(data, dict):
+            logger.error("Decrypted DNS credentials are not a JSON object")
+            return None
+        return data
+    except InvalidToken:
+        logger.warning("Failed to decrypt DNS provider credentials (invalid Fernet token)")
+        return None
+    except Exception as exc:  # noqa: BLE001
+        logger.error("Unexpected error decrypting DNS provider credentials: %s", exc)
+        return None

docker-compose.yml

@@ -71,15 +71,17 @@ services:
 
   # Frontend React App - pulls from Docker Hub by default
   # To build locally instead, uncomment the build section and run: docker compose build frontend
+  #
+  # NOTE: REACT_APP_* env vars are BUILD-time only for Create-React-App. The
+  # runtime container (serve -s build) does NOT consume them. The frontend
+  # uses same-origin (window.location) for /api/* and is routed by the nginx
+  # service below to the backend container. No env vars are required here.
   frontend:
     image: taylanbakircioglu/haproxy-openmanager-frontend:latest
     # build:
     #   context: ./frontend
     #   dockerfile: Dockerfile
     container_name: haproxy-openmanager-frontend
-    environment:
-      - REACT_APP_API_URL=
-      - NODE_ENV=production
     expose:
       - "3000"
     depends_on:
@@ -94,7 +96,9 @@ services:
 
   # Nginx Reverse Proxy
   nginx:
-    image: nginx:alpine
+    # Pinned to a patched release for the nginx "poolslip" advisory
+    # (mainline <=1.31.0 affected; fixed in mainline 1.31.1+ / stable 1.30.2+).
+    image: nginx:1.31.1-alpine
     container_name: haproxy-openmanager-nginx
     volumes:
       - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro

docs/STANDALONE_INSTALL.md

@@ -163,6 +163,11 @@ serve -s build -l 3000
 
 Option B — copy to nginx (recommended, see next section).
 
+> **Important: use the nginx URL on port 8080 as your entry point.** The `serve` option above (port 3000) hosts
+> only the static UI; there is no `/api` backend behind it, so the login page renders but cannot actually log you
+> in. nginx (next section) serves the UI *and* proxies `/api` to the backend on a single port, so do your login and
+> everyday use at `http://<server-ip>:8080`.
+
 ### Create a systemd service for the frontend (if using `serve`)
 
 ```bash
@@ -268,10 +273,10 @@ curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8080/
 # Login
 curl -s -X POST http://127.0.0.1:8080/api/auth/login \
     -H "Content-Type: application/json" \
-    -d '{"username": "admin", "password": "admin"}' | python3 -m json.tool
+    -d '{"username": "admin", "password": "admin123"}' | python3 -m json.tool
 ```
 
-> **Default credentials**: `admin` / `admin` — change the password immediately after first login.
+> **Default credentials**: `admin` / `admin123` — change the password immediately after first login.
 
 ## 9. Firewall
 
@@ -290,8 +295,8 @@ sudo ufw allow 8080/tcp
 | Service | Port | URL |
 |---------|------|-----|
 | Backend API | 8000 | `http://127.0.0.1:8000/api/health` |
-| Frontend | 3000 | `http://127.0.0.1:3000` |
-| Nginx (unified) | 8080 | `http://your-server-ip:8080` |
+| Frontend (static only) | 3000 | `http://127.0.0.1:3000` (UI only, no API; not the login URL) |
+| Nginx (unified, entry point) | 8080 | `http://your-server-ip:8080` (use this) |
 | PostgreSQL | 5432 | local |
 | Redis | 6379 | local |
 

frontend/.dockerignore

@@ -0,0 +1,34 @@
+# Build artifacts (regenerated inside the multi-stage builder)
+build
+node_modules
+coverage
+
+# Local dev-only overrides — MUST be excluded so the host's
+# `.env.local` (e.g. REACT_APP_API_URL=http://localhost:8000) does NOT
+# bleed into a production bundle via build-time inline-replace.
+.env
+.env.local
+.env.*.local
+.env.development
+.env.development.local
+.env.test
+.env.test.local
+
+# VCS / IDE / OS noise
+.git
+.gitignore
+.vscode
+.idea
+.DS_Store
+*.log
+
+# Tests
+**/__tests__
+**/*.test.js
+**/*.test.jsx
+**/*.test.ts
+**/*.test.tsx
+
+# Misc
+README.md
+.eslintcache

frontend/package-lock.json

@@ -1,20 +1,22 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.5.0",
+  "version": "1.7.8",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "haproxy-openmanager-frontend",
-      "version": "1.5.0",
+      "version": "1.7.8",
+      "license": "AGPL-3.0-or-later",
       "dependencies": {
         "@ant-design/icons": "^5.0.0",
         "@monaco-editor/react": "^4.6.0",
         "ace-builds": "^1.23.4",
         "antd": "^5.2.0",
-        "axios": "^1.3.0",
+        "axios": "^1.16.0",
         "moment": "^2.29.0",
         "monaco-editor": "^0.36.0",
+        "qrcode.react": "^4.0.0",
         "react": "^18.2.0",
         "react-ace": "^10.1.0",
         "react-dom": "^18.2.0",
@@ -1500,9 +1502,9 @@
       }
     },
     "node_modules/@babel/plugin-transform-modules-systemjs": {
-      "version": "7.29.0",
-      "resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.0.tgz",
-      "integrity": "sha512-PrujnVFbOdUpw4UHiVwKvKRLMMic8+eC0CuNlxjsyZUiBjhFdPsewdXCkveh2KqBA9/waD0W1b4hXSOBQJezpQ==",
+      "version": "7.29.4",
+      "resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.4.tgz",
+      "integrity": "sha512-N7QmZ0xRZfjHOfZeQLJjwgX2zS9pdGHSVl/cjSGlo4dXMqvurfxXDMKY4RqEKzPozV78VMcd0lxyG13mlbKc4w==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -4652,27 +4654,6 @@
         "url": "https://github.com/sponsors/gregberge"
       }
     },
-    "node_modules/@testing-library/dom": {
-      "version": "10.4.1",
-      "resolved": "https://registry.npmjs.org/@testing-library/dom/-/dom-10.4.1.tgz",
-      "integrity": "sha512-o4PXJQidqJl82ckFaXUeoAW+XysPLauYI43Abki5hABd853iMhitooc6znOnczgbTYmEP6U6/y1ZyKAIsvMKGg==",
-      "dev": true,
-      "license": "MIT",
-      "peer": true,
-      "dependencies": {
-        "@babel/code-frame": "^7.10.4",
-        "@babel/runtime": "^7.12.5",
-        "@types/aria-query": "^5.0.1",
-        "aria-query": "5.3.0",
-        "dom-accessibility-api": "^0.5.9",
-        "lz-string": "^1.5.0",
-        "picocolors": "1.1.1",
-        "pretty-format": "^27.0.2"
-      },
-      "engines": {
-        "node": ">=18"
-      }
-    },
     "node_modules/@testing-library/jest-dom": {
       "version": "5.17.0",
       "resolved": "https://registry.npmjs.org/@testing-library/jest-dom/-/jest-dom-5.17.0.tgz",
@@ -6543,12 +6524,12 @@
       }
     },
     "node_modules/axios": {
-      "version": "1.14.0",
-      "resolved": "https://registry.npmjs.org/axios/-/axios-1.14.0.tgz",
-      "integrity": "sha512-3Y8yrqLSwjuzpXuZ0oIYZ/XGgLwUIBU3uLvbcpb0pidD9ctpShJd43KSlEEkVQg6DS0G9NKyzOvBfUtDKEyHvQ==",
+      "version": "1.16.0",
+      "resolved": "https://registry.npmjs.org/axios/-/axios-1.16.0.tgz",
+      "integrity": "sha512-6hp5CwvTPlN2A31g5dxnwAX0orzM7pmCRDLnZSX772mv8WDqICwFjowHuPs04Mc8deIld1+ejhtaMn5vp6b+1w==",
       "license": "MIT",
       "dependencies": {
-        "follow-redirects": "^1.15.11",
+        "follow-redirects": "^1.16.0",
         "form-data": "^4.0.5",
         "proxy-from-env": "^2.1.0"
       }
@@ -6922,9 +6903,9 @@
       "license": "MIT"
     },
     "node_modules/body-parser": {
-      "version": "1.20.4",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz",
-      "integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
+      "version": "1.20.5",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.5.tgz",
+      "integrity": "sha512-3grm+/2tUOvu2cjJkvsIxrv/wVpfXQW4PsQHYm7yk4vfpu7Ekl6nEsYBoJUL6qDwZUx8wUhQ8tR2qz+ad9c9OA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -6936,7 +6917,7 @@
         "http-errors": "~2.0.1",
         "iconv-lite": "~0.4.24",
         "on-finished": "~2.4.1",
-        "qs": "~6.14.0",
+        "qs": "~6.15.1",
         "raw-body": "~2.5.3",
         "type-is": "~1.6.18",
         "unpipe": "~1.0.0"
@@ -10021,15 +10002,15 @@
       }
     },
     "node_modules/express": {
-      "version": "4.22.1",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.22.1.tgz",
-      "integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
+      "version": "4.22.2",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
+      "integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "accepts": "~1.3.8",
         "array-flatten": "1.1.1",
-        "body-parser": "~1.20.3",
+        "body-parser": "~1.20.5",
         "content-disposition": "~0.5.4",
         "content-type": "~1.0.4",
         "cookie": "~0.7.1",
@@ -10048,7 +10029,7 @@
         "parseurl": "~1.3.3",
         "path-to-regexp": "~0.1.12",
         "proxy-addr": "~2.0.7",
-        "qs": "~6.14.0",
+        "qs": "~6.15.1",
         "range-parser": "~1.2.1",
         "safe-buffer": "5.2.1",
         "send": "~0.19.0",
@@ -10145,9 +10126,9 @@
       "license": "MIT"
     },
     "node_modules/fast-uri": {
-      "version": "3.1.0",
-      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.0.tgz",
-      "integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==",
+      "version": "3.1.2",
+      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz",
+      "integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==",
       "dev": true,
       "funding": [
         {
@@ -10412,9 +10393,9 @@
       "license": "ISC"
     },
     "node_modules/follow-redirects": {
-      "version": "1.15.11",
-      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.11.tgz",
-      "integrity": "sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ==",
+      "version": "1.16.0",
+      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz",
+      "integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==",
       "funding": [
         {
           "type": "individual",
@@ -18237,10 +18218,19 @@
         "teleport": ">=0.2.0"
       }
     },
+    "node_modules/qrcode.react": {
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/qrcode.react/-/qrcode.react-4.2.0.tgz",
+      "integrity": "sha512-QpgqWi8rD9DsS9EP3z7BT+5lY5SFhsqGjpgW5DY/i3mK4M9DTBNz3ErMi8BWYEfI3L0d8GIbGmcdFAS1uIRGjA==",
+      "license": "ISC",
+      "peerDependencies": {
+        "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
+      }
+    },
     "node_modules/qs": {
-      "version": "6.14.2",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz",
-      "integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
+      "version": "6.15.2",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.2.tgz",
+      "integrity": "sha512-Rzq0KEyX/w/tEybncDgdkZrJgVUsUMk3xjh3t5bv3S1HTAtg+uOYt72+ZfwiQwKdysThkTBdL/rTi6HDmX9Ddw==",
       "dev": true,
       "license": "BSD-3-Clause",
       "dependencies": {
@@ -20357,9 +20347,9 @@
       }
     },
     "node_modules/shell-quote": {
-      "version": "1.8.3",
-      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz",
-      "integrity": "sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw==",
+      "version": "1.8.4",
+      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.4.tgz",
+      "integrity": "sha512-VsC6n6vz1ihYYyZZwX7YZSF5l5x36ca17OC+a69h94YqB7X6XLwf+5MOgynYir2SLFUbl8gIYvBo8K8RoNQ6bQ==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -21325,24 +21315,6 @@
         }
       }
     },
-    "node_modules/tailwindcss/node_modules/yaml": {
-      "version": "2.8.3",
-      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.3.tgz",
-      "integrity": "sha512-AvbaCLOO2Otw/lW5bmh9d/WEdcDFdQp2Z2ZUH3pX9U2ihyUY0nvLv7J6TrWowklRGPYbB/IuIMfYgxaCPg5Bpg==",
-      "dev": true,
-      "license": "ISC",
-      "optional": true,
-      "peer": true,
-      "bin": {
-        "yaml": "bin.mjs"
-      },
-      "engines": {
-        "node": ">= 14.6"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/eemeli"
-      }
-    },
     "node_modules/tapable": {
       "version": "2.3.2",
       "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.2.tgz",
@@ -22207,7 +22179,7 @@
       }
     },
     "node_modules/wbuf": {
-      "version": "1.7.3",
+      "version": "1.7.8",
       "resolved": "https://registry.npmjs.org/wbuf/-/wbuf-1.7.3.tgz",
       "integrity": "sha512-O84QOnr0icsbFGLS0O3bI5FswxzRr8/gHwWkDlQFskhSPryQXvrTMxjxGP4+iWYoauLoBvfDpkrOauZ+0iZpDA==",
       "dev": true,

frontend/package.json

@@ -1,13 +1,13 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.5.2",
+  "version": "1.8.5",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-router-dom": "^6.8.0",
-    "axios": "^1.3.0",
+    "axios": "^1.16.0",
     "antd": "^5.2.0",
     "@ant-design/icons": "^5.0.0",
     "recharts": "^2.5.0",
@@ -17,7 +17,8 @@
     "@monaco-editor/react": "^4.6.0",
     "react-ace": "^10.1.0",
     "ace-builds": "^1.23.4",
-    "react-window": "^1.8.10"
+    "react-window": "^1.8.10",
+    "qrcode.react": "^4.0.0"
   },
   "devDependencies": {
     "@types/react": "^18.0.0",

frontend/src/App.js

@@ -5,6 +5,7 @@ import {
   DashboardOutlined,
   SettingOutlined,
   CloudServerOutlined,
+  DeploymentUnitOutlined,
   FileTextOutlined,
   GlobalOutlined,
   SafetyCertificateOutlined,
@@ -42,6 +43,7 @@ import PoolManagement from './components/PoolManagement';
 import Login from './components/Login';
 import ClusterSelector from './components/ClusterSelector';
 import ClusterManagement from './components/ClusterManagement';
+import VIPManagement from './components/VIPManagement';
 import Configuration from './components/Configuration';
 import APIDocumentation from './components/APIDocumentation';
 import IPInventory from './components/IPInventory';
@@ -156,6 +158,11 @@ const { Text } = Typography;
       icon: <CloudServerOutlined />,
       label: <Link to="/clusters">Clusters</Link>,
     },
+    {
+      key: '/ha-vip',
+      icon: <DeploymentUnitOutlined />,
+      label: <Link to="/ha-vip">HA / VIP</Link>,
+    },
     {
       key: '/pools',
       icon: <GlobalOutlined />,
@@ -466,6 +473,7 @@ function AppContent() {
               <Route path="/pools" element={<PoolManagement />} />
               <Route path="/settings" element={<Settings />} />
               <Route path="/clusters" element={<ClusterManagement />} />
+              <Route path="/ha-vip" element={<VIPManagement />} />
               <Route path="/ip-inventory" element={<IPInventory />} />
             </Routes>
           </Content>

frontend/src/components/ACMEAutomation.js

@@ -1,8 +1,8 @@
 import React, { useState, useEffect, useCallback, useRef } from 'react';
 import {
   Card, Table, Button, Tag, Space, Modal, Form, Input, Select, Steps,
-  message, Row, Col, Statistic, Alert, Tooltip, Switch, theme, Segmented,
-  Tabs, Timeline, Spin, Empty
+  message, Row, Col, Statistic, Alert, Tooltip, Switch, theme, Segmented, Collapse,
+  Tabs, Timeline, Spin, Empty, Typography, Divider
 } from 'antd';
 import {
   SafetyCertificateOutlined, PlusOutlined, ReloadOutlined,
@@ -10,7 +10,7 @@ import {
   SyncOutlined, CloseCircleOutlined,
   DeleteOutlined, EyeOutlined,
   CloudDownloadOutlined, UserOutlined, InfoCircleOutlined,
-  RocketOutlined, ExperimentOutlined
+  RocketOutlined, ExperimentOutlined, KeyOutlined
 } from '@ant-design/icons';
 import { useNavigate } from 'react-router-dom';
 import { useCluster } from '../contexts/ClusterContext';
@@ -18,8 +18,26 @@ import axios from 'axios';
 
 const { Option } = Select;
 
-const getErrorMsg = (err, fallback) =>
-  err?.response?.data?.error?.message || err?.response?.data?.detail || fallback;
+const getErrorMsg = (err, fallback) => {
+  const data = err?.response?.data;
+  // FastAPI/Pydantic 422s wrap the specific field message in error.details.validation_errors[];
+  // the top-level error.message is generic ("Validation error in request data"), so prefer the
+  // field-level message (e.g. the EAB base64 hint) when present.
+  const fieldMsg = data?.error?.details?.validation_errors?.[0]?.message;
+  return fieldMsg || data?.error?.message || data?.detail || fallback;
+};
+
+// Issue #35: humanize the dotted event_type tokens emitted for DNS-01 orders so the diagnostics
+// timeline reads as a step-by-step progress log rather than raw machine strings. Unknown types
+// fall back to the raw token.
+const EVENT_LABELS = {
+  'acme.dns01.publish': 'Published DNS TXT record',
+  'acme.dns01.responded': 'Asked the CA to validate',
+  'acme.dns01.validation': 'DNS-01 validation result',
+  'acme.dns01.cleanup': 'Removed DNS TXT record',
+  'acme.order.requested': 'Certificate requested',
+};
+const humanizeEventType = (t) => EVENT_LABELS[t] || t;
 
 // Render letsencrypt_orders.error_detail (TEXT column). Backend now writes
 // structured JSON-strings (stage / http_status / ca_response / timestamp) for
@@ -83,6 +101,25 @@ const ACMEAutomation = () => {
   const [orderFilter, setOrderFilter] = useState('active');
   const { token } = theme.useToken();
 
+  // Issue #35: DNS-01 provider catalog + global enable flag (from GET /dns-providers).
+  const [dnsProviders, setDnsProviders] = useState([]);
+  const [dns01Enabled, setDns01Enabled] = useState(false);
+  const [confirming, setConfirming] = useState(false);
+  const regChallengeType = Form.useWatch('challenge_type', registerForm);
+  const regDnsProvider = Form.useWatch('dns_provider', registerForm);
+  const wizardAccountId = Form.useWatch('account_id', wizardForm);
+  const wizardDomains = Form.useWatch('domains', wizardForm);
+  const selectedDnsProvider = dnsProviders.find(p => p.name === regDnsProvider) || null;
+
+  // Issue #35: per-account DNS credential management (view/replace/clear after creation).
+  const [credModalVisible, setCredModalVisible] = useState(false);
+  const [credAccount, setCredAccount] = useState(null);
+  const [credMeta, setCredMeta] = useState(null);
+  const [credLoading, setCredLoading] = useState(false);
+  const [credSaving, setCredSaving] = useState(false);
+  const [credForm] = Form.useForm();
+  const credProvider = credAccount ? (dnsProviders.find(p => p.name === credAccount.dns_provider) || null) : null;
+
   // v1.5.0 Issue #13: ACME Diagnostic Panel state
   const [diagVisible, setDiagVisible] = useState(false);
   const [diagOrderId, setDiagOrderId] = useState(null);
@@ -105,18 +142,23 @@ const ACMEAutomation = () => {
   const fetchData = useCallback(async () => {
     setLoading(true);
     try {
-      const [ordersRes, accountsRes, renewalRes, clustersRes, prereqRes] = await Promise.allSettled([
+      const [ordersRes, accountsRes, renewalRes, clustersRes, prereqRes, dnsRes] = await Promise.allSettled([
         axios.get('/api/letsencrypt/orders'),
         axios.get('/api/letsencrypt/accounts'),
         axios.get('/api/letsencrypt/renewal-schedule'),
         axios.get('/api/clusters'),
         axios.get('/api/letsencrypt/prerequisites'),
+        axios.get('/api/letsencrypt/dns-providers'),
       ]);
       if (ordersRes.status === 'fulfilled') setOrders(ordersRes.value.data || []);
       if (accountsRes.status === 'fulfilled') setAccounts(accountsRes.value.data || []);
       if (prereqRes.status === 'fulfilled') setPrerequisites(prereqRes.value.data);
       if (renewalRes.status === 'fulfilled') setRenewalSchedule(renewalRes.value.data || []);
       if (clustersRes.status === 'fulfilled') setClusters(clustersRes.value.data?.clusters || []);
+      if (dnsRes.status === 'fulfilled') {
+        setDnsProviders(dnsRes.value.data?.providers || []);
+        setDns01Enabled(!!dnsRes.value.data?.dns01_enabled);
+      }
     } catch (err) {
       console.error('Error loading ACME data:', err);
     } finally {
@@ -133,9 +175,29 @@ const ACMEAutomation = () => {
     o.status === 'pending' || o.status === 'processing' || o.status === 'ready' ||
     (o.status === 'valid' && !o.ssl_certificate_id)
   );
+  // Track the order whose detail modal is open so the poll can refresh it WITHOUT
+  // making the interval depend on orderDetail (which would recreate it every poll).
+  const openDetailIdRef = useRef(null);
+  const detailRefetchInFlightRef = useRef(false);
+  useEffect(() => {
+    openDetailIdRef.current = (detailVisible && orderDetail?.id) ? orderDetail.id : null;
+  }, [detailVisible, orderDetail]);
   useEffect(() => {
     if (!hasInProgress) return undefined;
-    const interval = setInterval(() => { fetchData(); }, 30000);
+    const interval = setInterval(() => {
+      fetchData();
+      // Keep an open order-detail modal (e.g. a manual DNS-01 order awaiting validation)
+      // in sync so its status / TXT block / Verify button cannot go stale. Skip if a prior
+      // refetch is still in flight so slow backends don't pile up overlapping requests.
+      const oid = openDetailIdRef.current;
+      if (oid && !detailRefetchInFlightRef.current) {
+        detailRefetchInFlightRef.current = true;
+        axios.get(`/api/letsencrypt/orders/${oid}`)
+          .then((r) => setOrderDetail((prev) => (prev && prev.id === oid ? r.data : prev)))
+          .catch(() => { /* transient; the next poll retries */ })
+          .finally(() => { detailRefetchInFlightRef.current = false; });
+      }
+    }, 30000);
     return () => clearInterval(interval);
   }, [hasInProgress, fetchData]);
 
@@ -144,7 +206,10 @@ const ACMEAutomation = () => {
     if (!expiryDate) return null;
     return Math.ceil((new Date(expiryDate) - new Date()) / (1000 * 60 * 60 * 24));
   };
-  const nextRenewal = renewalSchedule.find(c => c.auto_renew && calcDaysLeft(c.expiry_date) > 0);
+  // Manual DNS-01 certs are not auto-renewed (even a legacy row left at auto_renew=TRUE), so they
+  // must not drive the "Next Renewal" countdown, which implies an automated event.
+  const isManualDnsCert = (c) => c.challenge_type === 'dns-01' && (c.dns_provider || 'manual') === 'manual';
+  const nextRenewal = renewalSchedule.find(c => c.auto_renew && !isManualDnsCert(c) && calcDaysLeft(c.expiry_date) > 0);
   const nextRenewalDays = nextRenewal ? calcDaysLeft(nextRenewal.expiry_date) : null;
   // Issue #11/#12: an order is "in progress" if it's pre-valid OR valid-but-not-downloaded (stuck).
   // Including 'ready' here ensures the dashboard counter & UI auto-refresh react to all in-flight states.
@@ -155,6 +220,21 @@ const ACMEAutomation = () => {
   const activeAccount = accounts.find(a => a.status === 'valid') || null;
   const acmeAccount = activeAccount || (accounts.length > 0 ? accounts[accounts.length - 1] : null);
   const acmeEnabledClusters = clusters.filter(c => c.acme_enabled && c.is_active);
+  // Issue #35: the cert wizard adapts to the selected account's challenge method.
+  const wizardAccount = accounts.find(a => a.id === wizardAccountId) || activeAccount || acmeAccount;
+  const wizardIsDns01 = (wizardAccount?.challenge_type === 'dns-01');
+  const wizardDnsManual = wizardIsDns01 && (wizardAccount?.dns_provider === 'manual');
+  // Wildcard certificates can only be issued over DNS-01. Catch this client-side so the user is
+  // told their mistake up front instead of waiting for a CA-side rejection.
+  const wizardHasWildcard = (wizardDomains || []).some(d => typeof d === 'string' && d.trim().startsWith('*.'));
+  const wizardWildcardBlocked = wizardHasWildcard && !wizardIsDns01;
+  // A DNS-01 account can exist while the global kill-switch is off (e.g. an admin disabled it later).
+  // Issuing would be rejected by the backend, so block it in the wizard with a clear reason.
+  const wizardDns01Disabled = wizardIsDns01 && !dns01Enabled;
+  // Surface a direct credentials shortcut on the dashboard card when the primary account uses an
+  // automated DNS provider (manual providers need no credentials).
+  const acmeAccountProvider = acmeAccount ? dnsProviders.find(p => p.name === acmeAccount.dns_provider) : null;
+  const acmeAccountNeedsCreds = acmeAccount?.challenge_type === 'dns-01' && (acmeAccountProvider?.credential_fields || []).length > 0;
 
   const filteredOrders = orders.filter(o => {
     if (orderFilter === 'active') return !['cancelled', 'invalid', 'valid'].includes(o.status);
@@ -170,12 +250,29 @@ const ACMEAutomation = () => {
         message.error('At least one domain is required');
         return;
       }
+      // Defense-in-depth: the Submit button is already disabled for these, but guard here too.
+      if (wizardWildcardBlocked) {
+        message.error('Wildcard certificates require a DNS-01 account. Select a DNS-01 account or remove the wildcard domain.');
+        return;
+      }
+      if (wizardDns01Disabled) {
+        message.error('DNS-01 is disabled by an administrator. Enable it in Settings > ACME to issue this certificate.');
+        return;
+      }
       setSubmitting(true);
+      // Resolve the chosen account's challenge method so DNS-01/wildcard requests are explicit.
+      // Use the same resolution as the wizard description (wizardAccount) so what the user reviewed
+      // matches what is sent.
+      const challengeType = wizardAccount?.challenge_type;  // 'http-01' | 'dns-01' | undefined
+      // Manual DNS-01 can't auto-renew (the wizard shows the switch off+disabled). Send false to
+      // match the displayed state rather than relying only on the backend to override it.
+      const autoRenew = wizardDnsManual ? false : (values.auto_renew !== false);
       const res = await axios.post('/api/letsencrypt/certificates', {
         domains: values.domains,
         cluster_ids: values.cluster_ids || [],
-        auto_renew: values.auto_renew !== false,
+        auto_renew: autoRenew,
         account_id: values.account_id || null,
+        challenge_type: challengeType || undefined,
       });
       message.success(res.data?.message || 'Certificate request submitted');
       if (res.data?.warnings?.length > 0) {
@@ -185,6 +282,16 @@ const ACMEAutomation = () => {
       setWizardStep(0);
       wizardForm.resetFields();
       fetchData();
+      // For a manual DNS-01 order the user must publish the TXT record(s) next, so open the
+      // order detail straight away instead of leaving them to hunt for it.
+      if (res.data?.challenge_type === 'dns-01'
+          && (res.data?.dns_provider || 'manual') === 'manual'
+          && res.data?.order_id) {
+        handleViewOrder(res.data.order_id);
+      } else if (res.data?.challenge_type === 'dns-01') {
+        // Automated DNS-01 (e.g. Cloudflare): reassure the user it is hands-off.
+        message.info('Automated DNS-01: the TXT records will be published and validated automatically. No action needed.', 6);
+      }
     } catch (err) {
       message.error(getErrorMsg(err, 'Failed to request certificate'));
     } finally {
@@ -234,7 +341,7 @@ const ACMEAutomation = () => {
       setOrderDetail(res.data);
       setDetailVisible(true);
     } catch (err) {
-      message.error('Failed to load order details');
+      message.error(getErrorMsg(err, 'Failed to load order details'));
     }
   };
 
@@ -453,21 +560,144 @@ const ACMEAutomation = () => {
     try {
       const values = await registerForm.validateFields();
       setRegistering(true);
+      const challengeType = dns01Enabled ? (values.challenge_type || 'http-01') : 'http-01';
+      const dnsProvider = challengeType === 'dns-01' ? (values.dns_provider || null) : null;
       const res = await axios.post('/api/letsencrypt/accounts', {
         email: values.email,
         tos_agreed: values.tos_agreed,
+        challenge_type: challengeType,
+        dns_provider: dnsProvider,
+        // EAB for CAs that require it (ZeroSSL/Google). Empty → backend falls back to global Settings.
+        eab_kid: (values.eab_kid || '').trim() || undefined,
+        eab_hmac_key: (values.eab_hmac_key || '').trim() || undefined,
       });
-      message.success(`ACME account registered: ${res.data?.email || values.email}`);
-      setRegisterVisible(false);
-      registerForm.resetFields();
+      const accountId = res.data?.id;
+      // For an automated DNS-01 provider, store the entered credentials (verified server-side).
+      const provider = dnsProviders.find(p => p.name === dnsProvider);
+      let credFailed = false;
+      if (challengeType === 'dns-01' && accountId && provider && (provider.credential_fields || []).length > 0) {
+        const creds = {};
+        (provider.credential_fields || []).forEach(f => {
+          const v = values[`cred_${f.key}`];
+          if (v != null && v !== '') creds[f.key] = v;
+        });
+        try {
+          const r2 = await axios.put(`/api/letsencrypt/accounts/${accountId}/dns-credentials`, {
+            dns_provider: dnsProvider,
+            credentials: creds,
+          });
+          message.success(r2.data?.detail || 'DNS provider credentials saved');
+        } catch (credErr) {
+          credFailed = true;
+          message.warning(getErrorMsg(credErr, `Account "${res.data?.email || values.email}" registered, but the DNS credentials could not be saved. You can fix them from the account's DNS credentials action.`), 8);
+        }
+      }
+      // Only close + reset on full success. On a credential-save failure, keep the modal open with
+      // the entered values so the user can correct the token and re-submit (the account already
+      // exists and the PUT re-verifies) — avoids discarding input, a dead-end, and a misleading
+      // success toast (Finding 7). The warning toast above explains what to fix.
+      if (!credFailed) {
+        message.success(`ACME account registered: ${res.data?.email || values.email}`);
+        setRegisterVisible(false);
+        registerForm.resetFields();
+      }
       fetchData();
     } catch (err) {
+      // Inline field-validation rejections already render under each field; don't also
+      // fire a generic error toast (mirrors handleSaveDnsCreds).
+      if (err?.errorFields) return;
       message.error(getErrorMsg(err, 'Account registration failed'));
     } finally {
       setRegistering(false);
     }
   };
 
+  // Issue #35: manual DNS-01 — user asserts the TXT records are published; tell the CA to validate.
+  const handleDnsConfirm = async (orderId) => {
+    if (confirming) return;  // guard the leading-edge double-click (loading alone does not block a synchronous re-fire)
+    try {
+      setConfirming(true);
+      const res = await axios.post(`/api/letsencrypt/orders/${orderId}/dns-confirm`);
+      message.success(res.data?.message || 'DNS-01 confirmation submitted; the CA will validate shortly.');
+      // Refresh the orders list AND the open detail modal so the user sees the new state
+      // (otherwise the modal shows a stale TXT block + an active Verify button).
+      fetchData();
+      try {
+        const fresh = await axios.get(`/api/letsencrypt/orders/${orderId}`);
+        // Only repopulate if the same order's detail is still open (the user may have
+        // closed it or navigated to another order while the request was in flight).
+        setOrderDetail((prev) => (prev && prev.id === orderId ? fresh.data : prev));
+      } catch (_e) { /* list refresh already happened; modal stays as-is */ }
+    } catch (err) {
+      message.error(getErrorMsg(err, 'Failed to confirm DNS-01'));
+    } finally {
+      setConfirming(false);
+    }
+  };
+
+  // Issue #35: view / replace / clear an account's DNS provider credentials after creation.
+  const openDnsCredsModal = async (account) => {
+    setCredAccount(account);
+    setCredMeta(null);
+    credForm.resetFields();
+    setCredModalVisible(true);
+    setCredLoading(true);
+    try {
+      const res = await axios.get(`/api/letsencrypt/accounts/${account.id}/dns-credentials`);
+      setCredMeta(res.data);
+    } catch (err) {
+      message.error(getErrorMsg(err, 'Failed to load DNS credentials'));
+    } finally {
+      setCredLoading(false);
+    }
+  };
+
+  const handleSaveDnsCreds = async () => {
+    if (!credAccount || !credProvider) return;
+    try {
+      const values = await credForm.validateFields();
+      setCredSaving(true);
+      const creds = {};
+      (credProvider.credential_fields || []).forEach(f => {
+        const v = values[`cred_${f.key}`];
+        if (v != null && v !== '') creds[f.key] = v;
+      });
+      const res = await axios.put(`/api/letsencrypt/accounts/${credAccount.id}/dns-credentials`, {
+        dns_provider: credAccount.dns_provider,
+        credentials: creds,
+      });
+      message.success(res.data?.detail || 'DNS provider credentials saved and verified');
+      setCredModalVisible(false);
+      credForm.resetFields();
+      fetchData();
+    } catch (err) {
+      if (err?.errorFields) return;  // antd validation errors shown inline
+      message.error(getErrorMsg(err, 'Failed to save DNS credentials'));
+    } finally {
+      setCredSaving(false);
+    }
+  };
+
+  const handleClearDnsCreds = () => {
+    if (!credAccount) return;
+    Modal.confirm({
+      title: 'Clear DNS credentials',
+      content: `Remove the stored DNS provider credentials for ${credAccount.email}? Automated DNS-01 issuance/renewal will stop working until you re-enter them.`,
+      okText: 'Clear',
+      okButtonProps: { danger: true },
+      onOk: async () => {
+        try {
+          await axios.delete(`/api/letsencrypt/accounts/${credAccount.id}/dns-credentials`);
+          message.success('DNS credentials cleared');
+          setCredModalVisible(false);
+          fetchData();
+        } catch (err) {
+          message.error(getErrorMsg(err, 'Failed to clear DNS credentials'));
+        }
+      },
+    });
+  };
+
   const handleDeactivateAccount = (accountId, email) => {
     Modal.confirm({
       title: 'Deactivate ACME Account',
@@ -564,7 +794,19 @@ const ACMEAutomation = () => {
       render: (status, record) => statusTag(status, record),
     },
     {
-      title: 'Account', dataIndex: 'account_email', key: 'account_email',
+      title: 'Method', key: 'method', width: 170,
+      render: (_, record) => {
+        const ct = record.challenge_type || 'http-01';
+        if (ct !== 'dns-01') return <Tag>HTTP-01</Tag>;
+        const prov = record.dns_provider || 'manual';
+        const needsAction = prov === 'manual' && (record.status === 'pending' || record.status === 'processing');
+        return needsAction
+          ? <Tag color="warning" icon={<ExclamationCircleOutlined />}>DNS-01 (manual): action needed</Tag>
+          : <Tag>DNS-01 ({prov})</Tag>;
+      },
+    },
+    {
+      title: 'Account', dataIndex: 'account_email', key: 'account_email', ellipsis: true,
       render: (e) => e || '-',
     },
     {
@@ -630,9 +872,27 @@ const ACMEAutomation = () => {
         return <Tag color={color}>{d} days</Tag>;
       },
     },
+    {
+      title: 'Method', key: 'method', width: 130,
+      render: (_, record) => {
+        const ct = record.challenge_type || 'http-01';
+        if (ct !== 'dns-01') return <Tag>HTTP-01</Tag>;
+        return <Tag>DNS-01 ({record.dns_provider || 'manual'})</Tag>;
+      },
+    },
     {
       title: 'Auto-Renew', dataIndex: 'auto_renew', key: 'auto_renew',
-      render: (v) => v ? <Tag color="green">Enabled</Tag> : <Tag>Disabled</Tag>,
+      render: (v, record) => {
+        const isManualDns = record.challenge_type === 'dns-01' && (record.dns_provider || 'manual') === 'manual';
+        if (isManualDns) {
+          return (
+            <Tooltip title="Manual DNS-01 cannot auto-renew unattended. Re-publish the TXT record and request renewal before expiry.">
+              <Tag color="warning" icon={<ExclamationCircleOutlined />}>Manual (re-publish TXT)</Tag>
+            </Tooltip>
+          );
+        }
+        return v ? <Tag color="green">Enabled</Tag> : <Tag>Disabled</Tag>;
+      },
     },
   ];
 
@@ -655,8 +915,19 @@ const ACMEAutomation = () => {
           <Alert
             type="info"
             showIcon
-            message="Each domain must resolve to an HAProxy node with ACME challenge routing enabled."
+            message={wizardIsDns01
+              ? "DNS-01: validated via a DNS TXT record, so no public port 80 is needed. Wildcards (*.example.com) are supported. Note that a wildcard does not cover the bare apex (example.com); add it as a separate domain if you need both."
+              : "Each domain must resolve to an HAProxy node with ACME challenge routing enabled (HTTP-01)."}
           />
+          {wizardWildcardBlocked && (
+            <Alert
+              type="warning"
+              showIcon
+              style={{ marginTop: 12 }}
+              message="Wildcard requires a DNS-01 account"
+              description="A wildcard domain (*.example.com) can only be validated over DNS-01. The currently selected account uses HTTP-01. Choose a DNS-01 account in the next step, or remove the wildcard domain."
+            />
+          )}
         </>
       ),
     },
@@ -668,20 +939,63 @@ const ACMEAutomation = () => {
             <Select mode="multiple" placeholder="Leave empty for global certificate" allowClear>
               {clusters.map(c => (
                 <Option key={c.id} value={c.id}>
-                  {c.name} {c.acme_enabled ? '' : '(ACME not enabled)'}
+                  {c.name} {wizardIsDns01 ? '' : (c.acme_enabled ? '' : '(ACME not enabled)')}
                 </Option>
               ))}
             </Select>
           </Form.Item>
-          <Form.Item name="auto_renew" label="Auto-Renew" valuePropName="checked" initialValue={true}>
-            <Switch defaultChecked />
-          </Form.Item>
+          {wizardIsDns01 && (
+            <Alert
+              type="info"
+              showIcon
+              style={{ marginBottom: 16 }}
+              message="DNS-01 needs no ACME Challenge Routing. Any active cluster works."
+              description={wizardDnsManual
+                ? "This account uses a manual DNS provider: after submitting, open the order and publish the shown TXT record, then confirm."
+                : "The DNS TXT record(s) will be published automatically."}
+            />
+          )}
+          {wizardDnsManual ? (
+            // Manual DNS-01 cannot auto-renew (the backend forces it off); show the control off and
+            // disabled so it matches the outcome rather than implying an automated renewal.
+            <Form.Item label="Auto-Renew">
+              <Switch checked={false} disabled />
+            </Form.Item>
+          ) : (
+            <Form.Item name="auto_renew" label="Auto-Renew" valuePropName="checked" initialValue={true}>
+              <Switch />
+            </Form.Item>
+          )}
+          {wizardDnsManual && (
+            <Alert
+              type="warning"
+              showIcon
+              style={{ marginTop: -8, marginBottom: 16 }}
+              message="Manual DNS-01 cannot auto-renew unattended. You will need to re-publish the TXT record at renewal time."
+            />
+          )}
           {accounts.length > 1 && (
-            <Form.Item name="account_id" label="ACME Account">
-              <Select placeholder="Use default account">
-                {accounts.map(a => (
-                  <Option key={a.id} value={a.id}>{a.email} ({a.directory_url})</Option>
-                ))}
+            <Form.Item
+              name="account_id"
+              label="ACME Account"
+              extra={<span style={{ fontSize: 12, color: token.colorTextSecondary }}>The validation method (HTTP-01 or DNS-01) is set by the chosen account. To use DNS-01, pick a DNS-01 account.</span>}
+            >
+              <Select placeholder="Use default account" optionLabelProp="label">
+                {accounts.map(a => {
+                  // Match the parenthesized "DNS-01 (provider)" form used in the tables and order detail.
+                  const methodLabel = a.challenge_type === 'dns-01'
+                    ? `DNS-01 (${a.dns_provider || 'manual'})`
+                    : 'HTTP-01';
+                  return (
+                    <Option key={a.id} value={a.id} label={`${a.email} · ${methodLabel}`}>
+                      <span>{a.email}{' '}
+                        <Typography.Text type="secondary" style={{ fontSize: 12 }}>
+                          {methodLabel} · {a.directory_url}
+                        </Typography.Text>
+                      </span>
+                    </Option>
+                  );
+                })}
               </Select>
             </Form.Item>
           )}
@@ -707,7 +1021,31 @@ const ACMEAutomation = () => {
               style={{ marginBottom: 16 }}
             />
           )}
-          {acmeEnabledClusters.length === 0 && (
+          {wizardDns01Disabled && (
+            <Alert
+              type="error"
+              showIcon
+              message="DNS-01 is disabled"
+              description={
+                <span>
+                  This account uses DNS-01, but DNS-01 is currently disabled by an administrator.{' '}
+                  <Button type="link" size="small" style={{ padding: 0 }} onClick={() => navigate('/settings?tab=acme')}>Enable it in Settings &gt; ACME</Button>
+                  {' '}to issue this certificate.
+                </span>
+              }
+              style={{ marginBottom: 16 }}
+            />
+          )}
+          {wizardWildcardBlocked && (
+            <Alert
+              type="error"
+              showIcon
+              message="Wildcard requires a DNS-01 account"
+              description="Remove the wildcard domain or select a DNS-01 account in the Configuration step."
+              style={{ marginBottom: 16 }}
+            />
+          )}
+          {!wizardIsDns01 && acmeEnabledClusters.length === 0 && (
             <Alert
               type="warning"
               showIcon
@@ -723,7 +1061,7 @@ const ACMEAutomation = () => {
               style={{ marginBottom: 16 }}
             />
           )}
-          {prerequisites?.steps?.find(s => s.key === 'config_applied' && s.ok === false) && (() => {
+          {!wizardIsDns01 && prerequisites?.steps?.find(s => s.key === 'config_applied' && s.ok === false) && (() => {
             const configStep = prerequisites.steps.find(s => s.key === 'config_applied');
             const pendingNames = (configStep?.pending_clusters || []).map(c => c.name).join(', ');
             return (
@@ -755,8 +1093,17 @@ const ACMEAutomation = () => {
             description={
               <ul style={{ margin: 0, paddingLeft: 20 }}>
                 <li>ACME Account: {activeAccount ? <Tag color="success">Active ({activeAccount.email})</Tag> : <Tag color="error">No active account</Tag>}</li>
-                <li>ACME-enabled Clusters: {acmeEnabledClusters.length > 0 ? <Tag color="success">{acmeEnabledClusters.map(c => c.name).join(', ')}</Tag> : <Tag color="warning">None</Tag>}</li>
-                <li>Domains must resolve to HAProxy node IPs for HTTP-01 validation</li>
+                {wizardIsDns01 ? (
+                  <>
+                    <li>Challenge Method: <Tag>DNS-01</Tag> (TXT record; no port 80 / ACME routing needed)</li>
+                    <li>DNS provider: <Tag>{wizardAccount?.dns_provider || 'manual'}</Tag></li>
+                  </>
+                ) : (
+                  <>
+                    <li>ACME-enabled Clusters: {acmeEnabledClusters.length > 0 ? <Tag color="success">{acmeEnabledClusters.map(c => c.name).join(', ')}</Tag> : <Tag color="warning">None</Tag>}</li>
+                    <li>Domains must resolve to HAProxy node IPs for HTTP-01 validation</li>
+                  </>
+                )}
               </ul>
             }
           />
@@ -807,9 +1154,15 @@ const ACMEAutomation = () => {
 
       {pendingOrders.length > 0 && (() => {
         const stuckCount = pendingOrders.filter(isOrderStuck).length;
-        const inFlightCount = pendingOrders.length - stuckCount;
+        // Manual DNS-01 orders are waiting on the USER to publish a TXT record, not on the CA —
+        // call them out separately so the action item is visible without scanning the table.
+        const manualDnsAwaiting = pendingOrders.filter(o =>
+          o.challenge_type === 'dns-01' && (o.dns_provider || 'manual') === 'manual'
+          && (o.status === 'pending' || o.status === 'processing')).length;
+        const inFlightCount = pendingOrders.length - stuckCount - manualDnsAwaiting;
         const parts = [];
         if (inFlightCount > 0) parts.push(`${inFlightCount} awaiting validation`);
+        if (manualDnsAwaiting > 0) parts.push(`${manualDnsAwaiting} manual DNS-01 awaiting your TXT record${manualDnsAwaiting > 1 ? 's' : ''}`);
         if (stuckCount > 0) parts.push(`${stuckCount} pending download (auto-retrying every 60s)`);
         return (
           <Alert
@@ -817,9 +1170,11 @@ const ACMEAutomation = () => {
             showIcon
             icon={<ExclamationCircleOutlined />}
             message={`${pendingOrders.length} certificate order(s) in progress: ${parts.join(', ')}`}
-            description={stuckCount > 0
-              ? "Stuck orders will auto-complete via the background task. You can also click \"Complete\" to retry immediately."
-              : undefined}
+            description={manualDnsAwaiting > 0
+              ? "Open a manual DNS-01 order to see the TXT record to publish, then confirm it."
+              : stuckCount > 0
+                ? "Stuck orders will auto-complete via the background task. You can also click \"Complete\" to retry immediately."
+                : undefined}
             style={{ marginBottom: 16 }}
           />
         );
@@ -866,6 +1221,11 @@ const ACMEAutomation = () => {
                   <InfoCircleOutlined /> Manage
                 </Button>
               )}
+              {acmeAccountNeedsCreds && (
+                <Button type="link" size="small" style={{ padding: 0 }} onClick={() => openDnsCredsModal(acmeAccount)}>
+                  <KeyOutlined /> DNS credentials
+                </Button>
+              )}
             </div>
           </Card>
         </Col>
@@ -965,7 +1325,7 @@ const ACMEAutomation = () => {
             </Button>
           )}
           {wizardStep === wizardSteps.length - 1 && (
-            <Button type="primary" onClick={handleRequestCert} loading={submitting} disabled={!activeAccount || acmeEnabledClusters.length === 0}>
+            <Button type="primary" onClick={handleRequestCert} loading={submitting} disabled={!activeAccount || wizardWildcardBlocked || wizardDns01Disabled || (!wizardIsDns01 && acmeEnabledClusters.length === 0)}>
               Submit Request
             </Button>
           )}
@@ -994,7 +1354,7 @@ const ACMEAutomation = () => {
                 </Tag>
               ) : orderDetail.status === 'valid' ? (
                 <Tag color="warning" icon={<ExclamationCircleOutlined />}>
-                  Pending download — auto-completion task will retry every 60s
+                  Pending download. Auto-completion task will retry every 60s
                 </Tag>
               ) : orderDetail.status === 'invalid' || orderDetail.status === 'cancelled' ? (
                 <Tag color="default">Not issued</Tag>
@@ -1010,6 +1370,71 @@ const ACMEAutomation = () => {
                 style={{ marginBottom: 16 }}
               />
             )}
+            {orderDetail.challenge_type === 'dns-01'
+              && !(orderDetail.challenges || []).some(c => c.challenge_type === 'dns-01')
+              && (orderDetail.status === 'pending' || orderDetail.status === 'processing') && (
+              <Alert
+                type="info"
+                showIcon
+                style={{ marginBottom: 16 }}
+                message="Preparing DNS-01 challenge"
+                description="The TXT record(s) for this order are being provisioned. They will appear here shortly; this view refreshes automatically."
+              />
+            )}
+            {orderDetail.challenge_type === 'dns-01' && (orderDetail.challenges || []).some(c => c.challenge_type === 'dns-01') && (() => {
+              const dnsChallenges = (orderDetail.challenges || []).filter(c => c.challenge_type === 'dns-01');
+              const isManual = (orderDetail.dns_provider || 'manual') === 'manual';
+              const canConfirm = isManual && (orderDetail.status === 'pending' || orderDetail.status === 'processing');
+              return (
+                <Alert
+                  type={isManual ? 'warning' : 'info'}
+                  showIcon
+                  style={{ marginBottom: 16 }}
+                  message={isManual
+                    ? 'DNS-01 (manual): publish these TXT record(s), then verify'
+                    : 'DNS-01 (automated): the TXT record(s) are published for you'}
+                  description={
+                    <div>
+                      <div style={{ marginBottom: 8 }}>
+                        Add the following DNS TXT record{dnsChallenges.length > 1 ? 's' : ''} at your DNS provider:
+                      </div>
+                      <Table
+                        size="small"
+                        pagination={false}
+                        dataSource={dnsChallenges}
+                        rowKey="id"
+                        scroll={{ x: 'max-content' }}
+                        columns={[
+                          { title: 'Record name', dataIndex: 'dns_record_name', key: 'name',
+                            render: (v) => v ? <Typography.Text code copyable style={{ wordBreak: 'break-all' }}>{v}</Typography.Text> : <Typography.Text type="secondary">-</Typography.Text> },
+                          { title: 'Type', key: 'type', width: 60, render: () => 'TXT' },
+                          { title: 'Value', dataIndex: 'dns_txt_value', key: 'val',
+                            render: (v) => v ? <Typography.Text code copyable style={{ wordBreak: 'break-all' }}>{v}</Typography.Text> : <Typography.Text type="secondary">-</Typography.Text> },
+                        ]}
+                      />
+                      {canConfirm && (
+                        <>
+                          <div style={{ marginTop: 12, fontSize: 12, color: token.colorTextSecondary }}>
+                            DNS changes can take a few minutes to propagate. If verification fails,
+                            wait a short while and try again. The order keeps retrying in the background.
+                          </div>
+                          <Button
+                            type="primary"
+                            size="small"
+                            loading={confirming}
+                            disabled={confirming}
+                            style={{ marginTop: 8 }}
+                            onClick={() => handleDnsConfirm(orderDetail.id)}
+                          >
+                            I've added the record(s), verify now
+                          </Button>
+                        </>
+                      )}
+                    </div>
+                  }
+                />
+              );
+            })()}
             {orderDetail.challenges?.length > 0 && (
               <>
                 <h4>Challenges</h4>
@@ -1020,7 +1445,10 @@ const ACMEAutomation = () => {
                   rowKey="id"
                   columns={[
                     { title: 'Domain', dataIndex: 'domain', key: 'domain' },
-                    { title: 'Token', dataIndex: 'token', key: 'token', ellipsis: true },
+                    { title: 'Method', dataIndex: 'challenge_type', key: 'method', width: 90,
+                      render: (ct) => <Tag>{(ct || 'http-01') === 'dns-01' ? 'DNS-01' : 'HTTP-01'}</Tag> },
+                    { title: 'Token', dataIndex: 'token', key: 'token', ellipsis: true,
+                      render: (t, r) => (r.challenge_type === 'dns-01') ? <Typography.Text type="secondary">TXT-based (see above)</Typography.Text> : t },
                     { title: 'Status', dataIndex: 'status', key: 'status', render: (s) => statusTag(s) },
                   ]}
                 />
@@ -1047,7 +1475,7 @@ const ACMEAutomation = () => {
             rowKey="id"
             columns={[
               {
-                title: 'Email', dataIndex: 'email', key: 'email',
+                title: 'Email', dataIndex: 'email', key: 'email', ellipsis: true,
                 render: (email) => <strong>{email}</strong>,
               },
               {
@@ -1098,6 +1526,10 @@ const ACMEAutomation = () => {
                                   </p>
                                 )}
                                 {record.eab_kid && <p><strong>EAB Key ID:</strong> {record.eab_kid}</p>}
+                                <p><strong>Challenge Method:</strong> {(record.challenge_type || 'http-01') === 'dns-01' ? 'DNS-01' : 'HTTP-01'}</p>
+                                {(record.challenge_type === 'dns-01') && (
+                                  <p><strong>DNS Provider:</strong> {record.dns_provider || 'manual'}</p>
+                                )}
                                 <p><strong>ToS Accepted:</strong> {record.tos_agreed ? 'Yes' : 'No'}</p>
                                 <p><strong>Registered:</strong> {record.created_at ? new Date(record.created_at).toLocaleString() : '-'}</p>
                               </div>
@@ -1106,6 +1538,16 @@ const ACMEAutomation = () => {
                         }}
                       />
                     </Tooltip>
+                    {record.challenge_type === 'dns-01'
+                      && (dnsProviders.find(p => p.name === record.dns_provider)?.credential_fields || []).length > 0 && (
+                      <Tooltip title="DNS Provider Credentials">
+                        <Button
+                          icon={<KeyOutlined />}
+                          size="small"
+                          onClick={() => openDnsCredsModal(record)}
+                        />
+                      </Tooltip>
+                    )}
                     {record.status !== 'deactivated' ? (
                       <Tooltip title="Deactivate">
                         <Button
@@ -1154,7 +1596,7 @@ const ACMEAutomation = () => {
           message="ACME account will be registered with the directory URL configured in Settings > ACME."
           style={{ marginBottom: 16 }}
         />
-        <Form form={registerForm} layout="vertical">
+        <Form form={registerForm} layout="vertical" preserve={false}>
           <Form.Item
             name="email"
             label="Contact Email"
@@ -1167,11 +1609,12 @@ const ACMEAutomation = () => {
           </Form.Item>
           <Form.Item
             name="tos_agreed"
+            label="Terms of Service"
             valuePropName="checked"
             initialValue={false}
             rules={[{ validator: (_, v) => v ? Promise.resolve() : Promise.reject('You must accept the Terms of Service') }]}
           >
-            <Switch checkedChildren="Accepted" unCheckedChildren="Not Accepted" />
+            <Switch checkedChildren="Accepted" unCheckedChildren="Not Accepted" aria-label="Accept Terms of Service" />
           </Form.Item>
           <div style={{ fontSize: 12, color: token.colorTextSecondary }}>
             By accepting, you agree to the ACME CA's Terms of Service (e.g.{' '}
@@ -1179,12 +1622,194 @@ const ACMEAutomation = () => {
               Let's Encrypt Subscriber Agreement
             </a>).
           </div>
+          {/* Issue #35: External Account Binding — required by ZeroSSL / Google Trust Services. */}
+          <Collapse
+            ghost
+            style={{ marginTop: 12 }}
+            items={[{
+              key: 'eab',
+              label: 'External Account Binding (EAB)',
+              children: (
+                <>
+                  <Alert
+                    type="info"
+                    showIcon
+                    style={{ marginBottom: 12 }}
+                    message="Required by some CAs (ZeroSSL, Google Trust Services). Copy the Key ID and HMAC Key from your CA account. Re-enter them on each registration; the HMAC key is not stored."
+                  />
+                  <Form.Item
+                    name="eab_kid"
+                    label="EAB Key ID"
+                    dependencies={['eab_hmac_key']}
+                    rules={[({ getFieldValue }) => ({
+                      validator(_, value) {
+                        const kid = (value || '').trim();
+                        const hmac = (getFieldValue('eab_hmac_key') || '').trim();
+                        if (!kid && hmac) {
+                          return Promise.reject(new Error('EAB Key ID is required when an HMAC Key is entered.'));
+                        }
+                        return Promise.resolve();
+                      },
+                    })]}
+                  >
+                    <Input placeholder="EAB Key Identifier" autoComplete="off" />
+                  </Form.Item>
+                  <Form.Item
+                    name="eab_hmac_key"
+                    label="EAB HMAC Key"
+                    dependencies={['eab_kid']}
+                    rules={[({ getFieldValue }) => ({
+                      validator(_, value) {
+                        const kid = (getFieldValue('eab_kid') || '').trim();
+                        const hmac = (value || '').trim();
+                        if (kid && !hmac) {
+                          return Promise.reject(new Error('EAB HMAC Key is required when a Key ID is entered.'));
+                        }
+                        return Promise.resolve();
+                      },
+                    })]}
+                  >
+                    <Input.Password placeholder="EAB HMAC Key (base64url)" autoComplete="new-password" />
+                  </Form.Item>
+                </>
+              ),
+            }]}
+          />
+          {dns01Enabled && (
+            <>
+              <Divider style={{ margin: '16px 0 12px' }} />
+              <Form.Item
+                name="challenge_type"
+                label="Challenge Method"
+                initialValue="http-01"
+                tooltip="HTTP-01 validates over port 80. DNS-01 validates via a DNS TXT record. It works for internal/isolated clusters (no public port 80) and supports wildcards."
+              >
+                <Select>
+                  <Option value="http-01">HTTP-01 (default)</Option>
+                  <Option value="dns-01">DNS-01 (TXT record)</Option>
+                </Select>
+              </Form.Item>
+              {regChallengeType === 'dns-01' && (
+                <>
+                  {dnsProviders.length === 0 ? (
+                    <Alert
+                      type="warning"
+                      showIcon
+                      message="No DNS providers are available"
+                      description="DNS-01 appears enabled, but no provider catalog was returned. Confirm DNS-01 is enabled in Settings and that the backend is reachable, then reopen this dialog."
+                    />
+                  ) : (
+                  <Form.Item
+                    name="dns_provider"
+                    label="DNS Provider"
+                    rules={[{ required: true, message: 'Select a DNS provider' }]}
+                  >
+                    <Select placeholder="Select a DNS provider">
+                      {dnsProviders.map(p => (
+                        <Option key={p.name} value={p.name}>{p.label}</Option>
+                      ))}
+                    </Select>
+                  </Form.Item>
+                  )}
+                  {selectedDnsProvider && (selectedDnsProvider.credential_fields || []).length > 0 && (
+                    <>
+                      <Alert
+                        type="info"
+                        showIcon
+                        style={{ marginBottom: 12 }}
+                        message="Provider credentials are encrypted at rest and verified before they are saved. The token needs permission to create and delete TXT records in your domain's DNS zone."
+                      />
+                      {(selectedDnsProvider.credential_fields || []).map(f => (
+                        <Form.Item
+                          key={f.key}
+                          name={`cred_${f.key}`}
+                          label={f.label}
+                          extra={f.help ? <span style={{ fontSize: 12, color: token.colorTextSecondary }}>{f.help}</span> : null}
+                          rules={f.required ? [{ required: true, message: `${f.label} is required` }] : []}
+                        >
+                          {f.type === 'password'
+                            ? <Input.Password placeholder={f.label} maxLength={f.max_length || undefined} />
+                            : <Input placeholder={f.label} maxLength={f.max_length || undefined} />}
+                        </Form.Item>
+                      ))}
+                    </>
+                  )}
+                  {selectedDnsProvider && !selectedDnsProvider.automated && (
+                    <Alert
+                      type="warning"
+                      showIcon
+                      message="Manual DNS provider"
+                      description="You will publish the DNS TXT record yourself and confirm it. Manual DNS-01 certificates cannot auto-renew unattended."
+                    />
+                  )}
+                </>
+              )}
+            </>
+          )}
         </Form>
       </Modal>
 
+      {/* Issue #35: DNS provider credentials for an existing DNS-01 account (view / replace / clear) */}
+      <Modal
+        title={<span><KeyOutlined /> DNS Provider Credentials{credAccount ? `: ${credAccount.email}` : ''}</span>}
+        open={credModalVisible}
+        onCancel={() => { setCredModalVisible(false); credForm.resetFields(); }}
+        width={560}
+        footer={[
+          <Button key="clear" danger onClick={handleClearDnsCreds} disabled={!credMeta?.configured || credSaving}>
+            Clear credentials
+          </Button>,
+          <Button key="cancel" onClick={() => { setCredModalVisible(false); credForm.resetFields(); }}>
+            Cancel
+          </Button>,
+          <Button key="save" type="primary" loading={credSaving} onClick={handleSaveDnsCreds}>
+            Save & verify
+          </Button>,
+        ]}
+      >
+        {credLoading ? (
+          <div style={{ textAlign: 'center', padding: 24 }}><Spin /></div>
+        ) : (
+          <>
+            <p style={{ marginBottom: 4 }}>
+              <strong>Provider:</strong> {credAccount?.dns_provider || '-'}
+            </p>
+            <p style={{ marginTop: 0, fontSize: 12, color: token.colorTextSecondary }}>
+              {credMeta?.configured
+                ? `Configured: ${(credMeta.credential_fields_present || []).join(', ') || '(none)'}${credMeta.updated_at ? ` · updated ${new Date(credMeta.updated_at).toLocaleString()}` : ''}`
+                : 'No credentials are stored yet for this account.'}
+            </p>
+            <Alert
+              type="info"
+              showIcon
+              style={{ marginBottom: 16 }}
+              message="Existing values are never shown. Enter the values to (re)store them; they are verified against the provider and encrypted at rest. The token needs permission to create and delete TXT records in your domain's DNS zone."
+            />
+            <Form form={credForm} layout="vertical">
+              {(credProvider?.credential_fields || []).map(f => (
+                <Form.Item
+                  key={f.key}
+                  name={`cred_${f.key}`}
+                  label={f.label}
+                  rules={f.required ? [{ required: true, message: `${f.label} is required` }] : []}
+                  extra={f.help ? <span style={{ fontSize: 12, color: token.colorTextSecondary }}>{f.help}</span> : null}
+                >
+                  {f.type === 'password'
+                    ? <Input.Password placeholder={f.label} maxLength={f.max_length || undefined} />
+                    : <Input placeholder={f.label} maxLength={f.max_length || undefined} />}
+                </Form.Item>
+              ))}
+              {(credProvider?.credential_fields || []).length === 0 && (
+                <Alert type="warning" showIcon message="This provider needs no credentials (manual mode)." />
+              )}
+            </Form>
+          </>
+        )}
+      </Modal>
+
       {/* v1.5.0 Issue #13: ACME Diagnostic Panel modal */}
       <Modal
-        title={diagOrderId ? `Diagnostics — Order #${diagOrderId}` : 'Diagnostics'}
+        title={diagOrderId ? `Diagnostics: Order #${diagOrderId}` : 'Diagnostics'}
         open={diagVisible}
         onCancel={closeDiagModal}
         width={920}
@@ -1266,12 +1891,21 @@ const ACMEAutomation = () => {
                           {
                             title: 'Re-run', key: 'rerun', width: 90,
                             render: (_, record) => (
-                              <Button
-                                size="small"
-                                icon={<ReloadOutlined />}
-                                loading={diagRunningCheckId === record.id}
-                                onClick={() => handleRerunCheck(record.id)}
-                              />
+                              // A skipped check (e.g. port 80 / routing for a DNS-01 order) has no
+                              // transient cause to re-test, so re-running just reproduces "skipped".
+                              record.status === 'skipped'
+                                ? <Typography.Text type="secondary">-</Typography.Text>
+                                : (
+                                  <Tooltip title="Re-run this check">
+                                    <Button
+                                      size="small"
+                                      aria-label="Re-run this check"
+                                      icon={<ReloadOutlined />}
+                                      loading={diagRunningCheckId === record.id}
+                                      onClick={() => handleRerunCheck(record.id)}
+                                    />
+                                  </Tooltip>
+                                )
                             ),
                           },
                         ]}
@@ -1327,13 +1961,13 @@ const ACMEAutomation = () => {
                         type="warning"
                         showIcon
                         style={{ marginBottom: 12 }}
-                        message="Event log partial — one or more sources failed"
+                        message="Event log partial: one or more sources failed"
                         description={
                           <div>
                             <ul style={{ margin: '4px 0 4px 16px' }}>
                               {diagEventsError.errors.map((err, i) => (
                                 <li key={i}>
-                                  <strong>{err.section}</strong>: {err.exception_type} — {err.message}
+                                  <strong>{err.section}</strong>: {err.exception_type}: {err.message}
                                 </li>
                               ))}
                             </ul>
@@ -1359,7 +1993,7 @@ const ACMEAutomation = () => {
                           children: (
                             <div>
                               <div style={{ fontSize: 12, color: '#888' }}>{ev.created_at} · {ev.source}</div>
-                              <div><strong>{ev.event_type}</strong></div>
+                              <div><strong>{humanizeEventType(ev.event_type)}</strong></div>
                               {ev.message && <div>{ev.message}</div>}
                             </div>
                           ),

frontend/src/components/AgentManagement.js

@@ -2378,7 +2378,7 @@ const AgentManagement = () => {
                         const element = document.createElement('a');
                         const file = new Blob([installScript], { type: 'text/plain' });
                         element.href = URL.createObjectURL(file);
-                        element.download = `install-haproxy-agent-${selectedPlatform}.sh`;
+                        element.download = `install-agent-${selectedPlatform}.sh`;
                         document.body.appendChild(element);
                         element.click();
                         document.body.removeChild(element);
@@ -2516,7 +2516,7 @@ const AgentManagement = () => {
                                     const element = document.createElement('a');
                                     const file = new Blob([uninstallScript], { type: 'text/plain' });
                                     element.href = URL.createObjectURL(file);
-                                    element.download = `uninstall-haproxy-agent-${selectedPlatform}.sh`;
+                                    element.download = `uninstall-agent-${selectedPlatform}.sh`;
                                     document.body.appendChild(element);
                                     element.click();
                                     document.body.removeChild(element);
@@ -3147,7 +3147,7 @@ const AgentManagement = () => {
                           const element = document.createElement('a');
                           const file = new Blob([deleteUninstallScript], { type: 'text/plain' });
                           element.href = URL.createObjectURL(file);
-                          element.download = `uninstall-haproxy-agent-${agentToDelete.platform || 'linux'}.sh`;
+                          element.download = `uninstall-agent-${agentToDelete.platform || 'linux'}.sh`;
                           document.body.appendChild(element);
                           element.click();
                           document.body.removeChild(element);

frontend/src/components/ApplyManagement.js

@@ -37,6 +37,7 @@ const ApplyManagement = () => {
     backends: [],
     waf_rules: [],
     ssl_certificates: [],
+    vips: [],
     total_count: 0
   });
   const [configVersions, setConfigVersions] = useState([]);
@@ -53,6 +54,8 @@ const ApplyManagement = () => {
   // Validation Error Modal state
   const [validationErrorModalVisible, setValidationErrorModalVisible] = useState(false);
   const [selectedValidationError, setSelectedValidationError] = useState(null);
+  // HA/VIP (Issue #27): VIP changes appear in the standard right-panel "Pending Versions"
+  // (vip-* config_versions) and use the standard "View Change" diff — no bespoke VIP modal.
 
   // Initial load on component mount
   useEffect(() => {
@@ -67,7 +70,7 @@ const ApplyManagement = () => {
   useEffect(() => {
     if (selectedCluster) {
       // CRITICAL: Clear all state immediately when cluster changes to prevent cross-cluster contamination
-      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], total_count: 0 });
+      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], vips: [], total_count: 0 });
       setConfigVersions([]);
       setAgentSync(null);
       setEntitySyncStates({});
@@ -77,7 +80,7 @@ const ApplyManagement = () => {
       fetchConfigVersions();
       fetchAgentSync();
     } else {
-      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], total_count: 0 });
+      setPendingChanges({ frontends: [], backends: [], waf_rules: [], ssl_certificates: [], vips: [], total_count: 0 });
       setConfigVersions([]);
       setAgentSync(null);
       setEntitySyncStates({});
@@ -125,26 +128,32 @@ const ApplyManagement = () => {
         'Pragma': 'no-cache'
       };
       
-      const [frontendsRes, backendsRes, wafRes, sslRes] = await Promise.all([
-        axios.get('/api/frontends', { 
+      const [frontendsRes, backendsRes, wafRes, sslRes, vipsRes] = await Promise.all([
+        axios.get('/api/frontends', {
           params: { cluster_id: selectedCluster.id, include_inactive: true },
           headers: cacheHeaders
         }).catch(() => ({ data: { frontends: [] } })),
-        
-        axios.get('/api/backends', { 
+
+        axios.get('/api/backends', {
           params: { cluster_id: selectedCluster.id, include_inactive: true },
           headers: cacheHeaders
         }).catch(() => ({ data: { backends: [] } })),
-        
-        axios.get('/api/waf/rules', { 
+
+        axios.get('/api/waf/rules', {
           params: { cluster_id: selectedCluster.id },
           headers: cacheHeaders
         }).catch(() => ({ data: { rules: [] } })),
-        
-        axios.get('/api/ssl/certificates', { 
+
+        axios.get('/api/ssl/certificates', {
           params: { cluster_id: selectedCluster.id },
           headers: cacheHeaders
-        }).catch(() => ({ data: [] }))
+        }).catch(() => ({ data: [] })),
+
+        // HA/VIP (Issue #27): pool-scoped, fetched cluster-scoped so it lists like other entities
+        axios.get('/api/vip', {
+          params: { cluster_id: selectedCluster.id },
+          headers: cacheHeaders
+        }).catch(() => ({ data: { vips: [] } }))
       ]);
 
       // DEBUG: Log raw backend data
@@ -180,14 +189,17 @@ const ApplyManagement = () => {
       
       const waf_rules = (wafRes.data.rules || []).filter(w => w.has_pending_config);
       const ssl_certificates = (sslRes.data.ssl_certificates || sslRes.data || []).filter(s => s.has_pending_config);
+      // VIP "pending" is its last_config_status (no has_pending_config flag).
+      const vips = (vipsRes.data.vips || []).filter(v => v.last_config_status === 'PENDING');
 
-      const total_count = frontends.length + backends.length + waf_rules.length + ssl_certificates.length;
+      const total_count = frontends.length + backends.length + waf_rules.length + ssl_certificates.length + vips.length;
 
       setPendingChanges({
         frontends,
         backends,
         waf_rules,
         ssl_certificates,
+        vips,
         total_count
       });
 
@@ -356,7 +368,7 @@ const ApplyManagement = () => {
       title: 'Apply All Configuration Changes',
       content: (
         <div>
-          <p>You are about to apply <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to apply <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -370,6 +382,15 @@ const ApplyManagement = () => {
             {pendingChanges.ssl_certificates.length > 0 && (
               <li><strong>{pendingChanges.ssl_certificates.length}</strong> SSL certificate changes</li>
             )}
+            {(pendingChanges.vips || []).length > 0 && (
+              <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
+            )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All changes will be applied together and sent to agents"
@@ -387,6 +408,74 @@ const ApplyManagement = () => {
     });
   };
 
+  // HA/VIP convergence tracking (Issue #27 follow-up). VIP teardown/deploy is asynchronous —
+  // member agents converge keepalived on their next poll. Mirror the HAProxy agent-sync widget
+  // so the progress popup keeps showing "Syncing HA/VIP... X/Y" until the nodes report the VIP
+  // ACTIVE, instead of flashing green while the HA/VIP page still shows SYNCING. Fire-and-forget
+  // recursive poll exactly like checkAgentSync (applyLoading is released immediately; this runs
+  // in the background and updates the floating widget). Bounded so an offline node can't poll
+  // forever — it then completes with an informational "still converging" note.
+  const trackVipConvergence = (clusterId, vipIds, startedAt) => {
+    const token = localStorage.getItem('token');
+    const total = vipIds.length;
+    const poll = async () => {
+      let vips = [];
+      try {
+        const r = await axios.get(`/api/vip?cluster_id=${clusterId}`, { headers: { Authorization: `Bearer ${token}` } });
+        vips = r.data.vips || [];
+      } catch (e) { /* transient — keep polling */ }
+      // A change has converged when the node reports the VIP ACTIVE (create/edit) OR the VIP is
+      // fully torn down (delete approval): deploy_status DELETED, or it has dropped off the list
+      // entirely once every member acked the teardown.
+      const isConverged = (vid) => {
+        const v = vips.find(x => x.id === vid);
+        if (!v) return true;                       // gone from the list → torn down / deleted
+        return v.deploy_status === 'ACTIVE' || v.deploy_status === 'DELETED';
+      };
+      const synced = vipIds.filter(isConverged).length;   // VIP-level (drives completion)
+      const errored = vips.filter(v => vipIds.includes(v.id)
+        && (v.deploy_status === 'ERROR' || v.deploy_status === 'ATTENTION')).length;
+      // Per-NODE progress: sum member acks across the tracked VIPs still present, so a multi-node
+      // VIP shows "1/2 node(s)" like the HA/VIP table — not just "1 change". Gone (torn-down) VIPs
+      // are already counted converged via isConverged.
+      let nodeTotal = 0, nodeSynced = 0;
+      for (const vid of vipIds) {
+        const v = vips.find(x => x.id === vid);
+        if (v) { nodeTotal += (v.deploy_total || 0); nodeSynced += (v.deploy_synced || 0); }
+      }
+      const label = nodeTotal > 0 ? `${nodeSynced}/${nodeTotal} node(s)` : `${synced}/${total} change(s)`;
+      const prog = Math.min(95, 60 + Math.round(35 * (nodeTotal > 0 ? nodeSynced / nodeTotal : synced / Math.max(1, total))));
+      updateEntityCounts(nodeTotal > 0 ? nodeSynced : synced, nodeTotal > 0 ? nodeTotal : total, 0, 0, 0);
+
+      if (synced === total) {
+        const msg = `${total} HA/VIP change(s) fully converged on all member node(s).`;
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.success(msg); }, 1500);
+        return;
+      }
+      if (errored > 0) {
+        const msg = 'HA/VIP applied, but a member node reported an issue — open Diagnostics on the HA / VIP page.';
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.warning(msg); }, 1800);
+        return;
+      }
+      if (Date.now() - startedAt > 300000) { // ~5 min cap (e.g. an offline member node)
+        const msg = `HA/VIP applied — ${label} converged; the rest are still converging (or a node's agent is offline). Track live status on the HA / VIP page.`;
+        setSyncProgress({ visible: true, step: msg, progress: 100 });
+        completeProgress(msg);
+        setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.info(msg); }, 2000);
+        return;
+      }
+      const step = `Syncing HA/VIP... ${label} converged (keepalived deploy can take a couple of minutes)`;
+      setSyncProgress({ visible: true, step, progress: prog });
+      updateProgress(step, prog, { 'Synced HA/VIP': label });
+      setTimeout(poll, 5000);
+    };
+    setTimeout(poll, 2000);
+  };
+
   const executeApplyAll = async () => {
     setApplyLoading(true);
     
@@ -429,13 +518,38 @@ const ApplyManagement = () => {
       
       // Detect if this is a restore operation (safe display-only check)
       const pendingVersions = configVersions.filter(v => v.status === 'PENDING');
-      const isRestoreOperation = totalEntities === 0 && pendingVersions.some(v => v.version_name.startsWith('restore-'));
-      
+      // HA/VIP (Issue #27): vip-* versions are VIP-owned and excluded from the HAProxy apply
+      // (cluster.py). Ignore them when deciding whether this apply has any HAProxy/cluster
+      // work to track — otherwise a VIP-only apply (which now stages a vip-* version) would
+      // look like it has a pending version and the agent-sync tracker would hang on 0/0.
+      const nonVipPendingVersions = pendingVersions.filter(v => !(v.version_name || '').startsWith('vip-'));
+      const isRestoreOperation = totalEntities === 0 && nonVipPendingVersions.some(v => v.version_name.startsWith('restore-'));
+      // HA/VIP-only apply: no HAProxy entity or config-version goes through the cluster-sync
+      // pipeline, so complete promptly after the isolated VIP apply (member nodes converge
+      // async on their next agent poll) instead of looping on "Entities: 0/0".
+      const vipCount = (pendingChanges.vips || []).length;
+      const isVipOnly = totalEntities === 0 && !isRestoreOperation && nonVipPendingVersions.length === 0 && vipCount > 0;
+      // Config-version-only apply (e.g. cluster ACME enable/disable): no entity rows, not a restore,
+      // but there ARE non-vip config versions to push. Without this branch it falls to the "else" and
+      // shows a misleading "Entities: 0/0" while still syncing agents.
+      const isConfigVersionOnly = totalEntities === 0 && !isRestoreOperation && vipCount === 0 && nonVipPendingVersions.length > 0;
+
       if (isRestoreOperation) {
         // Restore operation: Show "Configuration" instead of "Entities"
         setSyncProgress({ visible: true, step: `Applying configuration restore... Configuration: 0/1, Agents: ⏳`, progress: 20 });
         startProgress('apply', `Applying configuration restore... Configuration: 0/1, Agents: ⏳`);
         updateEntityCounts(0, 1, 0, totalAgents, disabledAgents); // Show 1 configuration item
+      } else if (isVipOnly) {
+        // HA/VIP-only: avoid the "Entities: 0/0" / agent-sync widget entirely.
+        setSyncProgress({ visible: true, step: `Applying ${vipCount} HA/VIP change(s)...`, progress: 20 });
+        startProgress('apply', `Applying ${vipCount} HA/VIP change(s)...`);
+        updateEntityCounts(0, vipCount, 0, 0, 0);
+      } else if (isConfigVersionOnly) {
+        // Config-version-only (ACME toggle, etc.): show "Configuration" instead of "Entities: 0/0".
+        const cfgCount = nonVipPendingVersions.length;
+        setSyncProgress({ visible: true, step: `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`, progress: 20 });
+        startProgress('apply', `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`);
+        updateEntityCounts(0, cfgCount, 0, totalAgents, disabledAgents);
       } else {
         // Normal operation: Show "Entities" as usual
         setSyncProgress({ visible: true, step: `Applying configuration changes... Entities: 0/${totalEntities}, Agents: ⏳`, progress: 20 });
@@ -445,16 +559,61 @@ const ApplyManagement = () => {
     
     try {
       const token = localStorage.getItem('token');
-      
-      const response = await axios.post(
-        `/api/clusters/${selectedCluster.id}/apply-changes`,
-        {},
-        { headers: { Authorization: `Bearer ${token}` } }
-      );
-      
+
+      // HA/VIP (Issue #27): apply pending VIPs first (isolated endpoint; safe no-op for
+      // HAProxy config). Done here so VIPs apply even when there are no HAProxy changes.
+      const pendingVips = pendingChanges.vips || [];
+      for (const vip of pendingVips) {
+        try {
+          await axios.post(`/api/vip/${vip.id}/apply`, {}, { headers: { Authorization: `Bearer ${token}` } });
+        } catch (vipErr) {
+          console.error(`[APPLY VIP ${vip.id}] failed:`, vipErr);
+          message.warning(`VIP "${vip.name}" apply failed: ${extractApiError(vipErr, 'error')}`);
+        }
+      }
+
+      // Apply HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (cluster ACME enable/disable, restore, bulk-import). Gating only on haproxyPending used to
+      // skip the call for config-version-only states, leaving those versions stuck PENDING.
+      const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
+        + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
+        ? await axios.post(
+            `/api/clusters/${selectedCluster.id}/apply-changes`,
+            {},
+            { headers: { Authorization: `Bearer ${token}` } }
+          )
+        : { data: { message: `Applied ${pendingVips.length} HA/VIP change(s)`, applied_count: pendingVips.length } };
+
       // CRITICAL DEBUG: Log apply response
       console.log('[APPLY ALL RESPONSE]:', response.data);
-      
+
+      // HA/VIP-only apply (Issue #27): nothing goes through the HAProxy config-version /
+      // cluster-sync pipeline, so the agent-sync tracker below would loop forever on
+      // "Entities: 0/0". The isolated VIP apply already staged each node's snapshot; member
+      // agents install/configure keepalived and converge on their next poll — live status
+      // shows on the HA / VIP page (PENDING → SYNCING → ACTIVE). Complete now (no flash).
+      if (haproxyPending === 0 && nonVipPendingVersions.length === 0) {
+        await fetchPendingChanges();
+        await fetchConfigVersions();
+        if (pendingVips.length > 0) {
+          // Keep the popup in a "Syncing HA/VIP... X/Y" state (like every other entity) until the
+          // member nodes report the VIP ACTIVE — instead of flashing green while the HA/VIP page
+          // still shows SYNCING. The poll runs in the background (finally{} releases applyLoading).
+          const step = `Applied — syncing HA/VIP... 0/${pendingVips.length} node(s) converged`;
+          updateEntityCounts(0, pendingVips.length, 0, 0, 0);
+          setSyncProgress({ visible: true, step, progress: 60 });
+          updateProgress(step, 60);
+          trackVipConvergence(selectedCluster.id, pendingVips.map(v => v.id), Date.now());
+        } else {
+          const vmsg = 'Changes applied.';
+          setSyncProgress({ visible: true, step: vmsg, progress: 100 });
+          completeProgress(vmsg);
+          setTimeout(() => { setSyncProgress({ visible: false, step: '', progress: 0 }); message.success(vmsg); }, 1500);
+        }
+        return; // finally{} resets applyLoading; the VIP poll runs in the background (like checkAgentSync)
+      }
+
       setSyncProgress({ visible: true, step: `Configuration applied, syncing agents... Entities: ${totalEntities}/${totalEntities}, Agents: 0/${totalAgents}`, progress: 60 });
       updateProgress(`Configuration applied, syncing agents... Entities: ${totalEntities}/${totalEntities}, Agents: 0/${totalAgents}`, 60);
       updateEntityCounts(totalEntities, totalEntities, 0, totalAgents, disabledAgents);
@@ -664,7 +823,7 @@ const ApplyManagement = () => {
       title: 'Reject All Configuration Changes',
       content: (
         <div>
-          <p>You are about to reject <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to reject <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -678,6 +837,15 @@ const ApplyManagement = () => {
             {pendingChanges.ssl_certificates.length > 0 && (
               <li><strong>{pendingChanges.ssl_certificates.length}</strong> SSL certificate changes</li>
             )}
+            {(pendingChanges.vips || []).length > 0 && (
+              <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
+            )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All pending changes will be permanently discarded"
@@ -718,12 +886,34 @@ const ApplyManagement = () => {
     
     try {
       const token = localStorage.getItem('token');
-      
-      const response = await axios.delete(
-        `/api/clusters/${selectedCluster.id}/pending-changes`,
-        { headers: { Authorization: `Bearer ${token}` } }
+
+      // HA/VIP (Issue #27): reject pending VIPs (isolated endpoint; restores last applied state).
+      const pendingVips = pendingChanges.vips || [];
+      for (const vip of pendingVips) {
+        try {
+          await axios.post(`/api/vip/${vip.id}/reject`, {}, { headers: { Authorization: `Bearer ${token}` } });
+        } catch (vipErr) {
+          console.error(`[REJECT VIP ${vip.id}] failed:`, vipErr);
+          message.warning(`VIP "${vip.name}" reject failed: ${extractApiError(vipErr, 'error')}`);
+        }
+      }
+
+      // Reject HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (e.g. cluster ACME enable/disable, restore, bulk-import) — the backend DELETE rejects all
+      // non-vip PENDING versions and rolls back their snapshots. Gating only on haproxyPending used
+      // to skip the call for config-version-only states, returning the misleading "Rejected 0 HA/VIP".
+      const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
+        + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
+      const nonVipPendingVersions = configVersions.filter(
+        v => v.status === 'PENDING' && !(v.version_name || '').startsWith('vip-')
       );
-      
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
+        ? await axios.delete(
+            `/api/clusters/${selectedCluster.id}/pending-changes`,
+            { headers: { Authorization: `Bearer ${token}` } }
+          )
+        : { data: { message: `Rejected ${pendingVips.length} HA/VIP change(s)` } };
+
       // CRITICAL DEBUG: Log reject response
       console.log('[REJECT ALL RESPONSE]:', response.data);
       
@@ -797,6 +987,23 @@ const ApplyManagement = () => {
   const appliedVersions = configVersions.filter(v => v.status === 'APPLIED');
   const rejectedVersions = configVersions.filter(v => v.status === 'REJECTED');
   const effectiveTotal = pendingChanges.total_count > 0 ? pendingChanges.total_count : pendingVersions.length;
+  // Issue #35: cluster ACME enable/disable produce `cluster-<id>-acme-<enable|disable>-<ts>` config
+  // versions that have NO entity-level pending flag, so they were neither categorized nor counted.
+  const ACME_VERSION_RE = /^cluster-\d+-acme-(enable|disable)-/;
+  const ENTITY_VERSION_PREFIXES = ['frontend-', 'backend-', 'server-', 'ssl-', 'waf-'];
+  const acmeVersions = pendingVersions.filter(v => ACME_VERSION_RE.test(v.version_name || ''));
+  // "Other" config versions for the confirm modal = non-vip, non-acme, non-entity-backed (i.e.
+  // restore-*/bulk-import-*/other cluster-level) — entity-backed versions are already counted via
+  // total_count, and vips are listed separately, so excluding them avoids double-counting.
+  const otherConfigVersions = pendingVersions.filter(v => {
+    const n = v.version_name || '';
+    if (n.startsWith('vip-') || ACME_VERSION_RE.test(n)) return false;
+    return !ENTITY_VERSION_PREFIXES.some(p => n.startsWith(p));
+  });
+  // Confirm-modal header count: entities+VIPs (total_count) + ACME + other config versions, so the
+  // header equals the sum of the listed <li> items in every state. The button-enable gate keeps
+  // using effectiveTotal (unchanged) so entity-only button/Alert behavior is byte-identical.
+  const modalChangeCount = (pendingChanges.total_count || 0) + acmeVersions.length + otherConfigVersions.length;
 
   const renderPendingItem = (item, type, icon) => {
     // For PENDING items, don't show sync status since they haven't been applied yet
@@ -1133,12 +1340,67 @@ const ApplyManagement = () => {
                     </div>
                   )}
 
-                  {pendingChanges.frontends.length === 0 && pendingChanges.backends.length === 0 && pendingChanges.waf_rules.length === 0 && pendingChanges.ssl_certificates.length === 0 && pendingVersions.length > 0 && (
+                  {/* HA/VIP Changes (Issue #27) */}
+                  {(pendingChanges.vips || []).length > 0 && (
+                    <div style={{ marginBottom: 16 }}>
+                      <Title level={5}>
+                        <CloudServerOutlined style={{ marginRight: 8, color: '#13c2c2' }} />
+                        HA / VIP Changes ({pendingChanges.vips.length})
+                      </Title>
+                      {pendingChanges.vips.map(item => (
+                        <div key={`vip-${item.id}`} style={{ display: 'flex', alignItems: 'center', gap: 8, padding: '8px 12px', marginBottom: 6, border: item.pending_delete ? '1px solid #ffccc7' : '1px solid #f0f0f0', borderRadius: 6, background: item.pending_delete ? '#fff1f0' : undefined }}>
+                          <CloudServerOutlined style={{ color: item.pending_delete ? '#cf1322' : '#13c2c2' }} />
+                          <span style={{ fontWeight: 500 }}>{item.name}</span>
+                          <Tag>{item.virtual_ip}/{item.prefix_length}</Tag>
+                          {item.pool_name && <Tag color="blue">{item.pool_name}</Tag>}
+                          {item.pending_delete
+                            ? <Tag color="red">DELETION — approve to tear down, reject to keep</Tag>
+                            : <Tag color="orange">PENDING</Tag>}
+                        </div>
+                      ))}
+                    </div>
+                  )}
+
+                  {/* Issue #35: ACME Challenge Routing (cluster-<id>-acme-*) versions have no entity
+                      flag. Render them in their own section REGARDLESS of whether entity sections are
+                      present, so a co-pending ACME toggle is never hidden in the left panel. */}
+                  {acmeVersions.length > 0 && (
+                    <div style={{ marginTop: 8, marginBottom: 8 }}>
+                      <Title level={5}>
+                        <SafetyCertificateOutlined style={{ marginRight: 8, color: '#1890ff' }} />
+                        ACME Challenge Routing ({acmeVersions.length})
+                      </Title>
+                      {acmeVersions.map(v => {
+                        const isEnable = /^cluster-\d+-acme-enable-/.test(v.version_name);
+                        return (
+                          <div key={v.id} style={{
+                            padding: 10, border: '1px dashed #1890ff', borderRadius: 6, marginBottom: 8,
+                            display: 'flex', alignItems: 'center', justifyContent: 'space-between', backgroundColor: '#f0f8ff'
+                          }}>
+                            <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
+                            <span>
+                              <Tag color={isEnable ? 'green' : 'default'}>{isEnable ? 'ENABLE' : 'DISABLE'}</Tag>
+                              <Tag color="orange">PENDING</Tag>
+                            </span>
+                          </div>
+                        );
+                      })}
+                      <div style={{ fontSize: 12, color: token.colorTextSecondary, marginTop: 4 }}>
+                        ACME challenge routing change. Apply to push the updated HAProxy config to the agents.
+                      </div>
+                    </div>
+                  )}
+
+                  {pendingChanges.frontends.length === 0 && pendingChanges.backends.length === 0 && pendingChanges.waf_rules.length === 0 && pendingChanges.ssl_certificates.length === 0 && (pendingChanges.vips || []).length === 0 && pendingVersions.length > 0 && (
                     <div style={{ marginTop: 8 }}>
                       {(() => {
                         const restoreVersions = pendingVersions.filter(v => v.version_name.startsWith('restore-'));
                         const bulkImportVersions = pendingVersions.filter(v => v.version_name.startsWith('bulk-import-'));
-                        const otherVersions = pendingVersions.filter(v => !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-'));
+                        // Exclude restore-/bulk-import- (own sections), vip-* (VIP section), and acme-*
+                        // (the dedicated ACME section above) so they aren't duplicated in "Other".
+                        const otherVersions = pendingVersions.filter(v =>
+                          !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-')
+                          && !v.version_name.startsWith('vip-') && !ACME_VERSION_RE.test(v.version_name));
                         
                         return (
                           <>
@@ -1401,6 +1663,10 @@ const ApplyManagement = () => {
                                     confusing error post-click. */}
                                 {(() => {
                                   const vn = version?.version_name || '';
+                                  // vip-* versions ARE undoable (Issue #27): the VIP router
+                                  // re-stages the rejected change as PENDING (reactivating a
+                                  // rejected create, or re-applying a rejected edit), so they
+                                  // use the normal Undo button below — not this disabled case.
                                   const destructive = (
                                     vn.startsWith('bulk-site-create-')
                                     || vn.startsWith('bulk-import-')

frontend/src/components/BackendServers.js

@@ -172,11 +172,13 @@ const BackendServers = () => {
     
     setLoading(true);
     try {
-      const params = { cluster_id: selectedCluster.id };
-      
+      // Issue #24: request inactive entities so DISABLED (toggled-OFF) servers
+      // are returned and can be reactivated from the UI (mirrors ApplyManagement).
+      const params = { cluster_id: selectedCluster.id, include_inactive: true };
+
       // CRITICAL FIX: Add cache busting to prevent stale data from appearing
       // Browser/axios may cache GET requests, causing deleted backends to reappear
-      const response = await axios.get('/api/backends', { 
+      const response = await axios.get('/api/backends', {
         params,
         headers: {
           'Cache-Control': 'no-cache, no-store, must-revalidate',
@@ -184,7 +186,21 @@ const BackendServers = () => {
           'Expires': '0'
         }
       });
-      const fetchedBackends = response.data.backends || [];
+      // Issue #24: we now request include_inactive=true so DISABLED servers come
+      // back. That ALSO returns soft-deleted (is_active=false) BACKENDS, which the
+      // pre-change default (include_inactive=false -> WHERE is_active=TRUE) hid.
+      // Restore that filter here so soft-deleted backends don't reappear in the
+      // normal view (preserve commit f34a6ee), while still keeping inactive
+      // SERVERS inside active backends. Then hide ONLY soft-deleted-pending
+      // servers (last_config_status==='DELETION'); DISABLED servers (toggled
+      // OFF — status PENDING/APPLIED) stay visible so operators can re-enable
+      // them. Single chokepoint covers counts, expanded rows, and All Servers tab.
+      const fetchedBackends = (response.data.backends || [])
+        .filter(b => b.is_active !== false)
+        .map(b => ({
+          ...b,
+          servers: (b.servers || []).filter(s => s.last_config_status !== 'DELETION'),
+        }));
       setBackends(fetchedBackends);
       // CRITICAL FIX: Apply status filters after fetching to maintain filter state
       // This prevents backends from disappearing when updated (e.g., APPLIED → PENDING)
@@ -1289,6 +1305,7 @@ const BackendServers = () => {
           />
           <strong>{text}</strong>
           {record.backup_server && <Tag color="orange">Backup</Tag>}
+          {!record.is_active && <Tag color="red">Inactive</Tag>}
         </Space>
       ),
     },

frontend/src/components/ClusterManagement.js

@@ -156,6 +156,7 @@ const ClusterManagement = () => {
       stats_socket_path: cluster.stats_socket_path || '/run/haproxy/admin.sock',
       haproxy_config_path: cluster.haproxy_config_path || '/etc/haproxy/haproxy.cfg',
       haproxy_bin_path: cluster.haproxy_bin_path,
+      keepalived_config_path: cluster.keepalived_config_path || '/etc/keepalived/keepalived.conf',
       agent_pool_id: cluster.pool_id || undefined,
       haproxy_user: cluster.haproxy_user || '',
       haproxy_group: cluster.haproxy_group || '',
@@ -197,6 +198,7 @@ const ClusterManagement = () => {
         stats_socket_path: values.stats_socket_path,
         haproxy_config_path: values.haproxy_config_path,
         haproxy_bin_path: values.haproxy_bin_path,
+        keepalived_config_path: values.keepalived_config_path,
         pool_id: values.agent_pool_id || null,
         haproxy_user: values.haproxy_user || null,
         haproxy_group: values.haproxy_group || null,
@@ -748,6 +750,14 @@ const ClusterManagement = () => {
             <Input placeholder="/usr/sbin/haproxy" />
           </Form.Item>
 
+          <Form.Item
+            label="Keepalived Config Path"
+            name="keepalived_config_path"
+            tooltip="Where the agent writes keepalived.conf for this cluster's HA/VIPs. The default (/etc/keepalived/keepalived.conf) is what the keepalived service loads on every distro — leave it unless you run a non-standard install AND have configured the keepalived service/unit to load this exact path."
+          >
+            <Input placeholder="/etc/keepalived/keepalived.conf" />
+          </Form.Item>
+
           <Form.Item
             label="ACME Challenge Routing"
             name="acme_enabled"

frontend/src/components/Login.js

@@ -1,20 +1,20 @@
-import React, { useState } from 'react';
-import { 
-  Card, 
-  Form, 
-  Input, 
-  Button, 
-  message, 
-  Typography, 
-  Row, 
+import React, { useEffect, useRef, useState } from 'react';
+import {
+  Card,
+  Form,
+  Input,
+  Button,
+  message,
+  Typography,
+  Row,
   Col,
   Alert,
-  Spin
 } from 'antd';
-import { 
-  UserOutlined, 
-  LockOutlined, 
-  ClusterOutlined 
+import {
+  UserOutlined,
+  LockOutlined,
+  ClusterOutlined,
+  SafetyCertificateOutlined,
 } from '@ant-design/icons';
 import axios from 'axios';
 import { useAuth } from '../contexts/AuthContext';
@@ -23,77 +23,260 @@ import './Login.css';
 
 const { Title, Text } = Typography;
 
+const PHASE_CREDENTIALS = 'credentials';
+const PHASE_MFA = 'mfa';
+const PHASE_SUBMITTING = 'submitting';
+
 const Login = () => {
-  const [form] = Form.useForm();
+  const [credentialsForm] = Form.useForm();
+  const [mfaForm] = Form.useForm();
+  const [phase, setPhase] = useState(PHASE_CREDENTIALS);
   const [loading, setLoading] = useState(false);
   const [error, setError] = useState('');
   const { login } = useAuth();
 
-  const handleSubmit = async (values) => {
+  // MFA-specific state — RAM only, never persisted.
+  const mfaTokenRef = useRef(null);
+  const [mfaExpiresAt, setMfaExpiresAt] = useState(null);
+  const [mfaCountdown, setMfaCountdown] = useState(0);
+
+  useEffect(() => {
+    if (phase !== PHASE_MFA || !mfaExpiresAt) return undefined;
+    const id = setInterval(() => {
+      const remaining = Math.max(0, Math.floor((mfaExpiresAt - Date.now()) / 1000));
+      setMfaCountdown(remaining);
+      if (remaining <= 0) {
+        clearInterval(id);
+        resetToCredentials('MFA session expired. Please log in again.');
+      }
+    }, 1000);
+    return () => clearInterval(id);
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, [phase, mfaExpiresAt]);
+
+  const resetToCredentials = (errMessage) => {
+    mfaTokenRef.current = null;
+    setMfaExpiresAt(null);
+    setMfaCountdown(0);
+    mfaForm.resetFields();
+    setPhase(PHASE_CREDENTIALS);
+    if (errMessage) setError(errMessage);
+  };
+
+  const completeAuth = (authData) => {
+    // Write storage + axios header ONLY after a full, MFA-cleared response.
+    localStorage.setItem('token', authData.access_token);
+    localStorage.setItem('authToken', authData.access_token);
+    localStorage.setItem('userData', JSON.stringify(authData.user));
+    localStorage.setItem('userRoles', JSON.stringify([]));
+    localStorage.setItem('userPermissions', JSON.stringify({}));
+    const expiryDate = new Date();
+    expiryDate.setSeconds(expiryDate.getSeconds() + authData.expires_in);
+    localStorage.setItem('tokenExpiry', expiryDate.toISOString());
+    const loginSuccess = login(authData);
+    if (loginSuccess) {
+      message.success(`Welcome back, ${authData.user.username}!`);
+    } else {
+      throw new Error('Failed to update authentication state');
+    }
+  };
+
+  const handleCredentialsSubmit = async (values) => {
     setLoading(true);
+    setPhase(PHASE_SUBMITTING);
     setError('');
-    
     try {
       const response = await axios.post('/api/auth/login', {
         username: values.username,
-        password: values.password
+        password: values.password,
       });
 
-      // Store authentication data - API returns access_token, not session_token!
-      localStorage.setItem('token', response.data.access_token);
-      localStorage.setItem('authToken', response.data.access_token);
-      localStorage.setItem('userData', JSON.stringify(response.data.user));
-      localStorage.setItem('userRoles', JSON.stringify([])); // API doesn't return roles directly
-      localStorage.setItem('userPermissions', JSON.stringify({})); // API doesn't return permissions directly
-      
-      // Calculate expiry from expires_in (seconds)
-      const expiryDate = new Date();
-      expiryDate.setSeconds(expiryDate.getSeconds() + response.data.expires_in);
-      localStorage.setItem('tokenExpiry', expiryDate.toISOString());
-
-      // Update authentication context
-      const loginSuccess = login(response.data);
-      
-      if (loginSuccess) {
-        message.success(`Welcome back, ${response.data.user.username}!`);
-        // The authentication context will automatically trigger a re-render
-        // and the user will be redirected to the main dashboard
-      } else {
-        throw new Error('Failed to update authentication state');
+      if (response.data && response.data.mfa_required) {
+        // Phase 2 — TOTP / backup code challenge. Keep credentials secret-free.
+        mfaTokenRef.current = response.data.mfa_token;
+        const ttlSeconds = response.data.expires_in || 300;
+        setMfaExpiresAt(Date.now() + ttlSeconds * 1000);
+        setMfaCountdown(ttlSeconds);
+        setPhase(PHASE_MFA);
+        return;
       }
-      
-    } catch (error) {
-      const errorMessage = extractApiError(error, 'Login failed. Please try again.');
+
+      completeAuth(response.data);
+    } catch (err) {
+      const errorMessage = extractApiError(err, 'Login failed. Please try again.');
       setError(errorMessage);
       message.error(errorMessage);
+      setPhase(PHASE_CREDENTIALS);
     } finally {
       setLoading(false);
     }
   };
 
+  const handleMfaSubmit = async (values) => {
+    if (!mfaTokenRef.current) {
+      resetToCredentials('MFA session lost. Please log in again.');
+      return;
+    }
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post(
+        '/api/auth/login/mfa-verify',
+        {
+          mfa_token: mfaTokenRef.current,
+          code: (values.code || '').trim(),
+        },
+        // Explicit opt-out: never attach a stale Authorization header here.
+        { headers: { Authorization: undefined } },
+      );
+      completeAuth(response.data);
+    } catch (err) {
+      const status = err && err.response && err.response.status;
+      const errorMessage = extractApiError(err, 'Verification failed.');
+      if (status === 410) {
+        resetToCredentials(errorMessage || 'MFA challenge invalidated. Please log in again.');
+      } else {
+        setError(errorMessage);
+        mfaForm.setFieldsValue({ code: '' });
+      }
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const renderCredentialsForm = () => (
+    <Form
+      form={credentialsForm}
+      name="login"
+      onFinish={handleCredentialsSubmit}
+      layout="vertical"
+      autoComplete="off"
+    >
+      <Form.Item
+        name="username"
+        rules={[
+          { required: true, message: 'Please enter your username!' },
+          { min: 3, message: 'Username must be at least 3 characters!' },
+        ]}
+      >
+        <Input prefix={<UserOutlined />} placeholder="Username" autoComplete="username" />
+      </Form.Item>
+
+      <Form.Item
+        name="password"
+        rules={[
+          { required: true, message: 'Please enter your password!' },
+          { min: 6, message: 'Password must be at least 6 characters!' },
+        ]}
+      >
+        <Input.Password
+          prefix={<LockOutlined />}
+          placeholder="Password"
+          autoComplete="current-password"
+        />
+      </Form.Item>
+
+      <Form.Item style={{ marginBottom: 0 }}>
+        <Button
+          type="primary"
+          htmlType="submit"
+          loading={loading}
+          block
+          className="login-button"
+        >
+          {loading ? 'Signing in...' : 'Sign In'}
+        </Button>
+      </Form.Item>
+    </Form>
+  );
+
+  const renderMfaForm = () => {
+    const minutes = Math.floor(mfaCountdown / 60);
+    const seconds = String(mfaCountdown % 60).padStart(2, '0');
+    return (
+      <Form form={mfaForm} name="mfa" onFinish={handleMfaSubmit} layout="vertical" autoComplete="off">
+        <Alert
+          message="Multi-Factor Authentication"
+          description={
+            <span>
+              Enter the 6-digit code from your authenticator app, or use a backup code
+              (format: <code>XXXX-YYYY</code>).
+              {mfaCountdown > 0 && (
+                <>
+                  {' '}Session expires in <strong>{minutes}:{seconds}</strong>.
+                </>
+              )}
+            </span>
+          }
+          type="info"
+          showIcon
+          style={{ marginBottom: 16 }}
+        />
+        <Form.Item
+          name="code"
+          rules={[
+            { required: true, message: 'Please enter your MFA code.' },
+            { min: 6, message: 'Code must be at least 6 characters.' },
+            { max: 10, message: 'Code is too long.' },
+          ]}
+        >
+          <Input
+            prefix={<SafetyCertificateOutlined />}
+            placeholder="123456 or XXXX-YYYY"
+            autoComplete="one-time-code"
+            inputMode="text"
+            maxLength={10}
+            autoFocus
+          />
+        </Form.Item>
+        <Form.Item style={{ marginBottom: 8 }}>
+          <Button
+            type="primary"
+            htmlType="submit"
+            loading={loading}
+            block
+            className="login-button"
+          >
+            {loading ? 'Verifying...' : 'Verify'}
+          </Button>
+        </Form.Item>
+        <Form.Item style={{ marginBottom: 0 }}>
+          <Button
+            type="default"
+            block
+            onClick={() => resetToCredentials('')}
+            disabled={loading}
+          >
+            Use a different account
+          </Button>
+        </Form.Item>
+      </Form>
+    );
+  };
+
   return (
     <div className="login-container">
-      <Row 
-        justify="center" 
-        align="middle" 
-        style={{ 
+      <Row
+        justify="center"
+        align="middle"
+        style={{
           minHeight: '100vh',
           minHeight: '100dvh',
           width: '100%',
-          margin: 0
+          margin: 0,
         }}
       >
-        <Col 
-          xs={24} 
-          sm={20} 
-          md={16} 
-          lg={12} 
-          xl={10} 
+        <Col
+          xs={24}
+          sm={20}
+          md={16}
+          lg={12}
+          xl={10}
           xxl={8}
-          style={{ 
+          style={{
             display: 'flex',
             justifyContent: 'center',
-            padding: '0 8px'
+            padding: '0 8px',
           }}
         >
           <Card className="login-card">
@@ -118,56 +301,13 @@ const Login = () => {
               />
             )}
 
-            <Form
-              form={form}
-              name="login"
-              onFinish={handleSubmit}
-              layout="vertical"
-              autoComplete="off"
-            >
-              <Form.Item
-                name="username"
-                rules={[
-                  { required: true, message: 'Please enter your username!' },
-                  { min: 3, message: 'Username must be at least 3 characters!' }
-                ]}
-              >
-                <Input
-                  prefix={<UserOutlined />}
-                  placeholder="Username"
-                  autoComplete="username"
-                />
-              </Form.Item>
-
-              <Form.Item
-                name="password"
-                rules={[
-                  { required: true, message: 'Please enter your password!' },
-                  { min: 6, message: 'Password must be at least 6 characters!' }
-                ]}
-              >
-                <Input.Password
-                  prefix={<LockOutlined />}
-                  placeholder="Password"
-                  autoComplete="current-password"
-                />
-              </Form.Item>
-
-              <Form.Item style={{ marginBottom: 0 }}>
-                <Button
-                  type="primary"
-                  htmlType="submit"
-                  loading={loading}
-                  block
-                  className="login-button"
-                >
-                  {loading ? 'Signing in...' : 'Sign In'}
-                </Button>
-              </Form.Item>
-            </Form>
+            {phase === PHASE_MFA ? renderMfaForm() : renderCredentialsForm()}
 
             <div className="login-footer">
-              <Text type="secondary" style={{ fontSize: 12, display: 'block', textAlign: 'center' }}>
+              <Text
+                type="secondary"
+                style={{ fontSize: 12, display: 'block', textAlign: 'center' }}
+              >
                 Centralized management for multiple HAProxy clusters
               </Text>
             </div>
@@ -178,4 +318,4 @@ const Login = () => {
   );
 };
 
-export default Login; 
+export default Login;

frontend/src/components/MFAEnrollModal.js

@@ -0,0 +1,306 @@
+import React, { useEffect, useRef, useState } from 'react';
+import {
+  Modal,
+  Steps,
+  Form,
+  Input,
+  Button,
+  Alert,
+  Typography,
+  Space,
+  Checkbox,
+  message,
+} from 'antd';
+import { QRCodeSVG } from 'qrcode.react';
+import axios from 'axios';
+import { extractApiError } from '../utils/apiError';
+
+const { Text, Paragraph } = Typography;
+
+const STEP_SETUP = 0;
+const STEP_VERIFY = 1;
+const STEP_BACKUP = 2;
+
+/**
+ * MFA enrollment wizard. Strictly modal-controlled: the modal cannot be
+ * dismissed via the X / mask in step 2/3 — backup codes are shown only once
+ * and the server-side pending row is opaque after enrollment confirms.
+ */
+const MFAEnrollModal = ({ open, onClose, onEnrolled }) => {
+  const [verifyForm] = Form.useForm();
+  const [step, setStep] = useState(STEP_SETUP);
+  const [loading, setLoading] = useState(false);
+  const [error, setError] = useState('');
+  const [otpauthUri, setOtpauthUri] = useState('');
+  const [secret, setSecret] = useState('');
+  const [backupCodes, setBackupCodes] = useState([]);
+  const [savedAcknowledged, setSavedAcknowledged] = useState(false);
+  const startedRef = useRef(false);
+
+  useEffect(() => {
+    if (!open) return undefined;
+    if (startedRef.current) return undefined;
+    startedRef.current = true;
+    startEnrollment();
+    return () => {
+      // No-op: cleanup happens via the explicit handleClose path.
+    };
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+  }, [open]);
+
+  const resetState = () => {
+    setStep(STEP_SETUP);
+    setLoading(false);
+    setError('');
+    setOtpauthUri('');
+    setSecret('');
+    setBackupCodes([]);
+    setSavedAcknowledged(false);
+    startedRef.current = false;
+    verifyForm.resetFields();
+  };
+
+  const startEnrollment = async () => {
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post('/api/mfa/enroll/start', {});
+      setOtpauthUri(response.data.otpauth_uri);
+      setSecret(response.data.secret);
+    } catch (err) {
+      setError(extractApiError(err, 'Could not start MFA enrollment.'));
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const handleVerify = async (values) => {
+    setLoading(true);
+    setError('');
+    try {
+      const response = await axios.post('/api/mfa/enroll/confirm', {
+        code: (values.code || '').trim(),
+      });
+      setBackupCodes(response.data.backup_codes || []);
+      setStep(STEP_BACKUP);
+    } catch (err) {
+      setError(extractApiError(err, 'Verification failed.'));
+      verifyForm.setFieldsValue({ code: '' });
+    } finally {
+      setLoading(false);
+    }
+  };
+
+  const handleCopyAll = () => {
+    const text = backupCodes.join('\n');
+    if (navigator.clipboard && navigator.clipboard.writeText) {
+      navigator.clipboard.writeText(text).then(
+        () => message.success('Backup codes copied to clipboard'),
+        () => message.error('Could not copy. Please copy manually.'),
+      );
+    } else {
+      message.warning('Clipboard API unavailable. Please copy manually.');
+    }
+  };
+
+  const handleDownload = () => {
+    const blob = new Blob(
+      [
+        'HAProxy OpenManager — MFA backup codes\n',
+        'Generated: ' + new Date().toISOString() + '\n',
+        'Each code is single-use. Store them somewhere safe and offline.\n\n',
+        ...backupCodes.map((c) => c + '\n'),
+      ],
+      { type: 'text/plain;charset=utf-8' },
+    );
+    const url = URL.createObjectURL(blob);
+    const link = document.createElement('a');
+    link.href = url;
+    link.download = 'haproxy-openmanager-mfa-backup-codes.txt';
+    document.body.appendChild(link);
+    link.click();
+    document.body.removeChild(link);
+    URL.revokeObjectURL(url);
+  };
+
+  const handleClose = (force = false) => {
+    if (step === STEP_BACKUP && !savedAcknowledged && !force) return;
+    resetState();
+    if (step === STEP_BACKUP) {
+      if (typeof onEnrolled === 'function') onEnrolled();
+    } else if (typeof onClose === 'function') {
+      onClose();
+    }
+  };
+
+  const renderSetup = () => (
+    <Space direction="vertical" size="middle" style={{ width: '100%' }}>
+      <Paragraph>
+        Open your authenticator app (Google Authenticator, Authy, 1Password, Microsoft
+        Authenticator) and scan this QR code, or enter the secret manually.
+      </Paragraph>
+      <div style={{ display: 'flex', justifyContent: 'center' }}>
+        {otpauthUri ? (
+          <QRCodeSVG value={otpauthUri} size={220} level="M" includeMargin />
+        ) : (
+          <Text type="secondary">Generating…</Text>
+        )}
+      </div>
+      {secret && (
+        <Alert
+          message="Trouble scanning?"
+          description={
+            <Space direction="vertical" size={4}>
+              <Text>Enter this secret manually in your authenticator app:</Text>
+              <Text code copyable={{ text: secret }} style={{ fontSize: 16 }}>
+                {secret}
+              </Text>
+            </Space>
+          }
+          type="info"
+          showIcon
+        />
+      )}
+      <div style={{ textAlign: 'right' }}>
+        <Space>
+          <Button onClick={() => handleClose(true)}>Cancel</Button>
+          <Button
+            type="primary"
+            onClick={() => setStep(STEP_VERIFY)}
+            disabled={!otpauthUri}
+          >
+            I&apos;ve added the account
+          </Button>
+        </Space>
+      </div>
+    </Space>
+  );
+
+  const renderVerify = () => (
+    <Form form={verifyForm} layout="vertical" onFinish={handleVerify}>
+      <Alert
+        message="Verify your authenticator"
+        description="Enter the 6-digit code displayed by your authenticator app. You have 5 attempts before the enrollment is invalidated and you'll need to start over."
+        type="info"
+        showIcon
+        style={{ marginBottom: 16 }}
+      />
+      <Form.Item
+        name="code"
+        label="Authenticator code"
+        rules={[
+          { required: true, message: 'Please enter the 6-digit code.' },
+          { len: 6, message: 'Code must be exactly 6 digits.' },
+        ]}
+      >
+        <Input
+          placeholder="123456"
+          autoComplete="one-time-code"
+          inputMode="numeric"
+          maxLength={6}
+          autoFocus
+        />
+      </Form.Item>
+      <div style={{ textAlign: 'right' }}>
+        <Space>
+          <Button onClick={() => setStep(STEP_SETUP)} disabled={loading}>
+            Back
+          </Button>
+          <Button onClick={() => handleClose(true)} disabled={loading}>
+            Cancel
+          </Button>
+          <Button type="primary" htmlType="submit" loading={loading}>
+            Verify
+          </Button>
+        </Space>
+      </div>
+    </Form>
+  );
+
+  const renderBackup = () => (
+    <Space direction="vertical" size="middle" style={{ width: '100%' }}>
+      <Alert
+        message="Save your backup codes now"
+        description={
+          <>
+            Each code can be used <strong>once</strong> when you can&apos;t access your
+            authenticator. <strong>They won&apos;t be shown again.</strong> If you lose
+            them, ask an administrator to reset your MFA.
+          </>
+        }
+        type="warning"
+        showIcon
+      />
+      <div
+        style={{
+          display: 'grid',
+          gridTemplateColumns: '1fr 1fr',
+          gap: '8px 16px',
+          padding: '12px',
+          backgroundColor: 'var(--ant-color-fill-quaternary, #fafafa)',
+          borderRadius: 6,
+        }}
+      >
+        {backupCodes.map((code) => (
+          <Text key={code} code style={{ fontSize: 15, letterSpacing: 1 }}>
+            {code}
+          </Text>
+        ))}
+      </div>
+      <Space>
+        <Button onClick={handleCopyAll}>Copy all</Button>
+        <Button onClick={handleDownload}>Download .txt</Button>
+      </Space>
+      <Checkbox
+        checked={savedAcknowledged}
+        onChange={(e) => setSavedAcknowledged(e.target.checked)}
+      >
+        I have saved my backup codes somewhere safe.
+      </Checkbox>
+      <div style={{ textAlign: 'right' }}>
+        <Button
+          type="primary"
+          disabled={!savedAcknowledged}
+          onClick={() => handleClose(false)}
+        >
+          Close
+        </Button>
+      </div>
+    </Space>
+  );
+
+  return (
+    <Modal
+      open={open}
+      title="Enable Multi-Factor Authentication"
+      width={520}
+      footer={null}
+      closable={false}
+      maskClosable={false}
+      destroyOnClose
+      keyboard={false}
+    >
+      <Steps
+        size="small"
+        current={step}
+        items={[{ title: 'Set up' }, { title: 'Verify' }, { title: 'Backup codes' }]}
+        style={{ marginBottom: 24 }}
+      />
+      {error && (
+        <Alert
+          message={error}
+          type="error"
+          showIcon
+          closable
+          onClose={() => setError('')}
+          style={{ marginBottom: 16 }}
+        />
+      )}
+      {step === STEP_SETUP && renderSetup()}
+      {step === STEP_VERIFY && renderVerify()}
+      {step === STEP_BACKUP && renderBackup()}
+    </Modal>
+  );
+};
+
+export default MFAEnrollModal;

frontend/src/components/Settings.js

@@ -212,6 +212,7 @@ const Settings = () => {
                 eab_kid: '',
                 eab_hmac_key: '',
                 challenge_backend_url: '',
+                dns01_enabled: false,
               }}
             >
               <Form.Item name="provider" label="ACME Provider">
@@ -308,6 +309,36 @@ const Settings = () => {
                 }]}
               />
 
+              {/* Issue #35: DNS-01 challenge support (global kill-switch). Per-account DNS provider
+                  credentials are configured on each ACME account in ACME Automation. */}
+              <Collapse
+                ghost
+                style={{ marginTop: 16 }}
+                items={[{
+                  key: 'dns01',
+                  label: 'DNS-01 Challenge (Advanced)',
+                  children: (
+                    <>
+                      <Alert
+                        type="info"
+                        showIcon
+                        style={{ marginBottom: 16 }}
+                        message="DNS-01 validates certificates via a DNS TXT record instead of HTTP on port 80."
+                        description="Use it for internal/isolated clusters with no public inbound port 80, or for wildcard certificates. When enabled, choose DNS-01 and a DNS provider per ACME account in ACME Automation. Leaving this off keeps the default HTTP-01 behavior unchanged."
+                      />
+                      <Form.Item
+                        name="dns01_enabled"
+                        label="Enable DNS-01 Challenge"
+                        valuePropName="checked"
+                        tooltip="Master switch. While off, DNS-01 options are hidden and no DNS-01 orders can be created."
+                      >
+                        <Switch />
+                      </Form.Item>
+                    </>
+                  ),
+                }]}
+              />
+
               <div style={{ marginTop: 24, display: 'flex', gap: 12 }}>
                 <Button type="primary" htmlType="submit" loading={acmeSaving}>
                   Save ACME Settings

frontend/src/components/UserManagement.js

@@ -32,11 +32,15 @@ import {
   HistoryOutlined,
   UserAddOutlined,
   KeyOutlined,
-  DownloadOutlined
+  DownloadOutlined,
+  SafetyCertificateOutlined,
+  ReloadOutlined,
+  InfoCircleOutlined,
 } from '@ant-design/icons';
 import axios from 'axios';
 import { useAuth } from '../contexts/AuthContext';
 import { extractApiError } from '../utils/apiError';
+import MFAEnrollModal from './MFAEnrollModal';
 
 const { TabPane } = Tabs;
 const { Option } = Select;
@@ -142,6 +146,17 @@ const PERMISSION_TREE = [
       { title: 'View Cluster Config', key: 'clusters.config' }
     ]
   },
+  {
+    title: '🛰️ HA / VIP Management',
+    key: 'vip',
+    children: [
+      { title: 'View VIPs', key: 'vip.read' },
+      { title: 'Create VIP', key: 'vip.create' },
+      { title: 'Edit VIP', key: 'vip.update' },
+      { title: 'Delete VIP', key: 'vip.delete' },
+      { title: 'Apply / Deploy VIP', key: 'vip.apply' }
+    ]
+  },
   {
     title: '📝 Configuration',
     key: 'config',
@@ -225,8 +240,16 @@ const PERMISSION_TREE = [
 ];
 
 const UserManagement = () => {
-  const { isAdmin } = useAuth(); // Get admin status from auth context
+  const { isAdmin, user: currentUser } = useAuth(); // Get admin status from auth context
   const { token } = theme.useToken();
+
+  // MFA UI state (Issue #18, v1.6.0)
+  const [mfaEnrollOpen, setMfaEnrollOpen] = useState(false);
+  const [mfaDisableOpen, setMfaDisableOpen] = useState(false);
+  const [mfaDisableForm] = Form.useForm();
+  const [mfaActionLoading, setMfaActionLoading] = useState(false);
+  const [mfaResetTarget, setMfaResetTarget] = useState(null);
+  const [mfaResetForm] = Form.useForm();
   const [activeTab, setActiveTab] = useState('users');
   
   // Users state
@@ -525,6 +548,47 @@ const UserManagement = () => {
     }
   };
 
+  // ---- MFA handlers (Issue #18, v1.6.0) -----------------------------------
+
+  const handleMfaEnrolled = () => {
+    setMfaEnrollOpen(false);
+    message.success('MFA enabled for your account.');
+    fetchUsers();
+  };
+
+  const handleMfaDisableSubmit = async (values) => {
+    setMfaActionLoading(true);
+    try {
+      await axios.post('/api/mfa/disable', { code: (values.code || '').trim() });
+      message.success('MFA disabled for your account.');
+      setMfaDisableOpen(false);
+      mfaDisableForm.resetFields();
+      fetchUsers();
+    } catch (error) {
+      message.error(extractApiError(error, 'Failed to disable MFA'));
+    } finally {
+      setMfaActionLoading(false);
+    }
+  };
+
+  const handleMfaAdminResetSubmit = async (values) => {
+    if (!mfaResetTarget) return;
+    setMfaActionLoading(true);
+    try {
+      await axios.post(`/api/mfa/admin-reset/${mfaResetTarget.id}`, {
+        reason: (values.reason || '').trim(),
+      });
+      message.success(`MFA reset for user '${mfaResetTarget.username}'.`);
+      setMfaResetTarget(null);
+      mfaResetForm.resetFields();
+      fetchUsers();
+    } catch (error) {
+      message.error(extractApiError(error, 'Failed to reset MFA'));
+    } finally {
+      setMfaActionLoading(false);
+    }
+  };
+
   const handleChangePassword = (user) => {
     setSelectedUser(user);
     passwordForm.resetFields();
@@ -717,6 +781,15 @@ const UserManagement = () => {
         </Space>
       )
     },
+    {
+      title: 'MFA',
+      dataIndex: 'mfa_enabled',
+      key: 'mfa_enabled',
+      width: 90,
+      render: (enabled) => enabled
+        ? <Tag icon={<SafetyCertificateOutlined />} color="green">ON</Tag>
+        : <Tag color="default">OFF</Tag>
+    },
     {
       title: 'Last Login',
       dataIndex: 'last_login_at',
@@ -726,31 +799,94 @@ const UserManagement = () => {
     {
       title: 'Actions',
       key: 'actions',
-      render: (_, record) => (
-        <Space>
-          {isAdmin() && (
-            <>
-              <Tooltip title="Edit User">
-                <Button 
-                  icon={<EditOutlined />} 
-                  size="small" 
-                  onClick={() => handleEditUser(record)}
+      render: (_, record) => {
+        const isSelf = currentUser && record.id === currentUser.id;
+        return (
+          <Space>
+            {isAdmin() && (
+              <>
+                <Tooltip title="Edit User">
+                  <Button
+                    icon={<EditOutlined />}
+                    size="small"
+                    onClick={() => handleEditUser(record)}
+                  />
+                </Tooltip>
+                <Tooltip title="Assign Roles">
+                  <Button
+                    icon={<TeamOutlined />}
+                    size="small"
+                    onClick={() => handleAssignRoles(record)}
+                  />
+                </Tooltip>
+                <Tooltip title="Change Password">
+                  <Button
+                    icon={<KeyOutlined />}
+                    size="small"
+                    onClick={() => handleChangePassword(record)}
+                  />
+                </Tooltip>
+              </>
+            )}
+            {isSelf && !record.mfa_enabled && (
+              <Tooltip title="Enable MFA for your account">
+                <Button
+                  icon={<SafetyCertificateOutlined />}
+                  size="small"
+                  type="primary"
+                  ghost
+                  onClick={() => setMfaEnrollOpen(true)}
+                >
+                  Enable MFA
+                </Button>
+              </Tooltip>
+            )}
+            {isSelf && record.mfa_enabled && (
+              <Tooltip title="Disable MFA for your account">
+                <Button
+                  icon={<SafetyCertificateOutlined />}
+                  size="small"
+                  onClick={() => {
+                    mfaDisableForm.resetFields();
+                    setMfaDisableOpen(true);
+                  }}
+                >
+                  Disable MFA
+                </Button>
+              </Tooltip>
+            )}
+            {isAdmin() && !isSelf && record.mfa_enabled && (
+              <Tooltip title="Reset this user's MFA (admin only)">
+                <Button
+                  icon={<ReloadOutlined />}
+                  size="small"
+                  danger
+                  onClick={() => {
+                    mfaResetForm.resetFields();
+                    setMfaResetTarget(record);
+                  }}
                 />
               </Tooltip>
-              <Tooltip title="Assign Roles">
-                <Button 
-                  icon={<TeamOutlined />} 
-                  size="small" 
-                  onClick={() => handleAssignRoles(record)}
-                />
-              </Tooltip>
-              <Tooltip title="Change Password">
-                <Button 
-                  icon={<KeyOutlined />} 
-                  size="small" 
-                  onClick={() => handleChangePassword(record)}
+            )}
+            {isAdmin() && !isSelf && !record.mfa_enabled && (
+              <Tooltip
+                title={
+                  <span>
+                    Only this user can enable their own MFA — the TOTP secret
+                    must be set up from their device. Ask them to sign in and
+                    click <b>Enable MFA</b> in their own row.
+                  </span>
+                }
+              >
+                <Button
+                  icon={<InfoCircleOutlined />}
+                  size="small"
+                  type="text"
+                  style={{ color: '#8c8c8c' }}
                 />
               </Tooltip>
+            )}
+            {isAdmin() && (
               <Popconfirm
                 title="Are you sure you want to delete this user?"
                 onConfirm={() => handleDeleteUser(record)}
@@ -758,20 +894,18 @@ const UserManagement = () => {
                 cancelText="No"
               >
                 <Tooltip title="Delete User">
-                  <Button 
-                    icon={<DeleteOutlined />} 
-                    danger 
+                  <Button
+                    icon={<DeleteOutlined />}
+                    danger
                     size="small"
                   />
                 </Tooltip>
               </Popconfirm>
-            </>
-          )}
-          {!isAdmin() && (
-            <Text type="secondary">View Only</Text>
-          )}
-        </Space>
-      )
+            )}
+            {!isAdmin() && !isSelf && <Text type="secondary">View Only</Text>}
+          </Space>
+        );
+      }
     }
   ];
 
@@ -1512,6 +1646,93 @@ const UserManagement = () => {
           </Form.Item>
         </Form>
       </Modal>
+
+      {/* MFA enrollment wizard (self) */}
+      <MFAEnrollModal
+        open={mfaEnrollOpen}
+        onClose={() => setMfaEnrollOpen(false)}
+        onEnrolled={handleMfaEnrolled}
+      />
+
+      {/* MFA self-disable modal */}
+      <Modal
+        title="Disable Multi-Factor Authentication"
+        open={mfaDisableOpen}
+        onCancel={() => setMfaDisableOpen(false)}
+        footer={null}
+        destroyOnClose
+      >
+        <Form form={mfaDisableForm} layout="vertical" onFinish={handleMfaDisableSubmit}>
+          <p>
+            Enter a current code from your authenticator app, or one of your backup
+            codes (format: <code>XXXX-YYYY</code>), to disable MFA.
+          </p>
+          <Form.Item
+            name="code"
+            label="MFA code"
+            rules={[
+              { required: true, message: 'Please enter your MFA code.' },
+              { min: 6, message: 'Code is too short.' },
+              { max: 10, message: 'Code is too long.' },
+            ]}
+          >
+            <Input
+              placeholder="123456 or XXXX-YYYY"
+              autoComplete="one-time-code"
+              maxLength={10}
+              autoFocus
+            />
+          </Form.Item>
+          <Form.Item style={{ marginBottom: 0, textAlign: 'right' }}>
+            <Space>
+              <Button onClick={() => setMfaDisableOpen(false)} disabled={mfaActionLoading}>
+                Cancel
+              </Button>
+              <Button type="primary" danger htmlType="submit" loading={mfaActionLoading}>
+                Disable MFA
+              </Button>
+            </Space>
+          </Form.Item>
+        </Form>
+      </Modal>
+
+      {/* Admin: reset another user's MFA */}
+      <Modal
+        title={mfaResetTarget ? `Reset MFA for '${mfaResetTarget.username}'` : 'Reset MFA'}
+        open={!!mfaResetTarget}
+        onCancel={() => setMfaResetTarget(null)}
+        footer={null}
+        destroyOnClose
+      >
+        <Form form={mfaResetForm} layout="vertical" onFinish={handleMfaAdminResetSubmit}>
+          <p>
+            This will disable MFA and invalidate all backup codes for{' '}
+            <strong>{mfaResetTarget && mfaResetTarget.username}</strong>. The user must
+            re-enroll on their next login. The action is recorded in the audit log.
+          </p>
+          <Form.Item
+            name="reason"
+            label="Reason (required, will be audit-logged)"
+            rules={[
+              { required: true, message: 'Please provide a reason.' },
+              { min: 3, message: 'Reason is too short.' },
+              { max: 500, message: 'Reason is too long (max 500 characters).' },
+            ]}
+          >
+            <Input.TextArea rows={3} placeholder="e.g. Lost phone — verified identity via ticket #12345." />
+          </Form.Item>
+          <Form.Item style={{ marginBottom: 0, textAlign: 'right' }}>
+            <Space>
+              <Button onClick={() => setMfaResetTarget(null)} disabled={mfaActionLoading}>
+                Cancel
+              </Button>
+              <Button type="primary" danger htmlType="submit" loading={mfaActionLoading}>
+                Reset MFA
+              </Button>
+            </Space>
+          </Form.Item>
+        </Form>
+      </Modal>
     </div>
   );
 };

frontend/src/components/VIPManagement.js

@@ -0,0 +1,583 @@
+import React, { useState, useEffect, useCallback } from 'react';
+import {
+  Table, Button, Space, Modal, Form, Input, InputNumber, Select, Tag, message,
+  Switch, Typography, Card, Alert, Tooltip, Spin
+} from 'antd';
+import {
+  PlusOutlined, EditOutlined, DeleteOutlined, ReloadOutlined, WarningOutlined,
+  PlayCircleOutlined, SyncOutlined, CrownOutlined, ClockCircleOutlined, FileSearchOutlined,
+  InfoCircleOutlined
+} from '@ant-design/icons';
+import { useCluster } from '../contexts/ClusterContext';
+import { extractApiError } from '../utils/apiError';
+
+const { Option } = Select;
+const { Title, Text } = Typography;
+
+// Interfaces that are never sensible VIP carriers — hidden from the dropdown.
+const IFACE_HIDE = /^(lo|docker|veth|br-|cni|flannel|kube|virbr)/i;
+
+const authHeaders = () => ({
+  'Content-Type': 'application/json',
+  'Authorization': `Bearer ${localStorage.getItem('authToken') || ''}`,
+});
+
+// Convergence-aware status (issue #27 follow-up): like other entities, a VIP only reads
+// "live" once its member agents have actually deployed & acked — never the instant Apply
+// is clicked. Backend returns deploy_status; we fall back to last_config_status.
+const DEPLOY_STATUS = {
+  PENDING:   { color: 'orange',     label: 'PENDING',  tip: 'Staged change — review and Apply (or Reject) it from the Apply Management page.' },
+  PENDING_DELETE: { color: 'volcano', label: 'PENDING DELETE', tip: 'Deletion staged for approval — the VIP keeps running untouched until you APPROVE it in Apply Management. Reject to keep it. The node is not changed until approval.' },
+  DELETING:  { color: 'processing', label: 'DELETING', tip: 'Deletion approved — member node(s) are stopping keepalived and releasing the VIP. Disappears once every node has torn down.' },
+  DELETED:   { color: 'default',    label: 'DELETED',  tip: 'All member nodes have torn keepalived down.' },
+  SYNCING:   { color: 'processing', label: 'SYNCING',  tip: 'Applied — an online member node is installing/configuring keepalived and will acknowledge on its next poll (~2–3 min).' },
+  AWAITING:  { color: 'gold',       label: 'AWAITING AGENT', tip: 'Applied, but the member node(s) that still need it are OFFLINE, so nothing can deploy yet. Bring the node\'s agent online — it converges on its next poll. (Not a hang.)' },
+  ACTIVE:    { color: 'green',      label: 'ACTIVE',   tip: 'Applied and every member node has deployed keepalived and acknowledged the current config.' },
+  ERROR:     { color: 'red',        label: 'ERROR',    tip: 'A member node failed to deploy keepalived — see Members / Live state for the node, then check that agent.' },
+  ATTENTION: { color: 'gold',       label: 'ATTENTION',tip: 'A member already runs a hand-managed keepalived; the agent left it untouched (externally managed). Resolve it on that node or remove it from the VIP.' },
+  APPLIED:   { color: 'green',      label: 'APPLIED',  tip: 'Applied.' },
+};
+
+// agents.capabilities / network_interfaces come from the API as JSONB → a JSON string
+// (asyncpg has no jsonb codec). Mirror AgentManagement.js and JSON.parse when needed.
+const parseArr = (v) => {
+  if (Array.isArray(v)) return v;
+  if (typeof v === 'string') { try { const p = JSON.parse(v); return Array.isArray(p) ? p : []; } catch { return []; } }
+  return [];
+};
+
+// This component uses raw fetch(), but extractApiError expects an axios-shaped error
+// (err.response.data). Read the fetch Response body and reuse the envelope-aware extractor
+// so backend messages — e.g. the 409 "node already in VIP X" — actually reach the user.
+const fetchApiError = async (res, fallback) => {
+  try { const data = await res.json(); return extractApiError({ response: { data } }, fallback); }
+  catch { return fallback; }
+};
+
+const VIPManagement = () => {
+  const { clusters } = useCluster();
+  const [vips, setVips] = useState([]);
+  const [loading, setLoading] = useState(false);
+  const [modalVisible, setModalVisible] = useState(false);
+  const [editing, setEditing] = useState(null);
+  const [selectedPoolId, setSelectedPoolId] = useState(null);
+  // One row per agent in the selected pool — the user toggles which participate.
+  const [memberRows, setMemberRows] = useState([]);
+  const [form] = Form.useForm();
+  // Delete confirmation (with opt-in package uninstall) + diagnostics modal state.
+  const [deleteTarget, setDeleteTarget] = useState(null);
+  const [showL2Note, setShowL2Note] = useState(false);
+  const [diagVip, setDiagVip] = useState(null);
+  const [diagData, setDiagData] = useState(null);
+  const [diagLoading, setDiagLoading] = useState(false);
+
+  // Distinct pools derived from the cluster list (cluster -> pool_id).
+  const pools = React.useMemo(() => {
+    const seen = new Map();
+    (clusters || []).forEach((c) => {
+      if (c.pool_id && !seen.has(c.pool_id)) seen.set(c.pool_id, c.name || `pool ${c.pool_id}`);
+    });
+    return Array.from(seen, ([id, name]) => ({ id, name }));
+  }, [clusters]);
+
+  const fetchVips = useCallback(async () => {
+    setLoading(true);
+    try {
+      const res = await fetch('/api/vip', { headers: authHeaders() });
+      if (res.ok) {
+        const data = await res.json();
+        setVips(data.vips || []);
+      } else if (res.status === 403) {
+        message.warning('You do not have permission to view VIPs (vip.read).');
+        setVips([]);
+      }
+    } catch (e) {
+      console.error('fetchVips failed', e);
+    } finally {
+      setLoading(false);
+    }
+  }, []);
+
+  useEffect(() => {
+    fetchVips();
+    const t = setInterval(fetchVips, 30000); // live MASTER/BACKUP via existing detection pipeline
+    return () => clearInterval(t);
+  }, [fetchVips]);
+
+  // Build the participating-nodes table from the pool's EXISTING agents (installed via the
+  // standard Agent Management process). On edit, pre-select the VIP's current members.
+  const buildMemberRows = (agents, existing) => {
+    const ex = {};
+    (existing || []).forEach((m) => { ex[m.agent_id] = m; });
+    return (agents || [])
+      .filter((a) => !String(a.name).startsWith('token_'))
+      .map((a) => {
+        const interfaces = parseArr(a.network_interfaces).filter((n) => !IFACE_HIDE.test(n));
+        const e = ex[a.id];
+        return {
+          agent_id: a.id,
+          agent_name: a.name,
+          ip_address: a.ip_address,
+          capable: parseArr(a.capabilities).includes('keepalived_management'),
+          interfaces,
+          participate: !!e,
+          role: e ? e.role : 'BACKUP',
+          priority: e ? e.priority : 100,
+          network_interface: e ? e.network_interface : (interfaces[0] || ''),
+        };
+      });
+  };
+
+  const loadPoolMembers = useCallback(async (poolId, existing) => {
+    if (!poolId) { setMemberRows([]); return; }
+    try {
+      const res = await fetch(`/api/agents?pool_id=${poolId}`, { headers: authHeaders() });
+      const data = res.ok ? await res.json() : { agents: [] };
+      setMemberRows(buildMemberRows(data.agents || [], existing));
+    } catch (e) {
+      console.error('loadPoolMembers failed', e);
+      setMemberRows([]);
+    }
+  }, []);
+
+  const setRow = (agentId, patch) =>
+    setMemberRows((rows) => rows.map((r) => (r.agent_id === agentId ? { ...r, ...patch } : r)));
+
+  // Toggling a node into the VIP: if no other participating node is MASTER yet, make this
+  // one the MASTER. This makes the single-node case work without the operator having to flip
+  // the role by hand (a one-node VIP's only node IS the master), and gives a sensible default
+  // for multi-node (first picked = master, the rest backup). Editing keeps stored roles.
+  const toggleParticipate = (agentId, on) =>
+    setMemberRows((rows) => {
+      const otherMaster = rows.some((r) => r.agent_id !== agentId && r.participate && r.role === 'MASTER');
+      return rows.map((r) => {
+        if (r.agent_id !== agentId) return r;
+        if (on && !otherMaster) return { ...r, participate: true, role: 'MASTER', priority: 150 };
+        return { ...r, participate: on };
+      });
+    });
+
+  const openCreate = () => {
+    setEditing(null);
+    setSelectedPoolId(null);
+    setMemberRows([]);
+    form.resetFields();
+    form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true });
+    setModalVisible(true);
+  };
+
+  const openEdit = (vip) => {
+    setEditing(vip);
+    setSelectedPoolId(vip.pool_id);
+    loadPoolMembers(vip.pool_id, vip.members);
+    form.resetFields();
+    form.setFieldsValue({
+      name: vip.name, description: vip.description, pool_id: vip.pool_id,
+      virtual_ip: vip.virtual_ip, prefix_length: vip.prefix_length,
+      virtual_router_id: vip.virtual_router_id, advert_int: vip.advert_int,
+      use_unicast: vip.use_unicast, track_haproxy: vip.track_haproxy,
+    });
+    setModalVisible(true);
+  };
+
+  const submit = async () => {
+    let values;
+    try { values = await form.validateFields(); }
+    catch { return; }
+
+    const chosen = memberRows.filter((r) => r.participate);
+    if (chosen.length < 1) { message.error('Select at least 1 participating node.'); return; }
+    const masters = chosen.filter((r) => r.role === 'MASTER');
+    if (masters.length !== 1) { message.error('Exactly one participating node must be MASTER.'); return; }
+    if (chosen.some((r) => !r.network_interface)) { message.error('Pick a network interface for every participating node.'); return; }
+    const maxBackup = Math.max(...chosen.filter((r) => r.role === 'BACKUP').map((r) => r.priority));
+    if (masters[0].priority <= maxBackup) { message.error('The MASTER must have a higher priority than every BACKUP.'); return; }
+
+    const body = {
+      ...values,
+      members: chosen.map((r) => ({
+        agent_id: r.agent_id, network_interface: r.network_interface, role: r.role, priority: r.priority,
+      })),
+    };
+    if (!body.auth_pass) delete body.auth_pass; // omit to keep existing on edit
+    try {
+      const url = editing ? `/api/vip/${editing.id}` : '/api/vip';
+      const res = await fetch(url, { method: editing ? 'PUT' : 'POST', headers: authHeaders(), body: JSON.stringify(body) });
+      if (res.ok) {
+        message.success(editing
+          ? 'VIP updated (PENDING) — apply it from the Apply Management page'
+          : 'VIP created (PENDING) — apply it from the Apply Management page');
+        setModalVisible(false);
+        fetchVips();
+      } else {
+        message.error(await fetchApiError(res, 'Failed to save VIP'));
+      }
+    } catch (e) {
+      message.error('Failed to save VIP: ' + e.message);
+    }
+  };
+
+  const deleteVip = async (vip, purge) => {
+    try {
+      const res = await fetch(`/api/vip/${vip.id}${purge ? '?purge_package=true' : ''}`,
+        { method: 'DELETE', headers: authHeaders() });
+      if (res.ok) {
+        let body = {};
+        try { body = await res.json(); } catch (_) { /* ignore */ }
+        // Backend returns staged=true (approval required, VIP still running) or staged=false
+        // (never-applied VIP removed at once). Surface its exact message either way.
+        (body.staged ? message.info : message.success)(
+          body.message || 'Deletion requested.');
+        setDeleteTarget(null); fetchVips();
+      } else message.error(await fetchApiError(res, 'Delete failed'));
+    } catch (e) { message.error('Delete failed: ' + e.message); }
+  };
+
+  // Diagnostics: per-member deploy state/ack from GET /api/vip/{id}/status — the live view
+  // of what each node reported (installing/applied/error/externally-managed), most useful
+  // while a freshly-applied VIP is SYNCING (keepalived install can take ~30s).
+  const openDiagnostics = async (vip) => {
+    setDiagVip(vip); setDiagData(null); setDiagLoading(true);
+    try {
+      const res = await fetch(`/api/vip/${vip.id}/status`, { headers: authHeaders() });
+      if (res.ok) setDiagData(await res.json());
+      else message.error(await fetchApiError(res, 'Failed to load diagnostics'));
+    } catch (e) { message.error('Diagnostics failed: ' + e.message); }
+    finally { setDiagLoading(false); }
+  };
+
+  // Colorful, IP-Inventory/Agent-consistent state: live VRRP MASTER (green, crowned) /
+  // BACKUP (orange) / FAULT (red); when the agent hasn't reported a live state yet, show
+  // the configured role as a dashed outline tag (same color) so it's clearly "intended,
+  // not yet observed". A node whose agent is too old gets an "awaiting agent" flag.
+  const renderMembers = (_, vip) => (
+    <Space direction="vertical" size={4}>
+      {(vip.members || []).map((m) => {
+        const live = m.keepalive_state && m.keepalive_state !== 'NONE' ? m.keepalive_state : null;
+        const roleColor = m.role === 'MASTER' ? 'green' : 'orange';
+        const awaiting = !live && !m.keepalived_capable;
+        return (
+          <Space key={m.agent_id} size={6}>
+            <Text style={{ fontSize: 12 }}>{m.agent_name || `agent ${m.agent_id}`}</Text>
+            {live ? (
+              <Tooltip title={`Live VRRP state: ${live}`}>
+                <Tag
+                  color={live === 'MASTER' ? 'green' : live === 'BACKUP' ? 'orange' : 'red'}
+                  icon={live === 'MASTER' ? <CrownOutlined /> : undefined}
+                  style={{ marginInlineEnd: 0, fontWeight: 600 }}
+                >
+                  {live}
+                </Tag>
+              </Tooltip>
+            ) : (
+              <Tooltip title="Configured role — live VRRP state not observed yet (agent offline or still converging).">
+                <Tag color={roleColor} style={{ marginInlineEnd: 0, borderStyle: 'dashed', opacity: 0.85 }}>
+                  {m.role}
+                </Tag>
+              </Tooltip>
+            )}
+            {awaiting && (
+              <Tooltip title="This node's agent does not advertise keepalived_management — upgrade the agent.">
+                <Tag color="gold" icon={<WarningOutlined />} style={{ marginInlineEnd: 0 }}>awaiting agent</Tag>
+              </Tooltip>
+            )}
+          </Space>
+        );
+      })}
+    </Space>
+  );
+
+  const columns = [
+    { title: 'Name', dataIndex: 'name', key: 'name' },
+    { title: 'Virtual IP', key: 'vip', render: (_, v) => <Text code>{v.virtual_ip}/{v.prefix_length}</Text> },
+    { title: 'Pool', dataIndex: 'pool_name', key: 'pool' },
+    { title: 'VRID', dataIndex: 'virtual_router_id', key: 'vrid' },
+    { title: 'Members / Live state', key: 'members', render: renderMembers },
+    {
+      title: 'Status', key: 'status', render: (_, v) => {
+        const s = v.deploy_status || v.last_config_status;
+        const d = DEPLOY_STATUS[s] || { color: 'default', label: s, tip: '' };
+        const count = (s === 'SYNCING' || s === 'AWAITING' || s === 'ACTIVE' || s === 'DELETING' || s === 'DELETED') && v.deploy_total
+          ? ` (${v.deploy_synced}/${v.deploy_total})` : '';
+        return (
+          <Tooltip title={d.tip}>
+            <Tag color={d.color} icon={(s === 'SYNCING' || s === 'DELETING') ? <SyncOutlined spin /> : s === 'AWAITING' ? <ClockCircleOutlined /> : undefined}>
+              {(d.label || s)}{count}
+            </Tag>
+          </Tooltip>
+        );
+      },
+    },
+    {
+      title: 'Actions', key: 'actions', render: (_, v) => (
+        <Space>
+          {v.last_config_status === 'PENDING' && (
+            <Tooltip title="Apply pending configuration changes">
+              <Button type="primary" size="small" icon={<PlayCircleOutlined />}
+                onClick={() => { window.location.href = '/apply-management'; }}
+                style={{ backgroundColor: '#1890ff', borderColor: '#1890ff' }}>
+                Apply
+              </Button>
+            </Tooltip>
+          )}
+          <Tooltip title="Edit VIP (changes become PENDING; apply from Apply Management)">
+            <Button size="small" icon={<EditOutlined />} onClick={() => openEdit(v)} />
+          </Tooltip>
+          <Tooltip title="Diagnostics — per-node keepalived deploy status & logs">
+            <Button size="small" icon={<FileSearchOutlined />} onClick={() => openDiagnostics(v)} />
+          </Tooltip>
+          <Tooltip title="Delete VIP">
+            <Button size="small" danger icon={<DeleteOutlined />}
+              onClick={() => setDeleteTarget(v)} />
+          </Tooltip>
+        </Space>
+      ),
+    },
+  ];
+
+  // Member-selection table inside the modal — the pool's installed agents (nodes).
+  const memberColumns = [
+    {
+      title: 'Node (agent)', key: 'node', render: (_, r) => (
+        <span>
+          <Text strong>{r.agent_name}</Text>{' '}
+          <Text type="secondary" style={{ fontSize: 12 }}>{r.ip_address ? `(${r.ip_address})` : '(no IP yet)'}</Text>
+          {!r.capable && (
+            <Tooltip title="This agent doesn't advertise keepalived_management — upgrade it or this node won't deploy.">
+              {' '}<Tag color="gold" icon={<WarningOutlined />}>agent too old</Tag>
+            </Tooltip>
+          )}
+        </span>
+      ),
+    },
+    {
+      title: 'Participate', key: 'participate', width: 100, render: (_, r) => (
+        <Switch checked={r.participate} onChange={(c) => toggleParticipate(r.agent_id, c)} />
+      ),
+    },
+    {
+      title: 'Role', key: 'role', width: 130, render: (_, r) => (
+        <Select size="small" style={{ width: 110 }} value={r.role} disabled={!r.participate}
+          onChange={(val) => setRow(r.agent_id, { role: val, priority: val === 'MASTER' ? 150 : 100 })}>
+          <Option value="MASTER">MASTER</Option>
+          <Option value="BACKUP">BACKUP</Option>
+        </Select>
+      ),
+    },
+    {
+      title: 'Priority', key: 'priority', width: 110, render: (_, r) => (
+        <InputNumber size="small" min={1} max={254} value={r.priority} disabled={!r.participate}
+          onChange={(val) => setRow(r.agent_id, { priority: val })} />
+      ),
+    },
+    {
+      title: 'Interface', key: 'iface', width: 160, render: (_, r) => (
+        <Select size="small" style={{ width: 140 }} value={r.network_interface || undefined}
+          placeholder="interface" disabled={!r.participate} showSearch
+          onChange={(val) => setRow(r.agent_id, { network_interface: val })}
+          notFoundContent="no interfaces reported"
+          options={(r.interfaces || []).map((n) => ({ label: n, value: n }))}
+          {...((r.interfaces || []).length === 0 ? { mode: 'tags' } : {})} />
+      ),
+    },
+  ];
+
+  return (
+    <div>
+      <Card>
+        <div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center', marginBottom: 16 }}>
+          <Title level={2} style={{ margin: 0 }}>HA / VIP (Keepalived)</Title>
+          <Space>
+            <Button icon={<ReloadOutlined />} onClick={fetchVips}>Refresh</Button>
+            <Button type="primary" icon={<PlusOutlined />} onClick={openCreate}>Create VIP</Button>
+          </Space>
+        </div>
+        {/* The cloud caveat is rarely relevant for the on-prem target audience, so it's a
+            subtle, collapsed-by-default info note (not a prominent yellow warning). */}
+        <div style={{ marginBottom: 12 }}>
+          <Button type="link" size="small" icon={<InfoCircleOutlined />} style={{ paddingLeft: 0 }}
+            onClick={() => setShowL2Note((v) => !v)}>
+            Network requirements (on-prem / L2)
+          </Button>
+          {showL2Note && (
+            <Alert
+              type="info" showIcon style={{ marginTop: 4 }}
+              message="On-prem / L2 networks"
+              description="VRRP-based VIP failover targets bare-metal / VMware / on-prem L2 segments. On AWS/Azure/GCP, cloud fabrics don't honor VRRP/gratuitous-ARP, so VIPs won't move. Ensure VRRP (IP protocol 112) is permitted by host firewalls."
+            />
+          )}
+        </div>
+        <Table rowKey="id" columns={columns} dataSource={vips} loading={loading} pagination={{ pageSize: 10 }} />
+      </Card>
+
+      <Modal
+        title={editing ? `Edit VIP — ${editing.name}` : 'Create VIP'}
+        open={modalVisible}
+        onCancel={() => setModalVisible(false)}
+        onOk={submit}
+        okText={editing ? 'Save (PENDING)' : 'Create (PENDING)'}
+        width={880}
+        destroyOnClose
+      >
+        <Form form={form} layout="vertical">
+          <Form.Item name="name" label="Name" rules={[{ required: true }]}>
+            <Input placeholder="web-vip" disabled={!!editing} />
+          </Form.Item>
+          <Form.Item name="description" label="Description">
+            <Input placeholder="optional" />
+          </Form.Item>
+          {!editing && (
+            <Form.Item name="pool_id" label="Pool" rules={[{ required: true }]}
+              tooltip="The VIP's nodes are the HAProxy servers (agents) already enrolled in this pool.">
+              <Select placeholder="Select a pool" onChange={(pid) => { setSelectedPoolId(pid); loadPoolMembers(pid); }}>
+                {pools.map((p) => <Option key={p.id} value={p.id}>{p.name}</Option>)}
+              </Select>
+            </Form.Item>
+          )}
+          <Space size="large" style={{ display: 'flex' }}>
+            <Form.Item name="virtual_ip" label="Virtual IP (IPv4)" rules={[{ required: true }]}>
+              <Input placeholder="10.0.0.100" />
+            </Form.Item>
+            <Form.Item name="prefix_length" label="Prefix" rules={[{ required: true }]}>
+              <InputNumber min={1} max={32} />
+            </Form.Item>
+            <Form.Item name="virtual_router_id" label="VRID (blank = auto)">
+              <InputNumber min={1} max={255} placeholder="auto" />
+            </Form.Item>
+            <Form.Item name="advert_int" label="Advert int (s)">
+              <InputNumber min={1} max={255} />
+            </Form.Item>
+          </Space>
+          <Space size="large">
+            <Form.Item name="use_unicast" label="Unicast VRRP" valuePropName="checked" tooltip="Recommended; works where multicast is blocked.">
+              <Switch />
+            </Form.Item>
+            <Form.Item name="track_haproxy" label="Fail over when HAProxy drops" valuePropName="checked">
+              <Switch />
+            </Form.Item>
+            <Form.Item name="auth_pass" label="VRRP secret (≤8 chars)">
+              <Input.Password placeholder={editing ? '•••• (unchanged)' : 'optional'} maxLength={8} />
+            </Form.Item>
+          </Space>
+
+          <Text strong>Participating nodes</Text>
+          <div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
+            These are the HAProxy servers (agents) already enrolled in this pool — toggle which join the VIP.
+            Pick <b>exactly one MASTER</b> (highest priority); the rest are BACKUP. A <b>single node</b> is allowed
+            (a keepalived-managed VIP <i>without</i> failover) — add a second node for real HA. Add new servers from
+            the standard Agent Management install flow.
+          </div>
+          <Alert
+            type="info" showIcon style={{ marginBottom: 8 }}
+            message="keepalived is installed automatically on Apply"
+            description={<>On Apply, any participating node that doesn’t already run keepalived will <b>install it from the node’s OS package repositories</b> (apt/dnf/yum/zypper/apk) — make sure the node can reach its repos (internet or an internal mirror). A node already running a <b>hand-managed</b> keepalived is left untouched (reported as “externally managed”).</>}
+          />
+          <Table
+            rowKey="agent_id"
+            size="small"
+            columns={memberColumns}
+            dataSource={memberRows}
+            pagination={false}
+            locale={{ emptyText: selectedPoolId ? 'No agents in this pool — install agents from Agent Management first.' : 'Select a pool to list its nodes.' }}
+          />
+        </Form>
+      </Modal>
+
+      {/* Delete confirmation with opt-in package uninstall. Enterprise-safe DEFAULT keeps the
+          package (just stop/disable + remove our config + release the VIP). */}
+      <Modal
+        title="Delete VIP — requires approval"
+        open={!!deleteTarget}
+        onCancel={() => setDeleteTarget(null)}
+        onOk={() => deleteVip(deleteTarget, false)}
+        okText="Stage deletion for approval"
+        okButtonProps={{ danger: true }}
+      >
+        {deleteTarget && (
+          <Space direction="vertical" size={12} style={{ width: '100%' }}>
+            <Alert
+              type="warning"
+              showIcon
+              message="This does NOT delete the VIP immediately"
+              description={<>It stages the deletion for approval. The VIP <b>keeps running, untouched</b>, on its
+                member node(s) until you <b>Approve</b> it on the <b>Apply Management</b> page — and you can
+                <b> Reject</b> it there to keep it. The node is changed <b>only after approval</b>, so an
+                accidental click can't tear down a production VIP.</>}
+            />
+            <Text>
+              Stage deletion of <Text strong>{deleteTarget.name}</Text> ({deleteTarget.virtual_ip}/{deleteTarget.prefix_length})?
+              When approved, the member node(s) <b>stop &amp; disable keepalived, remove the config we manage, and release the VIP</b>. The keepalived package itself is left installed, so re-adding a VIP later is instant.
+            </Text>
+          </Space>
+        )}
+      </Modal>
+
+      {/* Per-node keepalived deploy diagnostics + node-side log commands (esp. during SYNCING). */}
+      <Modal
+        title={diagVip ? `Diagnostics — ${diagVip.name}` : 'Diagnostics'}
+        open={!!diagVip}
+        onCancel={() => { setDiagVip(null); setDiagData(null); }}
+        width={780}
+        footer={[
+          <Button key="refresh" icon={<ReloadOutlined />} onClick={() => diagVip && openDiagnostics(diagVip)}>Refresh</Button>,
+          <Button key="close" type="primary" onClick={() => { setDiagVip(null); setDiagData(null); }}>Close</Button>,
+        ]}
+      >
+        {diagLoading && <div style={{ textAlign: 'center', padding: 24 }}><Spin /></div>}
+        {!diagLoading && diagData && (
+          <Space direction="vertical" size={12} style={{ width: '100%' }}>
+            <Text type="secondary">
+              Staging <Tag>{diagData.last_config_status}</Tag> — each node reports its deploy state after every poll; a fresh keepalived install can take ~30s.
+            </Text>
+            <Table
+              size="small" rowKey={(m) => m.agent_name} pagination={false}
+              dataSource={diagData.members || []}
+              columns={[
+                { title: 'Node', key: 'node', render: (_, m) => (
+                  <Space size={4}>
+                    <Text style={{ fontSize: 12 }}>{m.agent_name}</Text>
+                    {m.role === 'MASTER'
+                      ? <Tag color="green" icon={<CrownOutlined />} style={{ marginInlineEnd: 0 }}>MASTER</Tag>
+                      : <Tag color="orange" style={{ marginInlineEnd: 0 }}>BACKUP</Tag>}
+                  </Space>) },
+                { title: 'Agent', dataIndex: 'agent_status', key: 'agent',
+                  render: (s) => <Tag color={s === 'online' ? 'green' : 'red'}>{s || 'offline'}</Tag> },
+                { title: 'Live VRRP', dataIndex: 'keepalive_state', key: 'live',
+                  render: (s) => (s && s !== 'NONE')
+                    ? <Tag color={s === 'MASTER' ? 'green' : s === 'BACKUP' ? 'orange' : 'red'}>{s}</Tag>
+                    : <Text type="secondary">—</Text> },
+                { title: 'Deploy', key: 'deploy', render: (_, m) => {
+                    const st = m.deploy_state;
+                    const color = st === 'enabled' ? 'green' : st === 'error' ? 'red'
+                      : st === 'externally_managed' ? 'gold' : 'blue';
+                    return <Tooltip title={m.deploy_message || ''}><Tag color={color}>{m.convergence || st || 'pending'}</Tag></Tooltip>;
+                  } },
+                { title: 'Last ack', dataIndex: 'deploy_at', key: 'ack',
+                  render: (t) => t ? <Text style={{ fontSize: 11 }}>{new Date(t).toLocaleString()}</Text> : <Text type="secondary">—</Text> },
+              ]}
+            />
+            {(diagData.members || []).some((m) => m.deploy_message) && (
+              <Card size="small" title="Latest node messages" bodyStyle={{ padding: 8 }}>
+                {(diagData.members || []).filter((m) => m.deploy_message).map((m) => (
+                  <div key={m.agent_name} style={{ fontSize: 12 }}><Text strong>{m.agent_name}:</Text> {m.deploy_message}</div>
+                ))}
+              </Card>
+            )}
+            <Alert
+              type="info" showIcon
+              message="See the live install / VRRP logs on the node"
+              description={
+                <pre style={{ margin: 0, fontSize: 11, whiteSpace: 'pre-wrap' }}>{`systemctl status keepalived --no-pager
+journalctl -u keepalived --no-pager -n 50
+tail -n 100 /var/log/haproxy-agent/agent.log | grep -i keepalived`}</pre>
+              }
+            />
+          </Space>
+        )}
+        {!diagLoading && !diagData && <Text type="secondary">No diagnostics available.</Text>}
+      </Modal>
+    </div>
+  );
+};
+
+export default VIPManagement;

frontend/src/setupProxy.js

@@ -0,0 +1,31 @@
+/**
+ * CRA dev-server proxy.
+ *
+ * This file is consumed by Webpack DevServer (via `react-scripts start`)
+ * EXCLUSIVELY. It is NOT included in the production bundle (`npm run
+ * build` / `serve -s build`) and has zero effect in Kubernetes / Docker
+ * Compose deployments where nginx (`/api/*` → backend) handles routing.
+ *
+ * Default target is `http://localhost:8000` because the only realistic
+ * use of `npm start` is on a developer's host machine, where the backend
+ * is reachable on localhost. The target can be overridden with the
+ * `PROXY_TARGET` environment variable (e.g. `PROXY_TARGET=http://other-host:8000 npm start`).
+ *
+ * Why not `http://backend:8000`? That hostname only resolves inside the
+ * Docker Compose network. Defaulting to it caused all `/api/*` requests
+ * to fail with ENOTFOUND when running the dev-server on the host.
+ */
+const { createProxyMiddleware } = require('http-proxy-middleware');
+
+const target = process.env.PROXY_TARGET || 'http://localhost:8000';
+
+module.exports = function (app) {
+  app.use(
+    '/api',
+    createProxyMiddleware({
+      target,
+      changeOrigin: true,
+      logLevel: 'warn',
+    })
+  );
+};

frontend/src/utils/api.js

@@ -1,32 +1,28 @@
 /**
  * API Configuration
- * Centralized API URL management for the application
- * 
- * Environment Variables:
- * - REACT_APP_API_URL: Full API URL (e.g., https://api.example.com)
- * - NODE_ENV: Environment (development, production)
+ * Centralized API URL management for the application.
+ *
+ * Resolution strategy (same in dev and prod — keeps SPA same-origin):
+ *   1) REACT_APP_API_URL — explicit override at BUILD time. Use only when
+ *      the SPA must call a cross-origin API (CORS must be enabled there).
+ *   2) window.location.{protocol,host} — same-origin. In dev this routes
+ *      through CRA dev-server proxy (see frontend/src/setupProxy.js); in
+ *      prod through nginx ingress (`/api/*` → backend service).
+ *   3) Empty string — non-browser env (SSR/tests). Yields relative URLs.
+ *
+ * NOTE: Do NOT hardcode `http://localhost:8000` here. Even in unreachable
+ * branches CRA/Terser keeps string literals in the bundle, which would
+ * confuse anyone auditing the production artifact.
  */
-
-// Get API URL from environment or use default based on environment
 const getApiUrl = () => {
-  // Priority 1: Explicit environment variable
   if (process.env.REACT_APP_API_URL) {
     return process.env.REACT_APP_API_URL;
   }
-  
-  // Priority 2: Detect from window location (for production deployments)
-  // Use window.location.host (includes port) instead of hostname (excludes port)
-  // so non-standard ports like :8080 are preserved, preventing CORS issues
   if (typeof window !== 'undefined' && window.location) {
     const { protocol, host } = window.location;
-    
-    if (process.env.NODE_ENV === 'production') {
-      return `${protocol}//${host}`;
-    }
+    return `${protocol}//${host}`;
   }
-  
-  // Priority 3: Development default
-  return 'http://localhost:8000';
+  return '';
 };
 
 // API Base URL

k8s/manifests/03-secrets.yaml

@@ -23,6 +23,8 @@ metadata:
     app: haproxy-openmanager
     component: backend
 type: Opaque
-data:
-  # your-secret-key-change-this-in-production
-  SECRET_KEY: eW91ci1zZWNyZXQta2V5LWNoYW5nZS10aGlzLWluLXByb2R1Y3Rpb24= 
+# Both values are replaced at deploy time by the CI/CD pipeline (sed step)
+# before `kubectl apply` runs. Never commit real secrets — use the placeholders.
+stringData:
+  SECRET_KEY: secret_key_replace_me
+  MFA_ENCRYPTION_KEY: mfa_encryption_key_replace_me

k8s/manifests/07-configmaps.yaml

@@ -15,6 +15,33 @@ data:
   PUBLIC_URL: 'https://haproxy-openmanager.example.com'
   MANAGEMENT_BASE_URL: 'https://haproxy-openmanager.example.com'
 
+  # ---------------------------------------------------------------------------
+  # MFA rate-limits (slowapi format: "<count>/<second|minute|hour|day>").
+  # The shipped defaults assume the rate-limit key is per-USER (Bearer JWT)
+  # with a per-IP fallback. They are sized for thousands of concurrent
+  # operators in an org-wide MFA rollout. Pod restart required to apply.
+  #
+  # Defaults live in backend/middleware/mfa_rate_limits.py:
+  #   ENROLL_START='10/minute'      ENROLL_CONFIRM='10/minute'
+  #   DISABLE='10/minute'           REGENERATE_BACKUP_CODES='5/hour'
+  #   ADMIN_RESET='60/hour'         ADMIN_RESET_ALL='1/day'
+  # ---------------------------------------------------------------------------
+  # MFA_RATE_LIMIT_ENROLL_START: '10/minute'
+  # MFA_RATE_LIMIT_ENROLL_CONFIRM: '10/minute'
+  # MFA_RATE_LIMIT_DISABLE: '10/minute'
+  # MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES: '5/hour'
+  # MFA_RATE_LIMIT_ADMIN_RESET: '60/hour'
+  # MFA_RATE_LIMIT_ADMIN_RESET_ALL: '1/day'
+
+  # ---------------------------------------------------------------------------
+  # Trusted reverse-proxy CIDRs for X-Forwarded-For. When the request peer
+  # falls inside one of these CIDRs and the user is NOT authenticated, the
+  # first hop of XFF becomes the rate-limit bucket. Empty (default) disables
+  # XFF parsing entirely — the safe choice when the topology is unknown.
+  # Example (in-cluster service mesh + nginx ingress on the default ranges):
+  # ---------------------------------------------------------------------------
+  # MFA_TRUSTED_PROXY_CIDRS: '10.0.0.0/8,172.16.0.0/12,192.168.0.0/16'
+
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -25,9 +52,20 @@ metadata:
     app: haproxy-openmanager
     component: frontend
 data:
-  # Leave empty to use same-origin (window.location) in production
-  # For development, set to: 'http://localhost:8000'
-  REACT_APP_API_URL: ''
+  # NOTE — REACT_APP_* env vars are RUNTIME no-ops here.
+  # Create-React-App inlines REACT_APP_* values into the bundle at BUILD time
+  # (`npm run build`). The static bundle served by `serve -s build` does not
+  # consume runtime env vars, so anything set here has zero effect on
+  # window.location-based same-origin routing.
+  #
+  # The frontend always issues `/api/*` against window.location, which is
+  # routed by the ingress + nginx ConfigMap below to the backend service.
+  # No env vars are required for the frontend deployment.
+  #
+  # WARNING: Do NOT set REACT_APP_API_URL in your CI/CD pipeline either —
+  # if a value is supplied at build time it gets HARDCODED into the bundle
+  # and breaks every deployment that does not match that exact URL.
+  {}
 
 ---
 apiVersion: v1

k8s/manifests/10-nginx.yaml

@@ -22,7 +22,9 @@ spec:
       serviceAccountName: haproxy-openmanager-nginx
       containers:
       - name: nginx
-        image: nginx:alpine
+        # Pinned to a patched release for the nginx "poolslip" advisory
+        # (mainline <=1.31.0 affected; fixed in mainline 1.31.1+ / stable 1.30.2+).
+        image: nginx:1.31.1-alpine
         ports:
         - containerPort: 8080
           name: http

scripts/README.md

@@ -75,6 +75,35 @@ Test HAProxy stats parsing
 ### test-build.sh
 Run build tests for the project
 
+## 🔐 MFA Admin Scripts
+
+### admin-mfa-reset-all.sh
+**Purpose:** Emergency — disable Multi-Factor Authentication for **every** user
+in one call. Use only when there is a mass loss of authenticator devices /
+inherited platform without working operators (Issue #18, v1.6.0).
+
+**Usage:**
+```bash
+# Interactive prompts ask for the admin Bearer token + reason
+./scripts/admin-mfa-reset-all.sh
+
+# Non-interactive (still requires double confirmation typed at the keyboard)
+API_URL=https://hap.example.com \
+ADMIN_TOKEN=eyJhbGciOi... \
+  ./scripts/admin-mfa-reset-all.sh
+```
+
+**Features:**
+- Calls `POST /api/mfa/admin-reset-all` (requires `users.is_admin = TRUE`)
+- Double confirmation: type `yes`, then `RESET ALL MFA` exactly
+- Required reason is recorded in `user_activity_logs`
+  (`action='mfa.disabled.admin_bulk_reset'`)
+- Deletes every backup code and invalidates pending MFA challenges
+
+**Safety:**
+- Irreversible — all users must re-enroll MFA afterwards
+- All other authentication (password, JWT, roles) is unaffected
+
 ## 🚨 Emergency Use Cases
 
 **1. Cluster Migration/Cleanup:**
@@ -101,6 +130,12 @@ Run build tests for the project
 ./scripts/debug-agent-stats-function.sh
 ```
 
+**4. MFA Outage (mass lost authenticators):**
+```bash
+# Disable MFA for every user, then ask them to re-enroll
+./scripts/admin-mfa-reset-all.sh
+```
+
 ## ⚠️ Safety Notes
 
 - All cleanup scripts require admin authentication

scripts/admin-mfa-reset-all.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# scripts/admin-mfa-reset-all.sh
+# Issue #18 — v1.6.0 — Emergency: disable MFA for ALL users in one call.
+#
+# This is a break-glass tool. It calls POST /api/mfa/admin-reset-all on the
+# backend with a strict, double-confirmed body and writes the action into the
+# server's audit log (action=mfa.disabled.admin_bulk_reset). All users will be
+# able to log in with username/password alone afterwards and must re-enroll if
+# they want MFA again.
+#
+# Usage:
+#   ./scripts/admin-mfa-reset-all.sh
+#   API_URL=https://hap.example.com ADMIN_TOKEN=ey... ./scripts/admin-mfa-reset-all.sh
+#
+# Required: a JWT bearer token belonging to a user whose `users.is_admin = TRUE`.
+set -euo pipefail
+
+API_URL="${API_URL:-http://localhost:8000}"
+ADMIN_TOKEN="${ADMIN_TOKEN:-}"
+
+if [ -z "$ADMIN_TOKEN" ]; then
+  read -rsp "Admin Bearer token (user with is_admin=TRUE): " ADMIN_TOKEN
+  echo
+fi
+
+if [ -z "$ADMIN_TOKEN" ]; then
+  echo "ERROR: no admin token provided." >&2
+  exit 1
+fi
+
+cat <<'WARN'
+========================================================================
+  WARNING — IRREVERSIBLE BULK ACTION
+========================================================================
+  This will:
+    * set mfa_enabled = FALSE for every user
+    * delete every backup code
+    * invalidate all pending MFA login challenges and enrollments
+    * write a permanent entry to user_activity_logs
+  After this, all users can sign in with username/password only and must
+  re-enroll MFA from the Users page.
+========================================================================
+WARN
+
+read -rp "Type 'yes' to proceed: " confirm1
+if [ "$confirm1" != "yes" ]; then
+  echo "Aborted."
+  exit 1
+fi
+read -rp "Type 'RESET ALL MFA' (exact) to confirm: " confirm2
+if [ "$confirm2" != "RESET ALL MFA" ]; then
+  echo "Aborted."
+  exit 1
+fi
+read -rp "Reason (logged in audit): " reason
+if [ -z "$reason" ]; then
+  echo "Aborted: reason is required."
+  exit 1
+fi
+
+# Compose JSON safely (python3 for proper JSON escaping; reliably available
+# everywhere the backend already runs).
+payload=$(python3 -c "
+import json, sys
+print(json.dumps({'confirm': 'RESET ALL MFA', 'reason': sys.argv[1]}))
+" "$reason")
+
+echo "Calling $API_URL/api/mfa/admin-reset-all ..."
+http_response=$(curl -fsS -X POST "$API_URL/api/mfa/admin-reset-all" \
+  -H "Authorization: Bearer $ADMIN_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "$payload")
+
+echo "$http_response"
+echo
+echo "Done. Verify in user_activity_logs: action='mfa.disabled.admin_bulk_reset'."

version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.5.2",
-  "releaseName": "ACME Diagnostics Panel Hardening",
-  "releaseDate": "2026-05-13"
+  "version": "1.8.5",
+  "releaseName": "ACME completion-task SQL fix",
+  "releaseDate": "2026-07-03"
 }