perf(api): opt-in uvicorn workers + heartbeat query consolidation (v1.8.6, Issue #35)

A user running the API on a 2-core/4GB host reported slow-feeling API
responses (Issue #35 follow-up). Review of the hot paths found no
pathological defect; the dominant factors are the single uvicorn worker
(one core serves all requests) and the constant agent-poll baseline
(4 requests per agent every 30s). Two zero-risk improvements:

- backend/Dockerfile: CMD now honors UVICORN_WORKERS, falling back to
  WEB_CONCURRENCY and then 1. Flagless uvicorn natively honors
  WEB_CONCURRENCY, so the fallback keeps any deployment that relied on it
  byte-for-byte compatible; with the final default of 1 worker uvicorn runs
  in-process exactly as before. >1 enables the multiprocess supervisor so
  multi-core hosts can use all cores. Background tasks are already
  multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as exercised
  by the k8s HPA deployment (2-10 replicas). `exec` keeps uvicorn as PID 1
  (clean SIGTERM, verified ~1s docker stop with 2 workers).
  docker-compose.yml passes UVICORN_WORKERS through as empty-when-unset so a
  user-set WEB_CONCURRENCY is never overridden; .env.template documents it.

- routers/agent.py heartbeat (by-name endpoint): the agent's
  status/version/upgrade_status were read with three separate single-column
  SELECTs against the same row; now one SELECT. Identical values and None
  semantics (single consistent snapshot instead of three reads); saves two
  round-trips per heartbeat per agent every 30s. The legacy by-id heartbeat
  endpoint is untouched; the heartbeat API contract is unchanged for agents
  of every version.

- README: new "Performance Tuning" section (worker/replica scaling, and how
  to use the X-Response-Time header plus "Slow request detected" logs to
  pinpoint slow endpoints).

Verification: full backend suite in docker green (1063 passed, 151 skipped;
also re-run by the runtime image build); worker-count expansion matrix
(unset->1, UVICORN_WORKERS=2->2, WEB_CONCURRENCY=3->3, both->UVICORN_WORKERS,
empty->fallback) all correct; default run confirmed single-process with
uvicorn as PID 1 and healthy API; UVICORN_WORKERS=2 confirmed parent + 2
workers, healthy API, clean shutdown; live heartbeats verified for
register + existing-agent paths AND degraded agents (no stats socket /
haproxy stopped / garbage stats CSV / unknown backend in server_statuses):
all return 200, agent row updates correctly, zero backend errors.
No schema, API, or agent changes.

.env.template

@@ -59,6 +59,15 @@ AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
 # Config sync interval in seconds
 AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
 
+# ============================================================================
+# BACKEND PERFORMANCE
+# ============================================================================
+# Number of uvicorn worker processes for the backend API (default: 1).
+# On multi-core hosts, setting this to the core count (e.g. 2) lets the API
+# use all cores. Safe to increase: background tasks are multi-replica safe
+# (the k8s deployment already runs 2+ replicas via HPA).
+UVICORN_WORKERS=1
+
 # ============================================================================
 # CORS CONFIGURATION
 # ============================================================================

README.md

@@ -259,7 +259,7 @@ This architecture provides better security (no inbound connections to HAProxy se
 - **ACME Account Management**: Register, view, and deactivate ACME accounts from the UI
 - **Staging Mode**: Test certificate issuance with Let's Encrypt staging environment before production
 - **Custom Staging Endpoint** *(v1.4.0)*: Optional `staging_url_override` setting lets you point staging mode at a private ACME test CA (e.g. Pebble) without touching the production directory URL
-- **External Account Binding (EAB)**: Support for CAs that require EAB (ZeroSSL, Google Trust Services)
+- **External Account Binding (EAB)**: Support for CAs that require EAB (ZeroSSL, Google Trust Services). Enter the EAB Key ID and HMAC Key globally in Settings, or per-account in the Register Account dialog (a per-account value overrides the global setting; leave it blank to use the global one)
 - **Structured Error Diagnostics** *(v1.4.0)*: All ACME failures (challenge, finalize, download) persist structured JSON to `letsencrypt_orders.error_detail` for clear post-mortem analysis
 - **Audit Logging** *(v1.4.0)*: Every ACME operation (request, revoke, CA-chain import, account ops) is captured in `user_activity_logs` for compliance review
 - **ACME Diagnostic Panel** *(v1.5.0 — Issue #13)*: Live pre-flight + post-failure diagnostics (DNS / port-80 / routing / account / agents) and merged event timeline (`acme_order_events` + correlated `user_activity_logs`) accessible from the ACME Automation page; humanized error rendering for 11+ RFC8555 problem types with backwards-compatible fallback for legacy plain-string `error_detail`; per-user 5/min rate-limit
@@ -1738,6 +1738,19 @@ Add new HAProxy clusters through the web interface or directly via API:
 }
 ```
 
+### Performance Tuning *(v1.8.6)*
+
+The backend API defaults to a **single uvicorn worker process**, which uses one CPU core. Agents poll the API every 30 seconds (heartbeat, config, pending-requests, upgrade checks), so larger fleets add a constant baseline load. Two ways to scale:
+
+- **Docker Compose — worker processes**: set `UVICORN_WORKERS` in your `.env` (default `1`). On a multi-core host, matching the core count (e.g. `UVICORN_WORKERS=2` on a 2-core machine) lets the API use all cores:
+  ```bash
+  echo "UVICORN_WORKERS=2" >> .env && docker-compose up -d backend
+  ```
+- **Kubernetes/OpenShift — replicas**: the shipped manifests already include an HPA for the backend (2→10 replicas, `k8s/manifests/13-hpa.yaml`); raise `minReplicas`/`maxReplicas` as needed.
+
+Both are safe: all background tasks (ACME completion, renewals, agent monitoring) are multi-replica safe by design (atomic claims via `FOR UPDATE SKIP LOCKED`, PostgreSQL advisory locks).
+
+**Diagnosing slow requests**: every API response carries an `X-Response-Time` header, and the backend logs `Slow request detected` (WARNING) for any request taking longer than 1 second — check those log lines to pinpoint slow endpoints before tuning anything else.
 
 ## API Reference
 
@@ -2415,6 +2428,12 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.8.6** (2026-07-06) — **Performance: opt-in API workers + heartbeat micro-optimization** (Issue #35 follow-up): the backend container can now run multiple uvicorn worker processes via the new `UVICORN_WORKERS` environment variable (default **1** — behavior unchanged unless you opt in), letting the API use all cores on multi-core hosts; background tasks were already multi-replica safe, as exercised by the Kubernetes HPA deployment. The agent heartbeat handler now reads the agent's `status`/`version`/`upgrade_status` in one query instead of three (one round-trip per heartbeat, per agent, every 30s). Added a *Performance Tuning* section to the README (worker/replica scaling and how to use the `X-Response-Time` header and `Slow request detected` logs to pinpoint slow endpoints). Zero-risk release: no schema, API, or agent changes; defaults preserve existing behavior exactly.
+- **v1.8.5** (2026-07-03) — **ACME completion-task SQL fix** (Issue #35 follow-up): the background order-completion task (`complete_pending_acme_orders`, runs every 60s) died on **every cycle** with `syntax error at or near ")"` — an extra closing parenthesis introduced in v1.8.0's bounded DNS-01 retry claim query. Because that query is the task's first database call, **no background ACME work ran at all from v1.8.0 through v1.8.4**: orders were never claimed for finalize/download, the DNS-01 TXT record was never published (so DNS-01 with an automated provider such as Cloudflare could never validate), Site Wizard staged orders never left `wizard_staged`, and DNS-01 retry/TXT-cleanup never executed. The stray parenthesis is removed and a regression test now scans all ACME modules' SQL for unbalanced parentheses (the unit suite mocks the database, which is why a raw-SQL syntax error could slip through). One-line backend query fix; no schema, API, or agent changes — fully backward compatible.
+- **v1.8.4** (2026-06-27) — **Agent installer self-kill fix** (Issue #31): the Linux/macOS agent installer could abort during "pre-installation cleanup" (terminal showed `Killing processes matching: haproxy-agent` then `Killed`) when the install script's own filename contained "haproxy-agent". The cleanup killed processes by matching the bare string "haproxy-agent" against full command lines, which also matched the running installer (and a `sudo`/PAM ancestor the self-exclusion did not cover), so the installer terminated itself. Cleanup now targets only the installed agent (the `$INSTALL_DIR/haproxy-agent` binary and the agent service), never the bare string, and the UI now names the downloaded scripts `install-agent-<platform>.sh` / `uninstall-agent-<platform>.sh`. Installer-only change; the running agent and its privilege model (it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade) are unchanged.
+- **v1.8.3** (2026-06-25) — **Agent heartbeat JSON fix** (Issue #31): a self-hosted agent could fail every heartbeat with `HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host, leaving a stray comma in the hand-built heartbeat JSON. The agent script now substitutes a valid placeholder when that block is empty so it can no longer emit a stray comma, and the backend heartbeat endpoint now parses valid payloads as-is and, only when a body fails to parse, tolerates that specific malformed pattern (a leading or doubled comma) so an already-deployed agent recovers on its next heartbeat after this build is deployed. Backend + agent-script only; healthy agents of every version are byte-for-byte unaffected.
+- **v1.8.2** (2026-06-25) — **ACME nonce fix** (Issue #35 follow-up): the ACME client now scopes the anti-replay nonce **per certificate authority** so a nonce issued by one CA is never sent to another. This fixes ZeroSSL/Google account registration failing with `malformed: The Replay Nonce could not be base64url-decoded` (the client previously shared one nonce across CAs and only auto-retried on `badNonce`). Account registration now always uses a fresh nonce from the target CA, and the retry covers this case too. Backend-only; HTTP-01 and Let's Encrypt are unaffected.
+- **v1.8.1** (2026-06-24) — **ACME DNS-01 fixes** (Issue #35 follow-up): Cloudflare API tokens are now sanitized so a pasted token with quotes/spaces no longer fails with "Invalid request headers"; ZeroSSL/Google **External Account Binding (EAB)** can be entered per-account in the register dialog and EAB-required failures show a clear message; and **Apply Management** now categorizes cluster ACME enable/disable changes under their own "ACME Challenge Routing" section and **Apply/Reject All** correctly process them (previously "Rejected 0 HA/VIP change(s)"), consistent with every other entity. Fully backward compatible.
 - **v1.8.0** (2026-06-23) — **ACME DNS-01 challenge support** (Issue #35): Auto SSL can now validate via a **DNS TXT record** (`_acme-challenge.<domain>`) instead of HTTP-01 on port 80, enabling certificates for **internal/isolated clusters with no public ingress** and **wildcard** certificates (`*.example.com`). Pluggable **per-account DNS provider** (Manual + Cloudflare to start; credentials verified on save and **encrypted at rest**, never returned by the API or logged), the same **PENDING → APPLIED** pipeline, a **bounded automatic retry** on propagation lag, and a **DNS-01 event timeline** in the order detail. **Opt-in** via Settings → ACME (global switch, default off); **HTTP-01 is byte-for-byte unchanged**, with **zero agent or rendered-config changes**. Manual DNS-01 certificates cannot auto-renew unattended; the UI states this and disables auto-renew for them.
 - **v1.7.8** (2026-06-07) — HA / VIP apply progress now shows **per-node** convergence: a multi-node VIP's apply popup reads "Syncing HA/VIP… 1/2 node(s) converged" (matching the HA/VIP table) instead of a coarse per-change count. Frontend-only.
 - **v1.7.7** (2026-06-07) — HA / VIP apply-progress consistency: applying a VIP change (or approving a delete) used to flash the progress popup green instantly while the HA/VIP page still showed `SYNCING (0/1)` for a couple of minutes. The popup now **keeps showing "Syncing HA/VIP… X/Y node(s) converged"** until each member node reports the VIP `ACTIVE` (create/edit) or fully torn down (delete) — exactly like the HAProxy agent-sync widget — then completes green. It's a fire-and-forget background poll (the Apply button is released immediately), bounded at ~5 min so an offline node can't spin forever (then it completes with an informational "still converging — track on the HA/VIP page"). Frontend-only; no backend/agent/schema change.

backend/Dockerfile

@@ -37,5 +37,13 @@ USER appuser
 # Expose port
 EXPOSE 8000
 
-# Run the application in production mode (without --reload)
-CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+# Run the application in production mode (without --reload).
+# UVICORN_WORKERS (default 1) opts into multiple worker processes on multi-core
+# hosts; with 1 worker uvicorn runs in-process, identical to the flagless CMD
+# this replaces. Falls back to WEB_CONCURRENCY when UVICORN_WORKERS is unset
+# because flagless uvicorn honored WEB_CONCURRENCY (uvicorn config.py) — this
+# keeps any deployment that relied on it byte-for-byte compatible. Background
+# tasks are multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as
+# already exercised by the k8s HPA deployment. `exec` keeps uvicorn as PID 1
+# so signal handling is unchanged.
+CMD ["sh", "-c", "exec uvicorn main:app --host 0.0.0.0 --port 8000 --workers ${UVICORN_WORKERS:-${WEB_CONCURRENCY:-1}}"]

backend/main.py

@@ -8,7 +8,8 @@ import redis
 import asyncio
 from datetime import datetime, timedelta
 
-_version_info = {"version": "1.8.0", "releaseName": "ACME DNS-01 challenge support", "releaseDate": "2026-06-23"}
+# Build/deploy marker for the v1.8.x (Issue #35, DNS-01) rollout — ensures the pipeline ships this commit's image.
+_version_info = {"version": "1.8.4", "releaseName": "Agent installer self-kill fix", "releaseDate": "2026-06-27"}
 for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
     try:
         with open(_vpath) as _vf:
@@ -317,7 +318,6 @@ async def complete_pending_acme_orders():
                                     OR dns01_last_attempt_at < NOW() - (
                                         (CASE COALESCE(dns01_attempts, 0) WHEN 0 THEN 15 WHEN 1 THEN 30 ELSE 60 END)
                                         || ' minutes')::INTERVAL
-                                    )
                                 )
                             )
                         )

backend/routers/agent.py

@@ -29,6 +29,40 @@ AGENT_VERSIONS = {
     "linux": "2.0.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -1455,47 +1489,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
     
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")
@@ -1675,9 +1695,13 @@ async def agent_heartbeat_by_name(
             """, x_api_key)
 
         # Check if agent was in upgrading status and version has changed
-        current_agent_status = await conn.fetchval("SELECT status FROM agents WHERE id = $1", agent_id)
-        current_agent_version = await conn.fetchval("SELECT version FROM agents WHERE id = $1", agent_id)
-        current_upgrade_status = await conn.fetchval("SELECT upgrade_status FROM agents WHERE id = $1", agent_id)
+        # (v1.8.6: one round-trip instead of three; row is None exactly when the
+        # per-column fetchvals would each have returned None)
+        current_agent_row = await conn.fetchrow(
+            "SELECT status, version, upgrade_status FROM agents WHERE id = $1", agent_id)
+        current_agent_status = current_agent_row['status'] if current_agent_row else None
+        current_agent_version = current_agent_row['version'] if current_agent_row else None
+        current_upgrade_status = current_agent_row['upgrade_status'] if current_agent_row else None
         
         # Determine new status - preserve upgrading status unless version actually changed
         new_status = current_agent_status or 'online'

backend/routers/cluster.py

@@ -5048,9 +5048,15 @@ async def reject_all_pending_changes(cluster_id: int, authorization: str = Heade
         # Get all pending config versions for this cluster (CRITICAL: Include metadata for rollback!)
         # HA/VIP (Issue #27): exclude vip-* versions — they are rejected/reverted by the
         # VIP reject endpoint (which restores keepalived state), not the generic rollback.
+        # ORDER BY created_at ASC: the rollback loop dedups per entity and keeps the FIRST-processed
+        # snapshot, so the OLDEST snapshot must win — its old_values hold the true pre-change state.
+        # Critical when one entity has multiple pending versions (e.g. cluster ACME enable->disable->enable):
+        # rolling back to the oldest restores the original acme_enabled. (Matches the apply SELECT, which
+        # already orders created_at ASC.)
         pending_versions = await conn.fetch("""
             SELECT id, version_name, metadata FROM config_versions
             WHERE cluster_id = $1 AND status = 'PENDING' AND version_name NOT LIKE 'vip-%'
+            ORDER BY created_at ASC
         """, cluster_id)
         
         # CRITICAL FIX: Detect and clean orphan config versions

backend/routers/letsencrypt.py

@@ -1,6 +1,7 @@
 from fastapi import APIRouter, HTTPException, Header
 from pydantic import BaseModel, Field, field_validator, model_validator
 from typing import Optional, List, Dict
+import base64
 import json
 import logging
 import re
@@ -59,8 +60,11 @@ class AccountCreate(BaseModel):
     email: str
     directory_url: Optional[str] = None
     tos_agreed: bool = True
-    eab_kid: Optional[str] = None
-    eab_hmac_key: Optional[str] = None
+    # EAB (External Account Binding) for CAs that require it (ZeroSSL, Google). The KID is opaque
+    # (bound only); the HMAC key must be base64url so newAccount's _b64url_decode won't raise a
+    # cryptic binascii error (a common copy mistake is standard-base64 '+'/'/' vs urlsafe '-'/'_').
+    eab_kid: Optional[str] = Field(default=None, max_length=256)
+    eab_hmac_key: Optional[str] = Field(default=None, max_length=512)
     # Issue #35: per-account default challenge method + DNS provider (for dns-01).
     challenge_type: str = "http-01"
     dns_provider: Optional[str] = None
@@ -72,6 +76,17 @@ class AccountCreate(BaseModel):
             raise ValueError(f"challenge_type must be one of {_CHALLENGE_TYPES}")
         return v
 
+    @field_validator('eab_hmac_key')
+    @classmethod
+    def _validate_eab_hmac_key(cls, v):
+        if not v:
+            return v
+        try:
+            base64.urlsafe_b64decode(v + '=' * (-len(v) % 4))
+        except Exception:
+            raise ValueError("eab_hmac_key is not valid base64; copy it exactly from your CA account.")
+        return v
+
     @model_validator(mode='after')
     def _require_provider_for_dns01(self):
         if self.challenge_type == 'dns-01' and not (self.dns_provider or '').strip():
@@ -216,8 +231,19 @@ async def create_account(body: AccountCreate, authorization: str = Header(None))
             dns_provider=(body.dns_provider or None),
         )
         return result
+    except HTTPException:
+        # Preserve deliberate status codes (e.g. 409 DNS-01 disabled, 422 unsupported provider) —
+        # the broad except below would otherwise downgrade them all to 400.
+        raise
     except Exception as e:
         logger.error(f"ACME account registration failed: {e}")
+        # Humanize the common EAB-required failure (ZeroSSL/Google). The ACME error propagates as a
+        # string ("Account registration failed: {<dict>}"), so match the URN substring in str(e).
+        if 'externalaccountrequired' in str(e).lower():
+            raise HTTPException(status_code=400, detail=(
+                "This CA requires External Account Binding (EAB). Enter the EAB Key ID and HMAC Key "
+                "from your ZeroSSL/Google account and retry."
+            ))
         raise HTTPException(status_code=400, detail=str(e))
 
 

backend/services/acme_diagnostics.py

@@ -116,6 +116,10 @@ _PROBLEM_HUMANIZED: Dict[str, Dict[str, str]] = {
         "title": "HTTP-01 challenge response mismatch",
         "hint": "The CA fetched the challenge URL but received the wrong key authorization. Confirm the challenge was served from the right backend.",
     },
+    "urn:ietf:params:acme:error:externalAccountRequired": {
+        "title": "External Account Binding (EAB) required",
+        "hint": "This CA (e.g. ZeroSSL, Google) requires EAB. Enter the EAB Key ID and HMAC Key from your CA account when registering.",
+    },
     "urn:ietf:params:acme:error:invalidContact": {
         "title": "Invalid contact email",
         "hint": "The ACME account email is malformed. Update the LE account email.",

backend/services/acme_service.py

@@ -28,7 +28,7 @@ def _b64url(data: bytes) -> str:
 
 
 def _b64url_decode(s: str) -> bytes:
-    s += '=' * (4 - len(s) % 4)
+    s += '=' * (-len(s) % 4)  # pad to a multiple of 4 (0 pad when already aligned)
     return base64.urlsafe_b64decode(s)
 
 
@@ -37,7 +37,11 @@ class ACMEService:
 
     def __init__(self):
         self._directory_cache: Dict[str, dict] = {}
-        self._nonce: Optional[str] = None
+        # Anti-replay nonces are scoped PER CA (directory_url). A Replay-Nonce issued by one ACME
+        # server must never be sent in a JWS to another, or the second server rejects it (e.g. ZeroSSL
+        # "malformed: The Replay Nonce could not be base64url-decoded"). This client is a process-wide
+        # singleton shared across CAs, so a single shared nonce was leaking across them.
+        self._nonce_by_dir: Dict[str, str] = {}
 
     async def _get_settings(self) -> dict:
         conn = await get_database_connection()
@@ -71,17 +75,21 @@ class ACMEService:
                     raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
                 data = await resp.json()
                 if 'Replay-Nonce' in resp.headers:
-                    self._nonce = resp.headers['Replay-Nonce']
+                    self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
                 data['_fetched_at'] = time.time()
                 self._directory_cache[directory_url] = data
                 return data
 
     async def _get_nonce(self, directory_url: str) -> str:
-        if self._nonce:
-            nonce = self._nonce
-            self._nonce = None
-            return nonce
+        # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         directory = await self.get_directory(directory_url)
+        # get_directory may have just captured a nonce for this CA from the directory response.
+        cached = self._nonce_by_dir.pop(directory_url, None)
+        if cached:
+            return cached
         async with aiohttp.ClientSession() as session:
             async with session.head(directory['newNonce']) as resp:
                 return resp.headers['Replay-Nonce']
@@ -188,11 +196,17 @@ class ACMEService:
                     timeout=aiohttp.ClientTimeout(total=30),
                 ) as resp:
                     if 'Replay-Nonce' in resp.headers:
-                        self._nonce = resp.headers['Replay-Nonce']
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400:
+                    if resp.status == 400 and attempt < 2:
                         err = await resp.json()
-                        if err.get('type') == 'urn:ietf:params:acme:error:badNonce' and attempt < 2:
+                        etype = (err.get('type') or '')
+                        edetail = (err.get('detail') or '').lower()
+                        # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                        # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                        # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                        # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                        if etype.endswith('badNonce') or 'nonce' in edetail:
                             nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
                             protected['nonce'] = nonce
                             body = self._sign_jws(private_key, protected, payload)

backend/services/dns_providers/cloudflare.py

@@ -10,6 +10,7 @@ Token scope required: Zone:DNS:Edit + Zone:Read.
 from __future__ import annotations
 
 import logging
+import re
 from typing import Dict, List, Optional, Tuple
 from urllib.parse import quote
 
@@ -22,6 +23,14 @@ logger = logging.getLogger(__name__)
 CLOUDFLARE_API_BASE = "https://api.cloudflare.com/client/v4"
 _TIMEOUT = aiohttp.ClientTimeout(total=20)
 
+# Characters NOT valid in an HTTP bearer credential (RFC 6750 token68: A-Za-z0-9-._~+/=).
+# Cloudflare API tokens are a strict subset of this set, so removing anything outside it can
+# never corrupt a valid token, but it does strip the paste artifacts that make Cloudflare
+# reject the Authorization header with HTTP 400 "Invalid request headers" (CF code 6003):
+# surrounding/embedded quotes, interior spaces/tabs, zero-width/unicode chars, and CR/LF
+# (the latter would otherwise make aiohttp raise client-side before the request is even sent).
+_NON_TOKEN68 = re.compile(r"[^A-Za-z0-9._~+/=-]")
+
 
 def _strip_quotes(s: str) -> str:
     s = (s or "").strip()
@@ -30,6 +39,11 @@ def _strip_quotes(s: str) -> str:
     return s
 
 
+def _sanitize_token(s: str) -> str:
+    """Strip surrounding quotes/whitespace, then drop every character outside the token68 set."""
+    return _NON_TOKEN68.sub("", _strip_quotes(s))
+
+
 class CloudflareDNSProvider(DnsProvider):
     name = "cloudflare"
     label = "Cloudflare"
@@ -47,7 +61,10 @@ class CloudflareDNSProvider(DnsProvider):
 
     def __init__(self, credentials: Dict[str, str] | None = None):
         super().__init__(credentials)
-        self._token = (self.credentials.get("api_token") or "").strip()
+        self._raw_token = (self.credentials.get("api_token") or "").strip()
+        # Sanitize to the token68 set so a pasted token with quotes/spaces/control/unicode chars
+        # cannot produce an invalid Authorization header (CF 6003 "Invalid request headers").
+        self._token = _sanitize_token(self._raw_token)
 
     def _headers(self) -> Dict[str, str]:
         return {"Authorization": f"Bearer {self._token}", "Content-Type": "application/json"}
@@ -97,7 +114,13 @@ class CloudflareDNSProvider(DnsProvider):
                 detail = f"Cloudflare token valid; {total} zone(s) visible."
             return {"ok": True, "detail": detail}
         except DnsProviderError as exc:
-            return {"ok": False, "detail": str(exc)}
+            # Always surface the real Cloudflare reason (e.g. token scope). If sanitizing also changed
+            # the token, append a hint that stray characters were stripped (never echo the token).
+            detail = str(exc)
+            if self._raw_token != self._token:
+                detail += (" Note: the token contained characters that were stripped; if it still "
+                           "fails, re-copy it from Cloudflare without quotes or spaces.")
+            return {"ok": False, "detail": detail}
         except Exception:  # noqa: BLE001 — never leak an internal/transport error verbatim
             return {"ok": False, "detail": "Could not verify the Cloudflare token."}
 

backend/tests/test_acme_humanizer.py

@@ -81,6 +81,7 @@ def test_legacy_plain_string_with_brace_but_invalid_json_falls_back():
     ("urn:ietf:params:acme:error:rejectedIdentifier", "blacklisted",    "rejected"),
     ("urn:ietf:params:acme:error:serverInternal", "internal err",       "ACME server"),
     ("urn:ietf:params:acme:error:userActionRequired", "agree to ToS",   "User action"),
+    ("urn:ietf:params:acme:error:externalAccountRequired", "EAB required", "External Account Binding"),
 ])
 def test_known_problem_types_are_humanized(problem_type, detail_text, expected_title_contains):
     payload = json.dumps({"type": problem_type, "detail": detail_text, "status": 400})

backend/tests/test_acme_pydantic_validation.py

@@ -8,7 +8,31 @@ from pydantic import ValidationError
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from routers.letsencrypt import CertificateRequest
+from routers.letsencrypt import CertificateRequest, AccountCreate
+
+
+class TestAccountCreateEAB:
+    """Issue #35 follow-up: EAB HMAC key must be valid base64url; empty/None passes through
+    (falls back to global Settings) so non-EAB accounts (HTTP-01 / LE / Cloudflare) are unaffected."""
+
+    def test_no_eab_is_allowed(self):
+        acc = AccountCreate(email="a@b.com")
+        assert acc.eab_hmac_key is None and acc.eab_kid is None
+
+    def test_valid_base64url_hmac_accepted(self):
+        # urlsafe base64, unpadded and padded — both accepted.
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZGVmZ2g")
+        AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="YWJjZA==")
+
+    def test_invalid_base64_hmac_rejected(self):
+        # 5 base64 chars (count ≡ 1 mod 4) is undecodable — the exact shape that would otherwise
+        # make register_account's _b64url_decode raise a cryptic binascii error.
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="AAAAA")
+
+    def test_oversized_hmac_rejected(self):
+        with pytest.raises(ValidationError):
+            AccountCreate(email="a@b.com", eab_kid="kid-1", eab_hmac_key="A" * 600)
 
 
 class TestCertificateRequestDomains:

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,61 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"
+
+
+def test_cleanup_does_not_self_kill_via_bare_haproxy_agent_pattern():
+    # Issue #31 (v1.8.4): the pre-installation cleanup kills processes by pgrep -f "$pattern". A bare
+    # "haproxy-agent" pattern also matches the installer's OWN path (install-haproxy-agent-*.sh) and a
+    # sudo/PAM ancestor, so the installer killed itself. The kill loop must target ONLY the installed
+    # agent (binary path + service/label), never the bare string.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert 'for pattern in "haproxy-agent"' not in script, (
+            f"{name}: pre-install cleanup uses the bare 'haproxy-agent' kill pattern -> self-kill (issue #31)"
+        )
+        # The narrowed, installer-safe pattern must be present (binary path via $INSTALL_DIR).
+        assert 'for pattern in "$INSTALL_DIR/haproxy-agent"' in script, (
+            f"{name}: cleanup must match the installed binary path, not a bare substring"
+        )

backend/tests/test_dns01.py

@@ -68,3 +68,153 @@ def test_provider_registry_and_allow_list():
     except ValueError:
         raised = True
     assert raised
+
+
+def test_cloudflare_token_sanitize():
+    # Issue #35 follow-up: a pasted token with quotes/spaces/control/unicode chars produced an
+    # invalid Authorization header (CF 6003 "Invalid request headers"). The sanitizer strips them.
+    from services.dns_providers.cloudflare import _sanitize_token, CloudflareDNSProvider
+
+    # Surrounding double quotes stripped.
+    assert _sanitize_token('"abc123-_def"') == 'abc123-_def'
+    # Interior spaces / tabs / newlines removed.
+    assert _sanitize_token('abc 123\tdef\n') == 'abc123def'
+    # A clean token68 string is unchanged (cannot corrupt a valid Cloudflare token).
+    clean = 'A1b2-_C3.d4~e5+f6/g7=='
+    assert _sanitize_token(clean) == clean
+    # Single quotes and a zero-width char removed.
+    assert _sanitize_token("'tok" + chr(0x200b) + "en'") == 'token'
+
+    # The provider constructor sanitizes into _token and keeps the raw input for diagnostics.
+    p = CloudflareDNSProvider({"api_token": '"my-token_123"'})
+    assert p._token == 'my-token_123'
+    assert p._raw_token == '"my-token_123"'
+
+
+def test_b64url_decode_padding_roundtrip():
+    # Issue #35 v1.8.2: _b64url_decode must round-trip for EVERY length, including base64url strings
+    # whose length is a multiple of 4 (the case the old padding formula '=' * (4 - len%4) over-padded).
+    from services.acme_service import _b64url as enc_fn, _b64url_decode as dec_fn
+    for n in range(0, 20):
+        data = bytes(range(n))
+        assert dec_fn(enc_fn(data)) == data, f"round-trip failed at byte length {n}"
+
+
+def test_nonce_scoped_per_directory():
+    # Issue #35 v1.8.2: a nonce cached for one CA (directory_url) must never be returned for another,
+    # and must be single-use. Both directories are pre-cached so _get_nonce returns without network.
+    import asyncio
+    svc = ACMEService()
+    svc._nonce_by_dir = {"https://a.example/dir": "NONCE_A", "https://b.example/dir": "NONCE_B"}
+    got = asyncio.run(svc._get_nonce("https://a.example/dir"))
+    assert got == "NONCE_A"                                            # returns THIS CA's nonce
+    assert svc._nonce_by_dir.get("https://a.example/dir") is None      # consumed (single-use)
+    assert svc._nonce_by_dir.get("https://b.example/dir") == "NONCE_B" # the other CA is untouched
+
+
+def _sql_paren_depth(sql: str):
+    """Parenthesis depth of a SQL string, counting only OUTSIDE '...' literals (with ''
+    escapes), `--` line comments and /* */ block comments. Single-pass state machine so a
+    `--` inside a literal or a `'` inside a comment cannot corrupt the count. Dollar-quoted
+    strings are out of scope (not used in this codebase). Returns (final_depth, min_depth).
+    """
+    depth = 0
+    min_depth = 0
+    state = "normal"
+    i, n = 0, len(sql)
+    while i < n:
+        ch = sql[i]
+        nxt = sql[i + 1] if i + 1 < n else ""
+        if state == "normal":
+            if ch == "'":
+                state = "string"
+            elif ch == "-" and nxt == "-":
+                state = "line_comment"
+                i += 1
+            elif ch == "/" and nxt == "*":
+                state = "block_comment"
+                i += 1
+            elif ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                min_depth = min(min_depth, depth)
+        elif state == "string":
+            if ch == "'":
+                if nxt == "'":
+                    i += 1  # escaped '' stays inside the literal
+                else:
+                    state = "normal"
+        elif state == "line_comment":
+            if ch == "\n":
+                state = "normal"
+        else:  # block_comment
+            if ch == "*" and nxt == "/":
+                state = "normal"
+                i += 1
+        i += 1
+    return depth, min_depth
+
+
+def test_acme_sql_parentheses_balanced():
+    """Issue #35 v1.8.5: the completion task's order-claim query shipped (v1.8.0-v1.8.4) with an
+    extra closing parenthesis, so EVERY 60s cycle died with `syntax error at or near ")"` and no
+    background ACME work (claim/finalize/download, DNS-01 publish, wizard-staged promotion,
+    retry, TXT cleanup) ever ran. The suite never caught it because the DB layer is mocked and
+    raw SQL never reaches a real parser. This guard scans the ACME modules' SQL string literals
+    for unbalanced parentheses.
+
+    Guard scope is deliberately conservative to avoid false positives on production changes:
+    keyword matching is case-sensitive (SQL is uppercase in this codebase; prose in docstrings
+    is not) and f-string fragments are excluded (they split at `{`, so a fragment may be
+    legitimately unbalanced).
+    """
+    import ast
+    import re
+
+    backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    modules = [
+        "main.py",
+        os.path.join("services", "dns01_orchestrator.py"),
+        os.path.join("services", "acme_service.py"),
+        os.path.join("services", "letsencrypt_service.py"),
+        os.path.join("routers", "letsencrypt.py"),
+        os.path.join("routers", "acme_diagnostics.py"),
+    ]
+    problems = []
+    for rel in modules:
+        with open(os.path.join(backend_dir, rel), encoding="utf-8") as fh:
+            tree = ast.parse(fh.read())
+        fstring_parts = {
+            id(const)
+            for joined in ast.walk(tree) if isinstance(joined, ast.JoinedStr)
+            for const in ast.walk(joined) if isinstance(const, ast.Constant)
+        }
+        for node in ast.walk(tree):
+            if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+                continue
+            if id(node) in fstring_parts:
+                continue
+            sql = node.value
+            if not re.search(r"\b(SELECT|INSERT|UPDATE|DELETE)\b", sql):
+                continue
+            if not re.search(r"\b(FROM|INTO|SET|WHERE)\b", sql):
+                continue
+            depth, min_depth = _sql_paren_depth(sql)
+            if depth != 0 or min_depth < 0:
+                problems.append(f"{rel}:{node.lineno} (paren depth {depth:+d}, min {min_depth})")
+    assert not problems, f"Unbalanced parentheses in SQL literal(s): {problems}"
+
+
+def test_sql_paren_depth_scanner():
+    # The guard's scanner itself: parens in literals/comments must not count; '' escapes and
+    # block comments handled; an extra ')' is reported via min_depth even if a later '(' would
+    # re-balance the total.
+    assert _sql_paren_depth("SELECT (1)") == (0, 0)
+    assert _sql_paren_depth("SELECT (1))") == (-1, -1)                       # the v1.8.0 bug shape
+    assert _sql_paren_depth("SELECT ')' , '((' FROM t") == (0, 0)            # literals ignored
+    assert _sql_paren_depth("SELECT 'it''s ))' FROM t") == (0, 0)            # '' escape stays inside
+    assert _sql_paren_depth("SELECT 1 -- comment ) (\nFROM t") == (0, 0)     # line comment ignored
+    assert _sql_paren_depth("SELECT 1 /* ) */ FROM t") == (0, 0)             # block comment ignored
+    assert _sql_paren_depth("SELECT 'a--b' AND (x=1\n)") == (0, 0)           # -- inside literal is data
+    assert _sql_paren_depth("WHERE x) AND (y") == (0, -1)                    # net 0 but went negative

backend/utils/agent_scripts/linux_install.sh

@@ -470,7 +470,12 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "haproxy-agent.service"; do
+# issue #31: match ONLY the installed agent (binary path + service), never the bare string
+# "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command line
+# or a sudo/PAM ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself ("Killed", install aborts). The systemd service is also stopped below; the
+# "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "haproxy-agent.service"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -1055,7 +1060,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1357,7 +1367,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then

backend/utils/agent_scripts/macos_install.sh

@@ -318,7 +318,11 @@ safe_remove() {
 [[ "$QUIET_MODE" != "true" ]] && echo "Terminating existing HAProxy Agent processes..."
 KILLED_COUNT=0
 INSTALLER_PID=$$
-for pattern in "haproxy-agent" "/usr/local/bin/haproxy-agent" "com.haproxy.agent"; do
+# issue #31: match ONLY the installed agent (binary path + LaunchDaemon label), never the bare
+# string "haproxy-agent". With pgrep -f, that bare string can also match the installer's OWN command
+# line or a sudo ancestor (which the $$/$PPID guard does not fully cover), making the cleanup kill
+# the installer itself. The "$INSTALL_DIR/haproxy-agent" path still catches any running daemon.
+for pattern in "$INSTALL_DIR/haproxy-agent" "com.haproxy.agent"; do
     PIDS=$(pgrep -f "$pattern" 2>/dev/null || true)
     if [[ -n "$PIDS" ]]; then
         FILTERED=""
@@ -920,7 +924,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1191,7 +1200,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then

docker-compose.yml

@@ -51,6 +51,9 @@ services:
       - LOG_LEVEL=INFO
       - PUBLIC_URL=http://localhost:8080
       - MANAGEMENT_BASE_URL=http://localhost:8080
+      # Empty when unset on the host: the image CMD then falls back to
+      # WEB_CONCURRENCY (uvicorn's native env) and finally to 1.
+      - UVICORN_WORKERS=${UVICORN_WORKERS:-}
     volumes:
       - haproxy_configs:/etc/haproxy
     expose:

frontend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.8.0",
+  "version": "1.8.6",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {

frontend/src/components/ACMEAutomation.js

@@ -1,7 +1,7 @@
 import React, { useState, useEffect, useCallback, useRef } from 'react';
 import {
   Card, Table, Button, Tag, Space, Modal, Form, Input, Select, Steps,
-  message, Row, Col, Statistic, Alert, Tooltip, Switch, theme, Segmented,
+  message, Row, Col, Statistic, Alert, Tooltip, Switch, theme, Segmented, Collapse,
   Tabs, Timeline, Spin, Empty, Typography, Divider
 } from 'antd';
 import {
@@ -18,8 +18,14 @@ import axios from 'axios';
 
 const { Option } = Select;
 
-const getErrorMsg = (err, fallback) =>
-  err?.response?.data?.error?.message || err?.response?.data?.detail || fallback;
+const getErrorMsg = (err, fallback) => {
+  const data = err?.response?.data;
+  // FastAPI/Pydantic 422s wrap the specific field message in error.details.validation_errors[];
+  // the top-level error.message is generic ("Validation error in request data"), so prefer the
+  // field-level message (e.g. the EAB base64 hint) when present.
+  const fieldMsg = data?.error?.details?.validation_errors?.[0]?.message;
+  return fieldMsg || data?.error?.message || data?.detail || fallback;
+};
 
 // Issue #35: humanize the dotted event_type tokens emitted for DNS-01 orders so the diagnostics
 // timeline reads as a step-by-step progress log rather than raw machine strings. Unknown types
@@ -561,6 +567,9 @@ const ACMEAutomation = () => {
         tos_agreed: values.tos_agreed,
         challenge_type: challengeType,
         dns_provider: dnsProvider,
+        // EAB for CAs that require it (ZeroSSL/Google). Empty → backend falls back to global Settings.
+        eab_kid: (values.eab_kid || '').trim() || undefined,
+        eab_hmac_key: (values.eab_hmac_key || '').trim() || undefined,
       });
       const accountId = res.data?.id;
       // For an automated DNS-01 provider, store the entered credentials (verified server-side).
@@ -1613,6 +1622,59 @@ const ACMEAutomation = () => {
               Let's Encrypt Subscriber Agreement
             </a>).
           </div>
+          {/* Issue #35: External Account Binding — required by ZeroSSL / Google Trust Services. */}
+          <Collapse
+            ghost
+            style={{ marginTop: 12 }}
+            items={[{
+              key: 'eab',
+              label: 'External Account Binding (EAB)',
+              children: (
+                <>
+                  <Alert
+                    type="info"
+                    showIcon
+                    style={{ marginBottom: 12 }}
+                    message="Required by some CAs (ZeroSSL, Google Trust Services). Copy the Key ID and HMAC Key from your CA account. Re-enter them on each registration; the HMAC key is not stored."
+                  />
+                  <Form.Item
+                    name="eab_kid"
+                    label="EAB Key ID"
+                    dependencies={['eab_hmac_key']}
+                    rules={[({ getFieldValue }) => ({
+                      validator(_, value) {
+                        const kid = (value || '').trim();
+                        const hmac = (getFieldValue('eab_hmac_key') || '').trim();
+                        if (!kid && hmac) {
+                          return Promise.reject(new Error('EAB Key ID is required when an HMAC Key is entered.'));
+                        }
+                        return Promise.resolve();
+                      },
+                    })]}
+                  >
+                    <Input placeholder="EAB Key Identifier" autoComplete="off" />
+                  </Form.Item>
+                  <Form.Item
+                    name="eab_hmac_key"
+                    label="EAB HMAC Key"
+                    dependencies={['eab_kid']}
+                    rules={[({ getFieldValue }) => ({
+                      validator(_, value) {
+                        const kid = (getFieldValue('eab_kid') || '').trim();
+                        const hmac = (value || '').trim();
+                        if (kid && !hmac) {
+                          return Promise.reject(new Error('EAB HMAC Key is required when a Key ID is entered.'));
+                        }
+                        return Promise.resolve();
+                      },
+                    })]}
+                  >
+                    <Input.Password placeholder="EAB HMAC Key (base64url)" autoComplete="new-password" />
+                  </Form.Item>
+                </>
+              ),
+            }]}
+          />
           {dns01Enabled && (
             <>
               <Divider style={{ margin: '16px 0 12px' }} />

frontend/src/components/AgentManagement.js

@@ -2378,7 +2378,7 @@ const AgentManagement = () => {
                         const element = document.createElement('a');
                         const file = new Blob([installScript], { type: 'text/plain' });
                         element.href = URL.createObjectURL(file);
-                        element.download = `install-haproxy-agent-${selectedPlatform}.sh`;
+                        element.download = `install-agent-${selectedPlatform}.sh`;
                         document.body.appendChild(element);
                         element.click();
                         document.body.removeChild(element);
@@ -2516,7 +2516,7 @@ const AgentManagement = () => {
                                     const element = document.createElement('a');
                                     const file = new Blob([uninstallScript], { type: 'text/plain' });
                                     element.href = URL.createObjectURL(file);
-                                    element.download = `uninstall-haproxy-agent-${selectedPlatform}.sh`;
+                                    element.download = `uninstall-agent-${selectedPlatform}.sh`;
                                     document.body.appendChild(element);
                                     element.click();
                                     document.body.removeChild(element);
@@ -3147,7 +3147,7 @@ const AgentManagement = () => {
                           const element = document.createElement('a');
                           const file = new Blob([deleteUninstallScript], { type: 'text/plain' });
                           element.href = URL.createObjectURL(file);
-                          element.download = `uninstall-haproxy-agent-${agentToDelete.platform || 'linux'}.sh`;
+                          element.download = `uninstall-agent-${agentToDelete.platform || 'linux'}.sh`;
                           document.body.appendChild(element);
                           element.click();
                           document.body.removeChild(element);

frontend/src/components/ApplyManagement.js

@@ -368,7 +368,7 @@ const ApplyManagement = () => {
       title: 'Apply All Configuration Changes',
       content: (
         <div>
-          <p>You are about to apply <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to apply <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -385,6 +385,12 @@ const ApplyManagement = () => {
             {(pendingChanges.vips || []).length > 0 && (
               <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
             )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All changes will be applied together and sent to agents"
@@ -523,6 +529,10 @@ const ApplyManagement = () => {
       // async on their next agent poll) instead of looping on "Entities: 0/0".
       const vipCount = (pendingChanges.vips || []).length;
       const isVipOnly = totalEntities === 0 && !isRestoreOperation && nonVipPendingVersions.length === 0 && vipCount > 0;
+      // Config-version-only apply (e.g. cluster ACME enable/disable): no entity rows, not a restore,
+      // but there ARE non-vip config versions to push. Without this branch it falls to the "else" and
+      // shows a misleading "Entities: 0/0" while still syncing agents.
+      const isConfigVersionOnly = totalEntities === 0 && !isRestoreOperation && vipCount === 0 && nonVipPendingVersions.length > 0;
 
       if (isRestoreOperation) {
         // Restore operation: Show "Configuration" instead of "Entities"
@@ -534,6 +544,12 @@ const ApplyManagement = () => {
         setSyncProgress({ visible: true, step: `Applying ${vipCount} HA/VIP change(s)...`, progress: 20 });
         startProgress('apply', `Applying ${vipCount} HA/VIP change(s)...`);
         updateEntityCounts(0, vipCount, 0, 0, 0);
+      } else if (isConfigVersionOnly) {
+        // Config-version-only (ACME toggle, etc.): show "Configuration" instead of "Entities: 0/0".
+        const cfgCount = nonVipPendingVersions.length;
+        setSyncProgress({ visible: true, step: `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`, progress: 20 });
+        startProgress('apply', `Applying configuration change... Configuration: 0/${cfgCount}, Agents: ⏳`);
+        updateEntityCounts(0, cfgCount, 0, totalAgents, disabledAgents);
       } else {
         // Normal operation: Show "Entities" as usual
         setSyncProgress({ visible: true, step: `Applying configuration changes... Entities: 0/${totalEntities}, Agents: ⏳`, progress: 20 });
@@ -556,10 +572,12 @@ const ApplyManagement = () => {
         }
       }
 
-      // Apply HAProxy changes only if there are any (avoids a no-op call when only VIPs are pending).
+      // Apply HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (cluster ACME enable/disable, restore, bulk-import). Gating only on haproxyPending used to
+      // skip the call for config-version-only states, leaving those versions stuck PENDING.
       const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
         + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
-      const response = haproxyPending > 0
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
         ? await axios.post(
             `/api/clusters/${selectedCluster.id}/apply-changes`,
             {},
@@ -805,7 +823,7 @@ const ApplyManagement = () => {
       title: 'Reject All Configuration Changes',
       content: (
         <div>
-          <p>You are about to reject <strong>{effectiveTotal}</strong> pending changes:</p>
+          <p>You are about to reject <strong>{modalChangeCount}</strong> pending changes:</p>
           <ul style={{ marginTop: 10, marginBottom: 10 }}>
             {pendingChanges.frontends.length > 0 && (
               <li><strong>{pendingChanges.frontends.length}</strong> Frontend changes</li>
@@ -822,6 +840,12 @@ const ApplyManagement = () => {
             {(pendingChanges.vips || []).length > 0 && (
               <li><strong>{pendingChanges.vips.length}</strong> HA/VIP changes</li>
             )}
+            {acmeVersions.length > 0 && (
+              <li><strong>{acmeVersions.length}</strong> ACME Challenge Routing changes</li>
+            )}
+            {otherConfigVersions.length > 0 && (
+              <li><strong>{otherConfigVersions.length}</strong> Other configuration changes</li>
+            )}
           </ul>
           <Alert
             message="All pending changes will be permanently discarded"
@@ -874,10 +898,16 @@ const ApplyManagement = () => {
         }
       }
 
-      // Reject HAProxy changes only if there are any.
+      // Reject HAProxy changes if there are entity-level changes OR any non-vip config version
+      // (e.g. cluster ACME enable/disable, restore, bulk-import) — the backend DELETE rejects all
+      // non-vip PENDING versions and rolls back their snapshots. Gating only on haproxyPending used
+      // to skip the call for config-version-only states, returning the misleading "Rejected 0 HA/VIP".
       const haproxyPending = pendingChanges.frontends.length + pendingChanges.backends.length
         + pendingChanges.waf_rules.length + pendingChanges.ssl_certificates.length;
-      const response = haproxyPending > 0
+      const nonVipPendingVersions = configVersions.filter(
+        v => v.status === 'PENDING' && !(v.version_name || '').startsWith('vip-')
+      );
+      const response = (haproxyPending > 0 || nonVipPendingVersions.length > 0)
         ? await axios.delete(
             `/api/clusters/${selectedCluster.id}/pending-changes`,
             { headers: { Authorization: `Bearer ${token}` } }
@@ -957,6 +987,23 @@ const ApplyManagement = () => {
   const appliedVersions = configVersions.filter(v => v.status === 'APPLIED');
   const rejectedVersions = configVersions.filter(v => v.status === 'REJECTED');
   const effectiveTotal = pendingChanges.total_count > 0 ? pendingChanges.total_count : pendingVersions.length;
+  // Issue #35: cluster ACME enable/disable produce `cluster-<id>-acme-<enable|disable>-<ts>` config
+  // versions that have NO entity-level pending flag, so they were neither categorized nor counted.
+  const ACME_VERSION_RE = /^cluster-\d+-acme-(enable|disable)-/;
+  const ENTITY_VERSION_PREFIXES = ['frontend-', 'backend-', 'server-', 'ssl-', 'waf-'];
+  const acmeVersions = pendingVersions.filter(v => ACME_VERSION_RE.test(v.version_name || ''));
+  // "Other" config versions for the confirm modal = non-vip, non-acme, non-entity-backed (i.e.
+  // restore-*/bulk-import-*/other cluster-level) — entity-backed versions are already counted via
+  // total_count, and vips are listed separately, so excluding them avoids double-counting.
+  const otherConfigVersions = pendingVersions.filter(v => {
+    const n = v.version_name || '';
+    if (n.startsWith('vip-') || ACME_VERSION_RE.test(n)) return false;
+    return !ENTITY_VERSION_PREFIXES.some(p => n.startsWith(p));
+  });
+  // Confirm-modal header count: entities+VIPs (total_count) + ACME + other config versions, so the
+  // header equals the sum of the listed <li> items in every state. The button-enable gate keeps
+  // using effectiveTotal (unchanged) so entity-only button/Alert behavior is byte-identical.
+  const modalChangeCount = (pendingChanges.total_count || 0) + acmeVersions.length + otherConfigVersions.length;
 
   const renderPendingItem = (item, type, icon) => {
     // For PENDING items, don't show sync status since they haven't been applied yet
@@ -1314,12 +1361,46 @@ const ApplyManagement = () => {
                     </div>
                   )}
 
+                  {/* Issue #35: ACME Challenge Routing (cluster-<id>-acme-*) versions have no entity
+                      flag. Render them in their own section REGARDLESS of whether entity sections are
+                      present, so a co-pending ACME toggle is never hidden in the left panel. */}
+                  {acmeVersions.length > 0 && (
+                    <div style={{ marginTop: 8, marginBottom: 8 }}>
+                      <Title level={5}>
+                        <SafetyCertificateOutlined style={{ marginRight: 8, color: '#1890ff' }} />
+                        ACME Challenge Routing ({acmeVersions.length})
+                      </Title>
+                      {acmeVersions.map(v => {
+                        const isEnable = /^cluster-\d+-acme-enable-/.test(v.version_name);
+                        return (
+                          <div key={v.id} style={{
+                            padding: 10, border: '1px dashed #1890ff', borderRadius: 6, marginBottom: 8,
+                            display: 'flex', alignItems: 'center', justifyContent: 'space-between', backgroundColor: '#f0f8ff'
+                          }}>
+                            <span style={{ fontFamily: 'monospace' }}>{v.version_name}</span>
+                            <span>
+                              <Tag color={isEnable ? 'green' : 'default'}>{isEnable ? 'ENABLE' : 'DISABLE'}</Tag>
+                              <Tag color="orange">PENDING</Tag>
+                            </span>
+                          </div>
+                        );
+                      })}
+                      <div style={{ fontSize: 12, color: token.colorTextSecondary, marginTop: 4 }}>
+                        ACME challenge routing change. Apply to push the updated HAProxy config to the agents.
+                      </div>
+                    </div>
+                  )}
+
                   {pendingChanges.frontends.length === 0 && pendingChanges.backends.length === 0 && pendingChanges.waf_rules.length === 0 && pendingChanges.ssl_certificates.length === 0 && (pendingChanges.vips || []).length === 0 && pendingVersions.length > 0 && (
                     <div style={{ marginTop: 8 }}>
                       {(() => {
                         const restoreVersions = pendingVersions.filter(v => v.version_name.startsWith('restore-'));
                         const bulkImportVersions = pendingVersions.filter(v => v.version_name.startsWith('bulk-import-'));
-                        const otherVersions = pendingVersions.filter(v => !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-'));
+                        // Exclude restore-/bulk-import- (own sections), vip-* (VIP section), and acme-*
+                        // (the dedicated ACME section above) so they aren't duplicated in "Other".
+                        const otherVersions = pendingVersions.filter(v =>
+                          !v.version_name.startsWith('restore-') && !v.version_name.startsWith('bulk-import-')
+                          && !v.version_name.startsWith('vip-') && !ACME_VERSION_RE.test(v.version_name));
                         
                         return (
                           <>

version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.8.0",
-  "releaseName": "ACME DNS-01 challenge support",
-  "releaseDate": "2026-06-23"
+  "version": "1.8.6",
+  "releaseName": "Opt-in API workers + heartbeat micro-optimization",
+  "releaseDate": "2026-07-06"
 }