mirror of
https://github.com/taylanbakircioglu/haproxy-openmanager.git
synced 2026-09-11 21:38:55 +00:00
taylanbakircioglu
0ee227363e
Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.
DISCOVERY REPORTS THAT NEVER REACHED THE SERVER
The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.
- the report is cached only on a 2xx;
- GET /agents/{name}/keepalived-config now reports whether the server actually
holds a discovery for that agent, and the cache may only suppress while it
says yes - which is what lets nodes stuck from earlier releases recover on
their own, with nobody touching them;
- the flag is parsed with has() + tostring, not `// empty`: jq's alternative
operator returns the alternative for **false** as well as null, so the naive
form could not tell "no record" from "older backend" and the recovery would
have been completely inert;
- a 400/413/422 records the refusal so identical bytes are not re-posted
forever - 4xx and 5xx agent calls are never sampled out of the request log,
so an unattended loop would write a row carrying the whole config every
cycle - while 401 and 404 keep retrying, because here they mean a token
rotation or an agent row briefly absent, not a bad payload;
- the CLEAR path had the same exit-code defect, where it left a stale row
offering a managed node for adoption with nothing to ever retry it.
CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS
check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.
DETERMINISTIC CONFIG PATH
A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.
Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.
Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.
2531 lines
157 KiB
Markdown
2531 lines
157 KiB
Markdown
# HAProxy OpenManager
|
||
|
||
Modern, web-based management interface for HAProxy load balancers with multi-cluster support, agent-based pull architecture, and automated deployment capabilities.
|
||
|
||
|
||
|
||
## Table of Contents
|
||
|
||
1. [Project Overview](#project-overview)
|
||
- [Agent Pull Architecture](#agent-pull-architecture)
|
||
- [Features at a Glance](#features-at-a-glance)
|
||
2. [Screenshots](#screenshots)
|
||
- [Multi-Cluster & Agent Setup](#multi-cluster--agent-setup)
|
||
- [Dashboard - Real-time Monitoring](#dashboard---real-time-monitoring)
|
||
- [HAProxy Configuration](#haproxy-configuration)
|
||
- [Configuration Import & Viewer](#configuration-import--viewer)
|
||
- [Apply Management & Version Control](#apply-management--version-control)
|
||
- [Security & Certificate Management](#security--certificate-management)
|
||
- [ACME Automation](#acme-automation---automated-ssl-certificates)
|
||
- [Site Wizard - Guided Multi-Step Host Setup](#site-wizard---guided-multi-step-host-setup)
|
||
- [WAF Management](#waf-management)
|
||
- [IP Inventory](#ip-inventory---cross-cluster-ip-search)
|
||
3. [Key Capabilities](#key-capabilities)
|
||
- [Configuration & Entity Management](#configuration--entity-management)
|
||
- [Agent & Cluster Management](#agent--cluster-management)
|
||
- [Version Control & Change Management](#version-control--change-management)
|
||
- [Monitoring & Security](#monitoring--security)
|
||
- [ACME Auto SSL / Let's Encrypt](#acme-auto-ssl--lets-encrypt)
|
||
- [Integration & API](#integration--api)
|
||
4. [Architecture](#architecture)
|
||
- [System Architecture with Agent Pull Model](#system-architecture-with-agent-pull-model)
|
||
- [Component Details](#component-details)
|
||
- [Agent Pull Communication Flow](#agent-pull-communication-flow---detailed-entity-management)
|
||
5. [Features & User Interface](#features--user-interface)
|
||
- [Pool Management](#pool-management---agent-pool-organization)
|
||
- [Cluster Management](#cluster-management---multi-cluster-haproxy-administration)
|
||
- [Agent Management](#agent-management---haproxy-agent-deployment--monitoring)
|
||
- [Dashboard](#dashboard---main-overview--real-time-monitoring)
|
||
- [Frontend Management](#frontend-management---virtual-host--routing-configuration)
|
||
- [New Site Wizard](#new-site-wizard---guided-multi-step-host-setup)
|
||
- [Backend Servers](#backend-servers---server-pool-management)
|
||
- [Configuration](#configuration---haproxy-config-file-management)
|
||
- [Apply Management](#apply-management---change-tracking--deployment)
|
||
- [SSL Certificates](#ssl-certificates---centralized-tlsssl-certificate-management)
|
||
- [ACME Auto SSL](#acme-auto-ssl---automated-certificate-management)
|
||
- [WAF Management](#waf-management---web-application-firewall)
|
||
- [IP Inventory](#ip-inventory---cross-cluster-ip-search--discovery)
|
||
- [HA / VIP Management](#ha--vip-management---keepalived-virtual-ip-failover)
|
||
- [User Management](#user-management---access-control--authentication)
|
||
- [Settings](#settings---system-configuration)
|
||
6. [Getting Started - First-Time Usage](#getting-started---first-time-usage)
|
||
7. [Installation](#installation)
|
||
- [Docker Installation](#docker-installation)
|
||
- [Kubernetes/OpenShift Installation](#kubernetesopenshift-installation)
|
||
- [Standalone Installation](docs/STANDALONE_INSTALL.md)
|
||
- [Local Development](#local-development)
|
||
8. [Configuration](#configuration)
|
||
9. [API Reference](#api-reference)
|
||
10. [Development](#development)
|
||
11. [Troubleshooting](#troubleshooting)
|
||
12. [Contributing](#contributing)
|
||
13. [Use Cases](#use-cases)
|
||
14. [License](#license)
|
||
15. [Author](#author)
|
||
16. [Support](#support)
|
||
|
||
## Project Overview
|
||
|
||
HAProxy OpenManager is a comprehensive management platform designed to simplify HAProxy administration across multiple environments. It provides a unified interface for managing remote HAProxy instances through an agent-based pull architecture, supporting multi-cluster configurations with centralized control.
|
||
|
||
### Agent Pull Architecture
|
||
|
||
This project implements an **agent pull architecture** where management operations are executed through a lightweight agent service (`haproxy-agent`) installed on each target HAProxy server:
|
||
|
||
#### Conceptual Hierarchy
|
||
```
|
||
Pool (Logical Grouping)
|
||
└── Cluster (HAProxy Cluster Definition)
|
||
└── Agent 1 → HAProxy Server 1
|
||
└── Agent 2 → HAProxy Server 2
|
||
└── Agent 3 → HAProxy Server 3
|
||
```
|
||
|
||
#### Setup Flow
|
||
1. **Create Agent Pool**: First, create a pool to logically group agents (e.g., "Production Pool")
|
||
2. **Create Cluster**: Select the pool and create a cluster (e.g., "Production Cluster with 3 nodes")
|
||
3. **Deploy Agents**: Generate installation scripts for each HAProxy server, selecting the same pool
|
||
4. **Result**: Multiple agents in one pool, all managing one cluster
|
||
|
||
#### How It Works
|
||
1. **Automatic Polling**: Agents periodically poll the backend for configuration tasks (every 30 seconds)
|
||
2. **Local Execution**: Agents apply changes to local `haproxy.cfg` and reload HAProxy service
|
||
3. **No Direct Push**: Backend never directly pushes changes - agents pull on their schedule
|
||
4. **Unified Management**: All agents in a pool receive same cluster configuration
|
||
|
||
This architecture provides better security (no inbound connections to HAProxy servers), resilience (agents can retry on failure), and scalability (backend doesn't track agent state).
|
||
|
||
### Features at a Glance
|
||
|
||
✅ **Agent-Based Pull Architecture** - Secure, scalable management without inbound connections
|
||
✅ **Multi-Cluster & Pool Management** - Organize and manage multiple HAProxy clusters from one interface
|
||
✅ **Frontend/Backend/Server CRUD** - Complete entity management with visual UI
|
||
✅ **New Site Wizard** - Step-by-step guided setup for a complete proxied host (frontend + backend + servers + SSL/ACME) in one consolidated atomic apply
|
||
✅ **Bulk Config Import** - Import existing `haproxy.cfg` files with smart SSL auto-assignment
|
||
✅ **Version Control & Rollback** - Every change versioned with one-click restore capability
|
||
✅ **Real-Time Monitoring** - Live stats, health checks, and performance dashboards
|
||
✅ **SSL Certificate Management** - Centralized SSL with expiration tracking
|
||
✅ **CSR Creation** *(v1.9.0)* - Generate a private key + CSR in-app (RSA 2048/4096, ECDSA P-256/P-384, full subject + SANs), have it signed by any external CA, then import the signed certificate — the key never leaves the server
|
||
✅ **ACME Auto SSL (Let's Encrypt)** - Automated certificate issuance, renewal, and deployment via ACME protocol
|
||
✅ **ACME DNS-01 Challenge** *(v1.8.0)* - TXT-record validation for internal/isolated clusters (no public port 80) and wildcard certificates; pluggable DNS providers (Manual + Cloudflare + GoDaddy *(v1.10.0)*), opt-in, HTTP-01 unchanged
|
||
✅ **ACME Certificate Diagnostic Panel** - Automated preflight that checks agent readiness, DNS resolution, port 80 reachability, and ACME challenge ACL before issuing certificates
|
||
✅ **WAF Rules** - Web Application Firewall management and deployment
|
||
✅ **Agent Script Versioning** - Update agents via UI (Monaco editor) with auto-upgrade
|
||
✅ **Token-Based Agent Auth** - Secure token management with revoke/renew
|
||
✅ **IP Inventory** - Cross-cluster IP search to identify agents, VIPs, and backend servers by IP
|
||
✅ **HA / VIP (Keepalived) Management** - Create virtual IPs from the UI; the agent installs & configures Keepalived (unicast VRRP) with a HAProxy health-check so the VIP fails over automatically; live MASTER/BACKUP detection per node
|
||
✅ **Role-Based User Management** - Admin and user roles with granular permissions and access control
|
||
✅ **User Activity Audit Logs** - Complete audit trail of all system events
|
||
✅ **Request/Response Log** *(v1.11.0)* - Every inbound API call (GETs and errors included) and every outbound HTTP call the backend makes (ACME, Cloudflare, GoDaddy, agents) in one filterable timeline, with redacted, size-capped bodies and operator-configurable retention
|
||
✅ **REST API** - Full programmatic access for automation and CI/CD integration
|
||
|
||
|
||
## Screenshots
|
||
|
||
|
||
### Multi-Cluster & Agent Setup
|
||
|
||
**Multi-Cluster Management** - Cluster selector dropdown showing multiple clusters with agent pools and online agent counts - switch between different clusters
|
||
|
||
|
||
**Agent Management** - Agent setup wizard with platform selection (Linux/macOS) and architecture (x86_64/ARM64)
|
||
|
||
|
||
**Agent Management** - Generated installation script ready for deployment to target servers
|
||
|
||
|
||
### Dashboard - Real-time Monitoring
|
||
|
||
**Dashboard** - Overview tab showing system metrics, active servers, SSL certificates, and cluster agent health
|
||
|
||
|
||
**Dashboard** - Performance Trends with request rate and response time heatmap (24h view)
|
||
|
||
|
||
**Dashboard** - Health Matrix showing backend servers with detailed health check status and response times
|
||
|
||
|
||
**Dashboard** - Response Time Heatmap (24h) with color-coded latency visualization per backend
|
||
|
||
|
||
### HAProxy Configuration
|
||
|
||
**Frontend Management** - List view with SSL/TLS bindings, sync status, and config status
|
||
|
||
|
||
**Frontend Management** - Edit modal for configuring bind address, protocol mode, default backend, SSL/TLS, ACLs, and advanced options
|
||
|
||
|
||
**Backend Configuration** - Backend server and pool configuration page
|
||
|
||
|
||
**Backend Management** - Backends & Servers tab with 165 servers across 56 backends, health status overview
|
||
|
||
|
||
### Configuration Import & Viewer
|
||
|
||
**Bulk Config Import** - Paste existing haproxy.cfg and parse configuration
|
||
|
||
|
||
**Bulk Config Import** - Parsed entities showing 56 frontends, 55 backends, 165 servers with SSL and configuration recommendations
|
||
|
||
|
||
**Configuration Viewer** - Real-time configuration viewer to pull haproxy.cfg from live agents
|
||
|
||
|
||
### Apply Management & Version Control
|
||
|
||
**Apply Management** - Pending changes and deployment history with version tracking
|
||
|
||
|
||
**Version History** - Configuration diff viewer showing added/removed lines with summary
|
||
|
||
|
||
**Version History** - Complete version history with restore capabilities and deployment status
|
||
|
||
|
||
### Security & Certificate Management
|
||
|
||
**Security Management** - Agent API token management with active tokens, expiration tracking, and revoke capabilities
|
||
|
||
|
||
### ACME Automation - Automated SSL Certificates
|
||
|
||
**ACME Automation** - Setup guide with prerequisites checklist, certificate statistics, and order management
|
||
|
||
|
||
**ACME Automation** - Active certificate orders with PENDING status tracking and renewal schedule
|
||
|
||
|
||
**ACME Settings** - ACME/SSL Automation configuration with provider selection, staging mode, and auto-renewal settings
|
||
|
||
|
||
### Site Wizard - Guided Multi-Step Host Setup
|
||
|
||
**New Site Wizard** - Multi-step guided form for creating a complete proxied host (domains + backend pool + servers + frontend + SSL/ACME) with live HAProxy config validation and atomic apply
|
||
|
||
|
||
### WAF Management
|
||
|
||
**WAF Management** - WAF rule configuration with request filtering, rate limiting, and advanced options
|
||
|
||
|
||
### IP Inventory - Cross-Cluster IP Search
|
||
|
||
**IP Inventory** - Unified view of all IPs across clusters with search capability to identify which cluster, agent, or backend server an IP belongs to
|
||
|
||
|
||
## Key Capabilities
|
||
|
||
#### Configuration & Entity Management
|
||
- **Frontend Management**: Create, edit, update, and delete frontend configurations with SSL bindings, ACL rules, and routing
|
||
- **Backend Management**: Full CRUD operations for backend pools with load balancing algorithms and health checks
|
||
- **Server Management**: Add, update, remove, and configure backend servers with weights, maintenance mode, and connection limits
|
||
- **Bulk Configuration Import**: Import existing `haproxy.cfg` files to instantly manage all frontends, backends, and servers without manual recreation
|
||
- **Smart SSL Auto-Assignment**: Automatically matches and assigns SSL certificates during bulk import if they exist in SSL Management with matching names and SYNCED status - eliminates manual SSL configuration after import
|
||
- **Configuration Synchronization**: All entities (backends, frontends, servers, SSL certificates, WAF rules) auto-sync to agents
|
||
- **Configuration Viewer**: View live HAProxy configuration pulled from agents via Configuration Management page
|
||
|
||
#### Agent & Cluster Management
|
||
- **Agent-Based Management**: Lightweight agent service for secure, pull-based HAProxy management
|
||
- **Pool Management**: Organize agents into logical pools for better cluster organization
|
||
- **Multi-Cluster Support**: Manage multiple HAProxy clusters from a single interface with cluster selector
|
||
- **Agent Token Management**: Secure token-based agent installation with token revoke and renew capabilities
|
||
- **Agent Script Editor**: Monaco code editor for updating agent scripts via UI (not binary) - create new versions, rollback, and auto-upgrade all agents
|
||
- **Platform-Specific Scripts**: Auto-generated installation scripts for Linux and macOS (x86_64/ARM64)
|
||
- **HA / VIP (Keepalived) Management**: Create a virtual IP from the UI and the agents install & configure Keepalived (VRRP) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops — opt-in, with live MASTER/BACKUP per node, multi-distro install, cluster-driven config path, and Apply/Reject staging
|
||
|
||
#### Version Control & Change Management
|
||
- **Apply Management**: Centralized change tracking and deployment status monitoring across all agents
|
||
- **Version Control**: Every configuration change creates a versioned snapshot with complete history
|
||
- **Configuration Restore**: Restore any previous configuration version with one-click rollback capability
|
||
- **Diff Visualization**: See exactly what changed between versions with line-by-line comparison
|
||
- **Change Tracking**: Track who made changes, when, and what entities were affected
|
||
|
||
#### Monitoring & Security
|
||
- **Real-time Monitoring**: Live statistics, health checks, and performance metrics from HAProxy stats socket
|
||
- **SSL Management**: Centralized certificate management with expiration tracking and zero-downtime updates
|
||
- **WAF Rules**: Web Application Firewall rule management and deployment for security protection
|
||
- **User Management**: Role-based access control with admin and user roles
|
||
- **User Activity Logs**: Complete audit trail of all user actions, configuration changes, and system events
|
||
|
||
#### ACME Auto SSL / Let's Encrypt
|
||
- **Automated Certificate Issuance**: Request SSL certificates from Let's Encrypt or any ACME-compatible CA directly from the UI
|
||
- **Automatic Renewal**: Hourly background task monitors certificate expiry and creates renewal orders before expiration
|
||
- **Auto-Completion of Validated Orders** *(v1.4.0)*: Independent 60-second background task drives CA-validated orders through finalize -> download -> save without requiring user intervention; multi-replica safe via PostgreSQL `FOR UPDATE SKIP LOCKED` atomic claim, plus per-order session-level `pg_advisory_lock` to serialize concurrent completion attempts (UI "Complete" + auto-task race protection)
|
||
- **Zero-Touch Deployment**: Renewed certificates are automatically applied through the same PENDING -> APPLIED pipeline as manual SSL updates, with agent notification
|
||
- **Stuck Order Detection** *(v1.4.0)*: Setup wizard surfaces orders that the CA has validated but not yet downloaded, with one-click `Complete` action and automatic 60-second retry
|
||
- **Multi-Provider Support**: Configurable ACME directory URL supports Let's Encrypt, ZeroSSL, Google Trust Services, Buypass, and custom CAs
|
||
- **HTTP-01 Challenge**: Built-in challenge responder with automatic HAProxy routing injection; reserved backend name `_acme_challenge_backend` is auto-managed and protected from manual edits / agent sync collisions
|
||
- **DNS-01 Challenge** *(v1.8.0 — Issue #35)*: Validate via a DNS TXT record instead of HTTP on port 80, for **internal/isolated clusters with no public ingress** and for **wildcard** certificates (`*.example.com`). Pluggable per-account DNS provider (Manual + Cloudflare + GoDaddy *(v1.10.0)*; credentials encrypted at rest and verified on save), same PENDING → APPLIED pipeline, bounded automatic retry on propagation lag, and a DNS-01 event timeline. Opt-in via a global setting; HTTP-01 behaviour is unchanged. (See the *DNS-01 Challenge* subsection under ACME Auto SSL below.)
|
||
- **ACME Account Management**: Register, view, and deactivate ACME accounts from the UI
|
||
- **Staging Mode**: Test certificate issuance with Let's Encrypt staging environment before production
|
||
- **Custom Staging Endpoint** *(v1.4.0)*: Optional `staging_url_override` setting lets you point staging mode at a private ACME test CA (e.g. Pebble) without touching the production directory URL
|
||
- **External Account Binding (EAB)**: Support for CAs that require EAB (ZeroSSL, Google Trust Services). Enter the EAB Key ID and HMAC Key globally in Settings, or per-account in the Register Account dialog (a per-account value overrides the global setting; leave it blank to use the global one)
|
||
- **Structured Error Diagnostics** *(v1.4.0)*: All ACME failures (challenge, finalize, download) persist structured JSON to `letsencrypt_orders.error_detail` for clear post-mortem analysis
|
||
- **Audit Logging** *(v1.4.0)*: Every ACME operation (request, revoke, CA-chain import, account ops) is captured in `user_activity_logs` for compliance review
|
||
- **ACME Diagnostic Panel** *(v1.5.0 — Issue #13)*: Live pre-flight + post-failure diagnostics (DNS / port-80 / routing / account / agents) and merged event timeline (`acme_order_events` + correlated `user_activity_logs`) accessible from the ACME Automation page; humanized error rendering for 11+ RFC8555 problem types with backwards-compatible fallback for legacy plain-string `error_detail`; per-user 5/min rate-limit
|
||
- **New Site Setup Wizard** *(v1.5.0 — Issue #14)*: Single guided flow that creates a Backend + Servers + HTTP Frontend (and optional HTTPS Frontend with chosen SSL mode: ACME / Upload / Existing / None) in one atomic transaction, with diff preview, draft persistence (PEM stripped), and ACME-staged order completion gated by agent confirmation. Per-mode reject path cleanly rolls back including any wizard-staged ACME orders.
|
||
- **Backward Compatible**: ACME-managed and manually uploaded certificates coexist seamlessly; existing SSL workflows are completely unaffected
|
||
|
||
#### Integration & API
|
||
- **REST API**: Complete API for programmatic access and integration with CI/CD pipelines
|
||
- **Agent Pull Architecture**: Secure polling model - no inbound connections required to HAProxy servers
|
||
- **Redis Cache**: High-performance metrics caching for dashboard with configurable retention
|
||
|
||
|
||
## Architecture
|
||
|
||
### System Architecture with Agent Pull Model
|
||
|
||
```mermaid
|
||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'16px'}}}%%
|
||
graph TB
|
||
subgraph UI_LAYER["🌐 User Interface Layer"]
|
||
USER["👤 Admin/User"] --> WEB["React UI<br/>Nginx Proxy<br/>:8080"]
|
||
end
|
||
|
||
subgraph BACKEND_LAYER["⚙️ Backend Layer"]
|
||
WEB <--> API["FastAPI<br/>Backend<br/>:8000"]
|
||
API <--> DB[("PostgreSQL<br/>Configs & History")]
|
||
API <--> CACHE[("Redis<br/>Metrics Cache")]
|
||
end
|
||
|
||
subgraph AGENT_LAYER["📦 Agent Pool: Production Cluster"]
|
||
direction LR
|
||
|
||
AGENT1["🤖 Agent 1<br/>10.0.1.101<br/>(Poll Every 30s)"]
|
||
AGENT2["🤖 Agent 2<br/>10.0.1.102<br/>(Poll Every 30s)"]
|
||
AGENT3["🤖 Agent 3<br/>10.0.1.103<br/>(Poll Every 30s)"]
|
||
|
||
HAP1["⚡ HAProxy<br/>+ Stats Socket<br/>+ haproxy.cfg"]
|
||
HAP2["⚡ HAProxy<br/>+ Stats Socket<br/>+ haproxy.cfg"]
|
||
HAP3["⚡ HAProxy<br/>+ Stats Socket<br/>+ haproxy.cfg"]
|
||
|
||
AGENT1 --> HAP1
|
||
AGENT2 --> HAP2
|
||
AGENT3 --> HAP3
|
||
end
|
||
|
||
API <-.->|"🔄 Pull Config<br/>📤 Push Stats<br/>⬇️ SSL/WAF"| AGENT1
|
||
API <-.->|"🔄 Pull Config<br/>📤 Push Stats<br/>⬇️ SSL/WAF"| AGENT2
|
||
API <-.->|"🔄 Pull Config<br/>📤 Push Stats<br/>⬇️ SSL/WAF"| AGENT3
|
||
|
||
classDef uiStyle fill:#3a5a8a,stroke:#6a9eff,stroke-width:3px,color:#fff
|
||
classDef backendStyle fill:#5a3a8a,stroke:#b39ddb,stroke-width:2px,color:#fff
|
||
classDef dbStyle fill:#3a6a3a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
classDef agentStyle fill:#8a6a30,stroke:#ffb347,stroke-width:2px,color:#fff
|
||
classDef haproxyStyle fill:#2a6a5a,stroke:#4db6ac,stroke-width:3px,color:#fff
|
||
|
||
class USER,WEB uiStyle
|
||
class API backendStyle
|
||
class DB,CACHE dbStyle
|
||
class AGENT1,AGENT2,AGENT3 agentStyle
|
||
class HAP1,HAP2,HAP3 haproxyStyle
|
||
```
|
||
|
||
### Component Details
|
||
|
||
| Component | Technology | Purpose | Port |
|
||
|-----------|------------|---------|------|
|
||
| **Frontend** | React.js 18 + Ant Design | Web interface, dashboards, configuration UI | 3000 |
|
||
| **Backend** | FastAPI + Python | REST API, task queue, agent coordination | 8000 |
|
||
| **Database** | PostgreSQL 15 | Configuration storage, user management, agent tasks | 5432 |
|
||
| **Cache** | Redis 7 | Session storage, performance metrics caching | 6379 |
|
||
| **Reverse Proxy** | Nginx | Frontend/backend routing, SSL termination, ACME challenge routing | 8080 |
|
||
| **HAProxy Agent** | Bash Service | Polls backend, applies configs, manages HAProxy | N/A |
|
||
| **HAProxy Instances** | HAProxy 2.8+ | Load balancer instances being managed by agents | 8404+ |
|
||
| **ACME Service** | Built-in (Python) | ACME protocol client for automated certificate management | - |
|
||
|
||
### Agent Pull Communication Flow - Detailed Entity Management
|
||
|
||
This detailed sequence diagram shows the complete lifecycle of configuration management, including entity creation, agent polling, configuration application, SSL management, WAF rules, and agent upgrade process.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
participant U as 👤 User Browser
|
||
participant B as ⚙️ Backend API
|
||
participant D as 💿 PostgreSQL DB
|
||
participant R as 💾 Redis Cache
|
||
participant A as 🤖 Agent
|
||
participant CFG as 📄 haproxy.cfg
|
||
participant H as ⚡ HAProxy Service
|
||
participant S as 🔌 Stats Socket
|
||
|
||
Note over U,B: ═══ PHASE 1: USER CREATES ENTITIES ═══
|
||
|
||
U->>B: Create Backend "api-backend"
|
||
B->>D: INSERT backends + servers
|
||
B->>D: CREATE version v1.0.124
|
||
B-->>U: ✅ Backend Created
|
||
|
||
U->>B: Create Frontend "web-frontend"
|
||
B->>D: INSERT frontend + SSL binding
|
||
B->>D: UPDATE version v1.0.125
|
||
B-->>U: ✅ Frontend Created
|
||
|
||
U->>B: Upload SSL Certificate
|
||
B->>D: INSERT ssl_certificates (global)
|
||
B-->>U: ✅ SSL Ready for all clusters
|
||
|
||
U->>B: Create WAF Rule
|
||
B->>D: INSERT waf_rules (rate limit)
|
||
B-->>U: ✅ WAF Rule Created
|
||
|
||
U->>B: 🚀 Click "Apply Changes"
|
||
B->>D: Mark tasks READY_FOR_DEPLOYMENT
|
||
B-->>U: ✅ Queued for deployment
|
||
|
||
Note over A,R: ═══ PHASE 2: AGENT POLLING CYCLE (30s) ═══
|
||
|
||
loop Every 30 seconds
|
||
A->>B: GET /api/agent/tasks
|
||
B->>D: Query pending tasks
|
||
B-->>A: Tasks + Version v1.0.125
|
||
|
||
A->>S: socat show stat
|
||
S-->>A: CSV Stats
|
||
A->>B: POST /api/stats/upload
|
||
B->>R: Store metrics (5min TTL)
|
||
|
||
A->>B: Check agent version
|
||
B->>D: SELECT latest_agent_version
|
||
B-->>A: Current: 1.0.10, Latest: 1.0.11
|
||
end
|
||
|
||
Note over A,CFG: ═══ PHASE 3: AGENT APPLIES CONFIG ═══
|
||
|
||
A->>A: Parse task bundle
|
||
A->>A: Generate backend config
|
||
A->>A: Generate frontend config
|
||
|
||
A->>B: Download SSL certificate
|
||
B-->>A: PEM file (encrypted)
|
||
A->>A: Write /etc/haproxy/certs/*.pem
|
||
A->>A: chmod 600 (secure)
|
||
|
||
A->>A: Generate WAF stick-table
|
||
|
||
A->>CFG: Backup haproxy.cfg
|
||
A->>CFG: Write new configuration
|
||
A->>H: haproxy -c -f haproxy.cfg
|
||
|
||
alt Config Valid
|
||
A->>H: systemctl reload haproxy
|
||
H-->>A: ✅ Reload successful
|
||
A->>CFG: Keep new config
|
||
else Config Invalid
|
||
A->>CFG: Restore backup
|
||
A->>A: ❌ Rollback complete
|
||
end
|
||
|
||
Note over A,B: ═══ PHASE 4: AGENT REPORTS STATUS ═══
|
||
|
||
A->>B: POST /api/agent/tasks/complete
|
||
B->>D: UPDATE task status = COMPLETED
|
||
B->>D: INSERT activity_logs (JSON)
|
||
B-->>A: ✅ Acknowledged
|
||
|
||
Note over U,B: ═══ PHASE 5: USER MONITORS STATUS ═══
|
||
|
||
U->>B: GET /api/apply-changes/status
|
||
B->>D: Query all agent statuses
|
||
B-->>U: 3/3 agents ✅ v1.0.125
|
||
|
||
Note over U,R: ═══ PHASE 6: DASHBOARD METRICS ═══
|
||
|
||
U->>B: GET /api/dashboard/stats
|
||
B->>R: GET haproxy:cluster:5:stats
|
||
R-->>B: Latest metrics (30s fresh)
|
||
B-->>U: Real-time dashboard data
|
||
|
||
Note over A,B: ═══ PHASE 7: AGENT AUTO-UPGRADE ═══
|
||
|
||
A->>B: Check version (in polling)
|
||
B-->>A: New version 1.0.11 available
|
||
|
||
A->>B: GET /api/agent/download/1.0.11
|
||
B->>D: SELECT agent_script WHERE version=1.0.11
|
||
B-->>A: Download script (bash/python)
|
||
|
||
A->>A: Write new script to /tmp/agent-upgrade.sh
|
||
A->>A: chmod +x agent-upgrade.sh
|
||
A->>A: Execute: /tmp/agent-upgrade.sh
|
||
|
||
Note over A: Agent Self-Upgrade Process
|
||
A->>A: Stop current agent service
|
||
A->>A: Backup old agent binary
|
||
A->>A: Install new agent version
|
||
A->>A: Update systemd service
|
||
A->>A: Start agent with new version
|
||
|
||
A->>B: POST /api/agent/upgrade/complete
|
||
B->>D: UPDATE agent version = 1.0.11
|
||
B->>D: LOG upgrade activity
|
||
B-->>A: ✅ Upgrade confirmed
|
||
|
||
A->>CFG: Read haproxy.cfg (verify integrity)
|
||
A->>H: Check HAProxy service status
|
||
H-->>A: Service running normally
|
||
|
||
Note over A,B: Agent now running v1.0.11<br/>Continues normal polling cycle
|
||
|
||
Note over B,D: ═══ PHASE 8: ACME AUTO-RENEWAL (Hourly) ═══
|
||
|
||
B->>D: Query expiring ACME certs (within N days)
|
||
D-->>B: Cert "example.com" expires in 25 days
|
||
|
||
B->>B: Create ACME renewal order
|
||
B->>B: Respond to HTTP-01 challenges
|
||
|
||
Note over B: Next cycle: poll order status
|
||
B->>B: Order valid - download certificate
|
||
B->>D: Update cert content (PENDING)
|
||
B->>D: Create config versions (PENDING)
|
||
B->>D: Auto-apply: apply_ssl_related_configs()
|
||
B->>D: Create consolidated version (APPLIED)
|
||
B->>R: Notify agents via Redis
|
||
|
||
A->>B: Poll detects new config version
|
||
A->>B: Download renewed certificate
|
||
A->>A: Write updated cert to disk
|
||
A->>H: Reload HAProxy (zero downtime)
|
||
A->>B: Report success
|
||
```
|
||
|
||
**Key Flow Features:**
|
||
- 🔄 **Continuous Polling**: Agents poll every 30s for tasks, stats collection, and version checks
|
||
- 📄 **Config Management**: haproxy.cfg backup → validate → apply → rollback if needed
|
||
- 🔐 **SSL Handling**: Secure download, file permissions (chmod 600), automatic frontend binding
|
||
- 🤖 **ACME Auto-Renewal**: Automatic certificate renewal via ACME protocol with same Apply pipeline as manual SSL
|
||
- 🛡️ **WAF Integration**: Stick-tables generated and integrated into config
|
||
- 📊 **Stats Collection**: Parallel stats upload using socat + stats socket
|
||
- 🔄 **Auto-Upgrade**: Agents detect new versions and self-upgrade with zero-touch deployment
|
||
- ✅ **Validation**: Every config change is validated before HAProxy reload
|
||
- 📝 **Activity Logging**: All operations logged in JSON format for audit trail
|
||
|
||
|
||
## Features & User Interface
|
||
|
||
### Pool Management - Agent Pool Organization
|
||
- **Pool Creation**: Create logical pools to group HAProxy agents
|
||
- **Pool Assignment**: Associate clusters with specific pools
|
||
- **Pool Overview**: View all agents within a pool
|
||
- **Multi-Pool Support**: Manage multiple pools for different environments (prod, staging, etc.)
|
||
- **Pool-based Filtering**: Filter agents and clusters by pool
|
||
|
||
### Cluster Management - Multi-Cluster HAProxy Administration
|
||
- **Cluster Selector**: Top navigation cluster selector for context switching
|
||
- **Cluster Creation**: Define clusters and associate with agent pools
|
||
- **Cluster Status**: Monitor cluster health and connectivity
|
||
- **Default Cluster**: Set preferred default cluster
|
||
- **Cross-Cluster View**: Compare metrics across multiple clusters
|
||
- **Cluster-Scoped Operations**: All configurations are cluster-specific
|
||
|
||
### Agent Management - HAProxy Agent Deployment & Monitoring
|
||
|
||
#### Agent Script Management
|
||
Agent scripts are **not hardcoded** in the project. They are stored in the database and can be updated via the UI:
|
||
|
||
- **Script Editor**: Edit agent installation scripts directly in the UI
|
||
- **Platform Support**: Separate scripts for Linux and macOS (x86_64/ARM64)
|
||
- **Version Control**: Create new agent versions with changelog
|
||
- **Automatic Upgrade**: Deploy new agent versions to all agents with one click
|
||
- **Upgrade Process**: Agents check backend version → Download new script → Self-upgrade → Restart
|
||
|
||
**Agent Version Update Flow:**
|
||
|
||
```mermaid
|
||
%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%%
|
||
graph LR
|
||
subgraph USER["👤 User Actions"]
|
||
A["Edit Script<br/>+ Create v1.0.5"] --> B["Mark as<br/>Latest"]
|
||
end
|
||
|
||
subgraph AUTO["🤖 Automatic Process"]
|
||
C["Agents Poll"] --> D["Detect New<br/>Version"] --> E["Download"] --> F["Self-Upgrade"] --> G["✅ Running<br/>v1.0.5"]
|
||
end
|
||
|
||
B -.->|Trigger| C
|
||
|
||
style USER fill:#2a4a6a,stroke:#4a9eff,stroke-width:2px,color:#fff
|
||
style AUTO fill:#3a5a3a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
style A fill:#3a5a8a,stroke:#6a9eff,stroke-width:2px,color:#fff
|
||
style B fill:#3a5a8a,stroke:#6a9eff,stroke-width:2px,color:#fff
|
||
style C fill:#4a6a4a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
style D fill:#4a6a4a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
style E fill:#4a6a4a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
style F fill:#4a6a4a,stroke:#66bb6a,stroke-width:2px,color:#fff
|
||
style G fill:#2a5a2a,stroke:#66bb6a,stroke-width:3px,color:#fff
|
||
```
|
||
|
||
#### Agent Operations
|
||
- **Agent Registration**: Automatic agent registration after installation
|
||
- **Installation Scripts**: Generate platform-specific scripts (Linux/macOS ARM/x86)
|
||
- **Script Download**: One-click download of customized installation scripts
|
||
- **Agent Status**: Real-time agent connectivity and health monitoring
|
||
- **Platform Detection**: Automatic detection of OS and architecture
|
||
- **Last Seen**: Track when agents last connected to backend (updated every 30s)
|
||
- **Batch Operations**: Upgrade multiple agents simultaneously
|
||
|
||
#### Agent Logs & Monitoring
|
||
- **Log Location**: `/var/log/haproxy-agent/agent.log` on each agent server
|
||
- **Log Format**: Structured JSON for easy parsing and log aggregation
|
||
- **Log Fields**: timestamp, level, message, agent_id, cluster_id, task_id
|
||
- **Activity Tracking**: All agent activities (config updates, SSL changes, task execution)
|
||
- **Log Integration**: JSON format allows easy integration with ELK, Splunk, Datadog
|
||
- **Log Rotation**: Automatic rotation with configurable retention
|
||
|
||
**Example Log Entry (JSON):**
|
||
```json
|
||
{
|
||
"timestamp": "2025-01-15T10:30:45Z",
|
||
"level": "INFO",
|
||
"agent_id": "prod-haproxy-01",
|
||
"cluster_id": 5,
|
||
"task_id": 123,
|
||
"message": "SSL certificate updated successfully",
|
||
"details": {
|
||
"cert_name": "wildcard.example.com",
|
||
"action": "update",
|
||
"haproxy_reload": "success"
|
||
}
|
||
}
|
||
```
|
||
|
||
### Dashboard - Main Overview & Real-time Monitoring
|
||
|
||
The dashboard displays comprehensive real-time metrics collected by agents from HAProxy stats socket. Agents use `socat` to query the stats socket every 30 seconds and send CSV data to the backend, which parses and stores it in Redis for high-performance access.
|
||
|
||
**Tab-Based Navigation:**
|
||
|
||
1. **Overview Tab**:
|
||
- System-wide metrics: Total requests, active sessions, error rate
|
||
- Frontend/Backend/Server counts
|
||
- SSL certificate status and expiration warnings
|
||
- Quick health status indicators
|
||
|
||
2. **Frontends Tab**:
|
||
- Request rate per frontend with sparkline charts
|
||
- Session distribution across frontends
|
||
- Status (UP/DOWN) monitoring
|
||
- Filter by specific frontends
|
||
|
||
3. **Backends Tab**:
|
||
- Backend health matrix (all servers at a glance)
|
||
- Request distribution percentages
|
||
- Response time averages
|
||
- Active/backup server status
|
||
|
||
4. **Performance Trends Tab**:
|
||
- Response time trends (1h, 6h, 24h, 7d, 30d views)
|
||
- Request rate trends with interactive zoom
|
||
- Error rate analysis over time
|
||
- Session trends and capacity planning
|
||
|
||
5. **Capacity & Load Tab**:
|
||
- Current vs max connections
|
||
- Queue depth monitoring
|
||
- CPU/Memory utilization (if available)
|
||
- Load distribution heatmaps
|
||
|
||
6. **Health Matrix Tab**:
|
||
- Detailed server health status
|
||
- Last status check timestamps
|
||
- Check duration and latency
|
||
- Failure reasons and diagnostics
|
||
|
||
**Data Collection Flow:**
|
||
- Agents → Stats Socket (`show stat`) → CSV → Backend API → Redis → Dashboard UI
|
||
- **Agent Log Path**: `/var/log/haproxy-agent/agent.log` (JSON format for easy log aggregation)
|
||
- **Update Interval**: 30 seconds (configurable)
|
||
- **Data Retention**: 30 days in Redis (configurable)
|
||
|
||
### Frontend Management - Virtual Host & Routing Configuration
|
||
- **Frontend CRUD**: Create, edit, delete, and duplicate frontend configurations
|
||
- **Protocol Support**: HTTP, TCP, and health check mode configurations
|
||
- **Binding Configuration**: IP address, port, and SSL binding options
|
||
- **Backend Assignment**: Default backend selection and routing rules
|
||
- **SSL Configuration**: Certificate assignment and SSL protocol settings
|
||
- **ACL Rules**: Access control rules with pattern matching
|
||
- **Redirect Rules**: HTTP to HTTPS redirects and custom redirections
|
||
- **Advanced Options**: Connection limits, timeouts, and performance tuning
|
||
- **Search & Filter**: Real-time search and status-based filtering
|
||
|
||
### New Site Wizard - Guided Multi-Step Host Setup
|
||
|
||
The New Site Wizard is the recommended entry point for creating a complete proxied host. It collects every piece of configuration a host needs across a five-step guided form and creates the matching frontend + backend + servers + SSL binding as a single atomic apply.
|
||
|
||
- **5-step guided flow**:
|
||
1. **Domains** - host header / SNI / wildcard validation, IDN/Punycode normalization, port + bind-address collision check
|
||
2. **Backend** - pool name, load-balance algorithm, health check method/URI, cookie-based persistence (RFC 6265 and HAProxy-parser-safe character validation)
|
||
3. **Servers** - per-server address/port/weight, optional CA bundle reference (cluster-RBAC enforced), backup-server + cookie-value validators
|
||
4. **SSL** - three modes: existing certificate (cluster-RBAC), PEM upload (chain validation + SAN/CN match), or ACME order (account binding + preflight)
|
||
5. **Review + Dry-Run** - live `POST /api/sites/preview` runs the proposed config through `haproxy -c` and surfaces every WARNING/ERROR with a marker comment that pinpoints the offending block before anything touches the database
|
||
- **Draft persistence**: every step auto-saves to a server-side draft (30-day TTL, 50 drafts per user cap, cluster-scoped, PEM material stripped at rest)
|
||
- **Resume across sessions**: drafts can be reopened from a different browser; cluster swap mid-wizard surfaces a confirmation prompt to prevent cross-cluster contamination
|
||
- **Atomic apply**: frontend + backend + servers + SSL binding land as a single PENDING change reviewable from Apply Management - accept and apply, or reject as a whole
|
||
- **Cluster RBAC**: every step honours `user_pool_access` and cluster RBAC; SSL certificate references are validated against the target cluster at both preview AND create time
|
||
- **Hardening**: IDN/Punycode safe slug generation, cookie/header injection guards, path-traversal protection on SSL filenames, per-user-per-minute rate limit on create + preview, advisory locks against concurrent creation
|
||
- **Backwards compatible URLs**: legacy `/proxied-hosts/new` route continues to resolve to the new wizard
|
||
|
||
### Backend Servers - Server Pool Management
|
||
- **Server Management**: Add, edit, remove, and configure backend servers
|
||
- **Health Checks**: HTTP/TCP health check configuration and monitoring
|
||
- **Load Balancing**: Round-robin, least-conn, source, and URI algorithms
|
||
- **Server Weights**: Dynamic weight adjustment for load distribution
|
||
- **Maintenance Mode**: Enable/disable servers without removing configuration
|
||
- **Backup Servers**: Failover server configuration for high availability
|
||
- **Connection Limits**: Per-server connection and rate limiting
|
||
- **Monitoring Dashboard**: Real-time server status and performance metrics
|
||
|
||
### Configuration - HAProxy Config File Management
|
||
- **Syntax Editor**: Monaco editor with HAProxy syntax highlighting
|
||
- **Real-time Validation**: Configuration syntax checking and error detection
|
||
- **Configuration Templates**: Pre-built templates for common setups
|
||
- **Version Control**: Configuration versioning and rollback capabilities
|
||
- **Backup & Restore**: Automatic backups and manual restore functionality
|
||
- **Direct Editing**: Raw HAProxy configuration file editing
|
||
- **Deployment**: Apply configuration changes with validation
|
||
- **Help System**: Integrated HAProxy directive documentation
|
||
- **Real-time Config Pull**: Pull current haproxy.cfg from live HAProxy servers
|
||
- View actual running configuration
|
||
- Compare with managed configuration
|
||
- Import existing configurations
|
||
- Useful for migrating from manual to managed setup
|
||
|
||
### Apply Management - Change Tracking & Deployment
|
||
|
||
#### Change Tracking & Versioning
|
||
Every configuration change creates a new version with complete tracking:
|
||
|
||
- **Automatic Versioning**: Each apply creates a new version number (e.g., v1.0.123)
|
||
- **Version Snapshots**: Complete configuration state saved for each version
|
||
- **Diff Visualization**: See exactly what changed between versions
|
||
- Green: Added lines
|
||
- Red: Removed lines
|
||
- Yellow: Modified lines
|
||
- **Change Summary**: High-level summary of changes (added frontends, modified backends, etc.)
|
||
- **Version Metadata**: Timestamp, user, cluster, affected entities
|
||
|
||
#### Deployment & Status
|
||
- **Change Tracking**: Track all pending configuration changes before apply
|
||
- **Apply Status**: Real-time deployment status per agent
|
||
- **Bulk Apply**: Deploy changes to all agents in a cluster simultaneously
|
||
- **Success/Failure Tracking**: Monitor which agents successfully applied changes
|
||
- **Retry Mechanism**: Retry failed deployments per agent
|
||
- **Apply Logs**: Detailed logs for each deployment operation with error messages
|
||
|
||
#### Version History & Restore
|
||
|
||
**Version History Viewer:**
|
||
- View all historical configuration versions
|
||
- Compare any two versions side-by-side
|
||
- See who made changes and when
|
||
- Filter by date, user, or cluster
|
||
|
||
**Configuration Restore:**
|
||
- Restore any previous configuration version
|
||
- One-click rollback to last known good configuration
|
||
- Restore process:
|
||
1. Select previous version from history
|
||
2. Review diff (current vs target version)
|
||
3. Confirm restore
|
||
4. Agents pull restored configuration
|
||
5. HAProxy reloaded with previous config
|
||
|
||
**Use Cases:**
|
||
- Rollback after problematic deployment
|
||
- Audit configuration changes
|
||
- Compare production vs staging configurations
|
||
- Disaster recovery
|
||
|
||
### SSL Certificates - Centralized TLS/SSL Certificate Management
|
||
|
||
HAProxy OpenManager provides powerful centralized SSL/TLS certificate management with both global and cluster-specific scopes:
|
||
|
||
#### SSL Certificate Scopes
|
||
|
||
**1. Global SSL Certificates:**
|
||
- Defined once, distributed to **all clusters** and **all agents**
|
||
- Perfect for wildcard certificates or shared certificates
|
||
- When updated, all agents across all clusters automatically receive the new certificate
|
||
- Use case: `*.example.com` certificate used across all environments
|
||
|
||
**2. Cluster-Specific SSL Certificates:**
|
||
- Scoped to a specific cluster only
|
||
- Only agents in that cluster receive these certificates
|
||
- Use case: Environment-specific certificates (prod, staging, dev)
|
||
|
||
#### Certificate Operations
|
||
|
||
**Upload & Distribution:**
|
||
1. User uploads certificate via UI (PEM format: cert + private key)
|
||
2. Backend stores certificate in database
|
||
3. Agents poll backend and detect new/updated certificates
|
||
4. Agents download certificates to local disk (`/etc/haproxy/certs/`)
|
||
5. Agents update `haproxy.cfg` frontend bindings
|
||
6. HAProxy is safely reloaded with zero downtime
|
||
|
||
**Binding to Frontends:**
|
||
- SSL certificates are automatically bound to frontends based on configuration
|
||
- Frontend definition specifies which certificate to use
|
||
- Format: `bind *:443 ssl crt /etc/haproxy/certs/cert-name.pem`
|
||
|
||
**Centralized SSL Update Process:**
|
||
```
|
||
User Updates SSL in UI → All Agents Poll Backend (30s)
|
||
→ Download New Certificate → Update Frontend Bindings
|
||
→ Validate Config → Reload HAProxy (zero downtime)
|
||
```
|
||
|
||
#### CSR Workflow (external / corporate CAs) — v1.9.0
|
||
|
||
For certificates signed by an external or corporate CA, the **CSR tab** on the SSL Certificates page covers the whole flow without the private key ever leaving the server:
|
||
|
||
1. **Create CSR**: pick a name (becomes the certificate name / on-agent file path), Common Name, optional SANs and subject fields (O/OU/L/ST/C/email), and a key algorithm (RSA 2048/4096 or ECDSA P-256/P-384). The backend generates the key + CSR; only the CSR PEM is shown (copy or download as `.csr`).
|
||
2. **Get it signed**: submit the CSR to your Certificate Authority.
|
||
3. **Import**: paste the signed certificate (+ optional chain), choose Global or cluster-specific scope and usage type. The backend verifies the certificate matches the stored key, rejects expired certs, warns on SAN drift, and creates a normal SSL certificate entry (source: `CSR`).
|
||
4. **Deploy**: the imported certificate goes through the standard **PENDING → Apply Management → agent pull** pipeline like any other certificate.
|
||
|
||
#### Key Features
|
||
- **Certificate Upload**: PEM format certificate and private key upload
|
||
- **ACME Automation**: Automatic certificate issuance and renewal via Let's Encrypt / ACME protocol (see [ACME Auto SSL](#acme-auto-ssl---automated-certificate-management))
|
||
- **Certificate Store**: Centralized repository with encryption at rest
|
||
- **Expiration Monitoring**: Automatic expiration tracking with alerts (30, 15, 7 days)
|
||
- **Global Update**: Update one certificate, deploy to all clusters simultaneously
|
||
- **Domain Binding**: Associate certificates with specific domains/frontends
|
||
- **Certificate Validation**: Syntax and format validation before deployment
|
||
- **Renewal Tracking**: Certificate renewal status and history
|
||
- **Source Tracking**: Certificates display their source ("Manual" or "Auto (ACME)") for clear management
|
||
- **ACME Protection**: ACME-managed certificates are protected from manual content edits to prevent accidental overwrites
|
||
- **Security Profiles**: SSL/TLS protocol and cipher suite configuration
|
||
- **Zero Downtime**: Safe HAProxy reload ensures no dropped connections
|
||
|
||
### ACME Auto SSL - Automated Certificate Management
|
||
|
||
HAProxy OpenManager includes a built-in ACME client for automated SSL/TLS certificate management using the ACME protocol (RFC 8555). This enables zero-touch certificate issuance and renewal from Let's Encrypt and other ACME-compatible Certificate Authorities.
|
||
|
||
#### How ACME Auto SSL Works
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
participant U as Admin UI
|
||
participant B as Backend API
|
||
participant CA as ACME CA<br/>(Let's Encrypt)
|
||
participant H as HAProxy<br/>(via Agent)
|
||
|
||
Note over U,CA: === Certificate Request ===
|
||
U->>B: Request certificate for example.com
|
||
B->>CA: Create ACME order
|
||
CA-->>B: Order + HTTP-01 challenge token
|
||
B->>B: Store challenge token in DB
|
||
|
||
Note over CA,H: === Domain Validation ===
|
||
CA->>H: GET /.well-known/acme-challenge/{token}
|
||
H->>B: Proxy challenge request (ACME ACL)
|
||
B-->>H: Challenge response (key authorization)
|
||
H-->>CA: Challenge response
|
||
CA->>CA: Validate domain ownership
|
||
|
||
Note over B,CA: === Certificate Issuance ===
|
||
B->>CA: Finalize order (submit CSR)
|
||
CA-->>B: Signed certificate
|
||
B->>B: Store cert, status = PENDING
|
||
B->>B: Create config version (PENDING)
|
||
|
||
Note over U,B: === Manual Apply (New Certs) ===
|
||
U->>B: Click "Apply Changes"
|
||
B->>B: Mark APPLIED + consolidated config
|
||
B->>H: Notify agents
|
||
H->>H: Pull cert + reload HAProxy
|
||
```
|
||
|
||
#### Auto-Renewal Flow
|
||
|
||
Certificates are automatically renewed before expiration through a background task that runs hourly:
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
participant BG as Background Task<br/>(Hourly)
|
||
participant B as Backend
|
||
participant CA as ACME CA
|
||
participant DB as Database
|
||
participant H as HAProxy Agents
|
||
|
||
BG->>DB: Find certs expiring within N days
|
||
BG->>CA: Create renewal order
|
||
CA-->>BG: Challenge token
|
||
BG->>DB: Store challenge
|
||
|
||
Note over BG,CA: Next cycle (or same)
|
||
BG->>CA: Check order status
|
||
CA-->>BG: Status: valid
|
||
BG->>CA: Download renewed certificate
|
||
|
||
Note over BG,DB: Auto-Apply (same as manual Apply)
|
||
BG->>DB: Update cert (PENDING)
|
||
BG->>DB: Create config versions (PENDING)
|
||
BG->>DB: apply_ssl_related_configs()
|
||
BG->>DB: Create consolidated version (APPLIED)
|
||
BG->>DB: Mark cert APPLIED
|
||
BG->>H: Notify agents via Redis
|
||
H->>H: Pull updated cert + reload
|
||
```
|
||
|
||
**Key behavior**: Auto-renewed certificates follow the **exact same Apply pipeline** as manual SSL updates. The cert transitions `PENDING -> APPLIED` automatically, agents are notified, and cross-cluster propagation works identically to manual Apply. No separate deployment mechanism is used.
|
||
|
||
#### ACME Certificate Diagnostic Panel
|
||
|
||
Available from the ACME Automation list (`Diagnose` button on each order, or by clicking the order's status tag), the Diagnostic Panel runs a full preflight checklist before a certificate is issued or renewed and surfaces every blocker in a single operator-friendly view:
|
||
|
||
- **Agent reachability** - verifies that at least one agent in each target cluster is online and within the last-seen window
|
||
- **DNS resolution** - resolves every domain on the order (A / AAAA / CNAME) and flags wildcards and non-resolvable names with the exact upstream resolver error
|
||
- **Port 80 reachability** - checks that the HTTP-01 challenge port is reachable from the public internet via the agent's egress path
|
||
- **ACME challenge ACL preview** - renders the `acl1 !acl1` + `http-request return` block that will be injected at apply time, so the operator can see exactly what HAProxy will receive
|
||
- **HSTS / rate-limit policy collision check** - warns if existing frontend rules would short-circuit the challenge route
|
||
- **Account binding check** - confirms a valid ACME account exists for the selected provider + cluster combination
|
||
- **Event Log timeline** - merged view of typed `acme_order_events` rows and correlated `user_activity_logs` entries; auto-tails every 5s while the order is in `pending`/`processing`
|
||
- **Humanized error display** - covers 11+ RFC8555 problem types (`badNonce`, `caa`, `connection`, `rateLimited`, `unauthorized`, ...) with full backwards compatibility for legacy plain-string error details
|
||
- **Operator-friendly error envelope** - every failure returns `{ correlation_id, field, cause, remediation }` so root-causing is one API call away
|
||
|
||
#### ACME Features
|
||
|
||
| Feature | Description |
|
||
|---------|-------------|
|
||
| **Certificate Request Wizard** | Step-by-step UI wizard: select domains, account, target clusters |
|
||
| **Multi-Provider Support** | Let's Encrypt, ZeroSSL, Google Trust Services, Buypass, or custom ACME CA |
|
||
| **Staging Mode** | Test with LE staging environment (no rate limits) before production |
|
||
| **Account Management** | Register, view details, and deactivate ACME accounts |
|
||
| **Order Tracking** | View all orders with status, domains, and challenge details |
|
||
| **Auto-Renewal** | Configurable renewal window (default: 30 days before expiry) |
|
||
| **Renewal Schedule** | Dashboard showing all ACME certs with expiry dates and renewal status |
|
||
| **HTTP-01 Challenges** | Built-in challenge responder with automatic HAProxy ACL injection |
|
||
| **EAB Support** | External Account Binding for CAs that require it |
|
||
| **Certificate Protection** | ACME-managed certs are read-only in SSL Management (content cannot be manually edited) |
|
||
| **CA Chain Import** | One-click import of CA intermediate certificates |
|
||
| **Certificate Revocation** | Revoke compromised certificates through the ACME CA |
|
||
|
||
#### ACME Configuration
|
||
|
||
ACME settings are managed through **Settings > ACME / SSL Automation** tab:
|
||
|
||
| Setting | Description | Default |
|
||
|---------|-------------|---------|
|
||
| `acme.provider` | ACME CA provider selection | `letsencrypt` |
|
||
| `acme.directory_url` | ACME directory endpoint URL | LE production URL |
|
||
| `acme.staging_mode` | Use staging environment | `false` |
|
||
| `acme.auto_renew_enabled` | Enable automatic certificate renewal | `false` |
|
||
| `acme.renew_before_days` | Days before expiry to trigger renewal | `30` |
|
||
| `acme.eab_kid` | External Account Binding Key ID (if required) | - |
|
||
| `acme.eab_hmac_key` | External Account Binding HMAC Key (if required) | - |
|
||
|
||
#### Cluster ACME Setup
|
||
|
||
For HTTP-01 challenges to work, each cluster needs ACME challenge routing enabled:
|
||
|
||
1. Go to **Cluster Management** > Edit cluster
|
||
2. Enable **ACME Challenge Routing** toggle
|
||
3. Optionally set a custom ACME backend URL
|
||
4. Apply changes
|
||
|
||
When enabled, HAProxy configuration is automatically injected with:
|
||
- An ACL matching `/.well-known/acme-challenge/` requests
|
||
- A `use_backend` rule directing challenge traffic to the management backend
|
||
- A dedicated `_acme_challenge_backend` section
|
||
|
||
This injection only affects HTTP-mode frontends and is completely removed when ACME is disabled.
|
||
|
||
#### ACME Distributed Architecture
|
||
|
||
Understanding how HTTP-01 challenges work in a distributed HAProxy environment is critical for successful certificate issuance:
|
||
|
||
```
|
||
┌─────────────────┐ DNS A Record ┌──────────────────┐
|
||
│ Let's Encrypt │ ──────────────────► │ HAProxy Node(s) │
|
||
│ (CA Server) │ HTTP GET :80 │ (VIP / Public) │
|
||
└─────────────────┘ /.well-known/ └────────┬─────────┘
|
||
acme-challenge/ │
|
||
{token} │ ACL match →
|
||
│ use_backend
|
||
│ _acme_challenge_backend
|
||
▼
|
||
┌──────────────────┐
|
||
│ OpenManager │
|
||
│ (Backend Server) │
|
||
│ Serves token │
|
||
│ response from DB │
|
||
└──────────────────┘
|
||
```
|
||
|
||
**Key Points:**
|
||
|
||
1. **Let's Encrypt connects to your HAProxy**, not directly to OpenManager
|
||
2. **DNS must resolve** the requested domain to your HAProxy node's public IP (or VIP/NAT)
|
||
3. **Port 80 must be open** from the internet to HAProxy for HTTP-01 validation
|
||
4. **HAProxy routes** the `/.well-known/acme-challenge/` path to OpenManager via the injected backend
|
||
5. **OpenManager serves** the challenge token response stored in the database
|
||
6. The `MANAGEMENT_BASE_URL` or `acme.challenge_backend_url` setting tells HAProxy where to find OpenManager
|
||
|
||
**Network Scenarios:**
|
||
|
||
| Scenario | HAProxy | OpenManager | DNS Target |
|
||
|----------|---------|-------------|------------|
|
||
| Single server | localhost:80 | localhost:5000 | Server's public IP |
|
||
| Separate servers | 10.0.0.10:80 | 10.0.0.20:5000 | HAProxy's public IP |
|
||
| Behind NAT/VIP | VIP: 1.2.3.4:80 | Internal:5000 | VIP address |
|
||
| Multi-cluster | Multiple HAProxy nodes | Central OpenManager | Each domain → respective HAProxy |
|
||
|
||
#### DNS-01 Challenge — Internal/Isolated Clusters & Wildcards *(v1.8.0 — Issue #35)*
|
||
|
||
The default **HTTP-01** challenge validates over **port 80**, so the domain must resolve publicly to an HAProxy node with ACME Challenge Routing enabled. **DNS-01** validates via a **DNS TXT record** (`_acme-challenge.<domain>`) instead, so it needs **no inbound port 80 and no public ingress** to the HAProxy node. Use it for:
|
||
|
||
- **Internal / isolated clusters** (behind a VPN/firewall, no public port 80) where you still control the domain's DNS.
|
||
- **Wildcard certificates** (`*.example.com`) — which can *only* be issued via DNS-01.
|
||
|
||
DNS-01 is **opt-in** and fully backward compatible: it is disabled until an administrator enables it, and existing HTTP-01 certificates are completely unaffected.
|
||
|
||
- **Enable it**: Settings → ACME / SSL Automation → **DNS-01 Challenge (advanced)** → turn on *Enable DNS-01 Challenge* and Save. While off, DNS-01 options are hidden and no DNS-01 orders can be created.
|
||
- **Per-account provider**: in ACME Automation, create (or reconfigure) an ACME account with **Challenge Method = DNS-01** and a **DNS Provider**. Provider credentials are **verified before saving** and **encrypted at rest** (Fernet, mirroring the VRRP/MFA secret pattern); they are never returned by the API or written to logs.
|
||
- **Supported providers**: **Manual** (publish the TXT record yourself in any DNS — including fully internal DNS — then click *Verify*; works everywhere but cannot auto-renew unattended), **Cloudflare** (API token with `Zone:DNS:Edit` + `Zone:Read`; the TXT record is created and cleaned up automatically and renews unattended), and **GoDaddy** *(v1.10.0)* (a **Production** API Key + Secret pair from `developer.godaddy.com/keys` — the first key that dashboard issues is an OTE/test key and is rejected; the zone must be in the same GoDaddy account, which needs at least one registered domain before GoDaddy allows DNS API access at all. A **Personal Access Token** works too: paste it as the API Key and leave the Secret blank — that is the forward path as GoDaddy retires the `sso-key` scheme. TXT records are created and cleaned up automatically and renew unattended). The provider interface is pluggable — more providers can be added without changing the issuance flow.
|
||
- **Same pipeline**: after validation the certificate follows the normal PENDING → APPLIED flow (assign to clusters / Apply Management) and the agent serves it — identical to HTTP-01 from finalize onward, with **zero agent or rendered-config changes** for DNS-01.
|
||
- **Manual flow**: the order detail shows the exact `_acme-challenge.<domain>` record name + TXT value (copyable); publish it and click *I've added the records — Verify*. For Cloudflare and GoDaddy it is automatic.
|
||
- **Resilience**: a propagation-lag failure is recovered by a **bounded fresh-order retry chain** (1 original + 3 retries with increasing backoff, kept under Let's Encrypt's rate limits); any orphaned TXT record is cleaned up by a reconcile sweep. The order detail shows a DNS-01 event timeline (publish → validation → cleanup).
|
||
- **Wildcards**: `*.example.com` is validated at `_acme-challenge.example.com`; it does **not** cover the apex — add `example.com` as a separate name if you need both (the providers handle the two coexisting TXT values automatically).
|
||
- **Scope (this release)**: the Site Wizard remains HTTP-01-only; issue DNS-01 / wildcard certificates from **ACME Automation**.
|
||
|
||
#### ACME Quick Start Guide
|
||
|
||
Follow these steps to obtain your first Let's Encrypt certificate:
|
||
|
||
**Step 1: Configure ACME Settings**
|
||
- Navigate to **Settings > ACME / SSL Automation** tab
|
||
- Select your ACME provider (default: Let's Encrypt)
|
||
- For testing, enable **Staging Mode** to avoid rate limits
|
||
- Save changes
|
||
|
||
**Step 2: Register an ACME Account**
|
||
- On the **SSL Certificates > ACME Automation** page, find the ACME Account card
|
||
- Click **Register Account** and provide a valid email address
|
||
- Accept the Terms of Service
|
||
|
||
**Step 3: Enable ACME on Clusters**
|
||
- Go to **Cluster Management** > Edit your cluster
|
||
- Enable **ACME Challenge Routing** toggle
|
||
- Set the correct **ACME Backend URL** if OpenManager is on a different server
|
||
- Save the cluster configuration
|
||
|
||
**Step 4: Apply Configuration**
|
||
- Go to **Apply Management** and apply the pending HAProxy configuration changes
|
||
- This injects the ACME challenge routing rules into HAProxy
|
||
|
||
**Step 5: Verify Challenge Routing**
|
||
|
||
Test that the challenge path is reachable through HAProxy:
|
||
```bash
|
||
curl -v http://your-haproxy-ip/.well-known/acme-challenge/test
|
||
```
|
||
Expected: HTTP 404 from OpenManager (not HAProxy's default 503). This confirms routing works.
|
||
|
||
**Step 6: Ensure DNS Resolution**
|
||
- Your domain(s) must have DNS A/AAAA records pointing to your HAProxy node's public IP
|
||
- Verify: `dig +short yourdomain.com` should return the HAProxy IP
|
||
|
||
**Step 7: Request Certificate**
|
||
- Go to **SSL Certificates > ACME Automation** tab
|
||
- Click **New Certificate** and follow the wizard
|
||
- The UI will guide you through prerequisite checks before submission
|
||
|
||
#### Troubleshooting ACME
|
||
|
||
If certificate issuance fails, check the following:
|
||
|
||
| Issue | Diagnostic Step |
|
||
|-------|----------------|
|
||
| Challenge 404 | Check `ACME-CHALLENGE` log entries in OpenManager logs |
|
||
| DNS mismatch | Verify `dig +short yourdomain.com` returns HAProxy IP |
|
||
| Port 80 blocked | Test from external: `curl http://yourdomain.com/.well-known/acme-challenge/test` |
|
||
| Config not applied | Check **Apply Management** for pending changes |
|
||
| ACME not enabled | Verify cluster has **ACME Challenge Routing** enabled |
|
||
| Account issues | Check ACME account status in **ACME Automation** page |
|
||
|
||
**Diagnostic Logging:** OpenManager provides detailed ACME logging with prefixes:
|
||
- `ACME:` — Order creation, challenge responses, finalization
|
||
- `ACME-CHALLENGE:` — Incoming challenge token requests and responses
|
||
|
||
Review application logs to trace the complete ACME flow when troubleshooting issues.
|
||
|
||
### WAF Management - Web Application Firewall
|
||
- **Rate Limiting**: Request rate limiting by IP, URL, or custom patterns
|
||
- **IP Filtering**: Whitelist/blacklist IP addresses and CIDR ranges
|
||
- **Geographic Blocking**: Country-based access restrictions
|
||
- **DDoS Protection**: Automated DDoS detection and mitigation rules
|
||
- **Custom Rules**: User-defined security rules with regex patterns
|
||
- **Attack Monitoring**: Real-time attack detection and logging
|
||
- **Rule Templates**: Pre-configured security rule templates
|
||
- **Statistics Dashboard**: WAF activity analytics and blocked request metrics
|
||
|
||
### IP Inventory - Cross-Cluster IP Search & Discovery
|
||
|
||
The IP Inventory page provides a unified view of all IP addresses across every cluster, independent of the global cluster selector. It enables quick identification of which cluster, agent, or backend server a given IP belongs to.
|
||
|
||
**Key Features:**
|
||
- **Cross-Cluster Search**: Search any IP address across all clusters simultaneously
|
||
- **IP Type Identification**: Instantly determine if an IP is an agent server IP, a keepalived VIP, or a backend server address
|
||
- **Unified Results Card**: Search results show matched IP type (Server IP, VIP, Backend), agent/server name, cluster association, and status
|
||
- **Agent Tab**: Browse all agents with cluster name, hostname, agent IP, VIP (keepalive), VRRP state (MASTER/BACKUP), status, platform, and last seen time
|
||
- **Backend Servers Tab**: Browse all backend servers with cluster, backend name, server address, status, weight, and check status
|
||
- **Cluster Color Coding**: Each cluster is assigned a consistent color tag for quick visual identification
|
||
- **Auto-Refresh**: Data refreshes automatically every 30 seconds
|
||
- **Filters**: Filter by cluster and status within each tab
|
||
|
||
**Use Cases:**
|
||
- Identify which HAProxy cluster an IP belongs to during incident response
|
||
- Find which agent is running as MASTER or BACKUP in a keepalived pair
|
||
- Locate a backend server IP across multiple clusters
|
||
- Audit all IP addresses managed by the platform
|
||
|
||
### HA / VIP Management - Keepalived Virtual IP Failover
|
||
|
||
The HA / VIP page manages **highly-available virtual IPs** backed by Keepalived (VRRP) directly from the UI — no SSHing into nodes to install/configure Keepalived by hand. It builds on the agent pull-architecture: you define the VIP centrally, click Apply, and the agents converge.
|
||
|
||
**What it does:**
|
||
- **VIP dashboard**: lists each virtual IP with its pool, VRID, per-node interface, and a **live MASTER / BACKUP / FAULT** column per node (refreshed every 30s from the existing keepalive-state heartbeat pipeline).
|
||
- **Creation form**: enter the virtual IP (+ prefix), pick the pool, and select which of the pool's HAProxy nodes participate — each with a network interface (from the node's reported interfaces), a VRRP role (MASTER/BACKUP) and a priority. VRID auto-allocates per pool; an optional VRRP secret can be set.
|
||
- **Hands-off automation**: on **Apply**, each member's `keepalived.conf` is rendered and delivered; the agent installs Keepalived if missing, writes the config + a HAProxy health-check (`vrrp_script` + `track_script`), validates with `keepalived -t`, and starts the service. When HAProxy drops on the active node, the health-check lowers VRRP priority and the **VIP fails over automatically** to a backup.
|
||
|
||
**How it works (pull-based, isolated):**
|
||
- New tables `vip_instances` + `vip_members`; the agent polls `GET /api/agents/{name}/keepalived-config` (key-authenticated, agent-bound) and converges. It is fully **isolated** from the global HAProxy apply flow — VIP changes never regenerate `haproxy.cfg`.
|
||
- **Cluster-driven path**: the `keepalived.conf` location is a cluster setting (`keepalived_config_path`, default `/etc/keepalived/keepalived.conf`) delivered to the agent dynamically — just like the HAProxy paths — so non-standard installs are supported with zero per-node effort. The default is what the keepalived service loads on every distro; if you set a non-default path, ensure the keepalived unit is configured to load it.
|
||
|
||
**Apply / Reject staging:**
|
||
- VIP edits are staged as **PENDING** and only go live on **Apply**, which records an applied snapshot. **Reject** discards pending changes and **restores the previous applied state** (a never-applied VIP is discarded). Editing a live VIP never disrupts it until you re-Apply.
|
||
|
||
**Enterprise safety:**
|
||
- **Opt-in & backward compatible** — nodes/clusters without a VIP do nothing new.
|
||
- **Never clobbers a hand-managed Keepalived** — if an unmanaged `keepalived.conf` exists, the agent reports "externally managed" and leaves it untouched.
|
||
- **Multi-distro install** — Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine (apt/dnf/yum/zypper/apk).
|
||
- **Reliable detection across platforms** — MASTER/BACKUP is detected via systemd journal, syslog, and a portable interface-based check (exact-match, all VIPs).
|
||
- **Secure** — the VRRP secret is encrypted at rest and never returned by the API; the delivery endpoint requires the node's own API key.
|
||
- **Scope** — VRRP targets bare-metal / VMware / on-prem L2 networks; the UI clearly notes that AWS/Azure/GCP don't honor VRRP/gratuitous-ARP. macOS agents can't run Keepalived and are excluded (the UI warns if one is selected).
|
||
|
||
**RBAC:** gated by the `vip` permission group (`vip.read/create/update/delete/apply`); admins bypass.
|
||
|
||
### User Management - Access Control & Authentication
|
||
- **User Accounts**: Create, edit, and manage user accounts
|
||
- **Role-based Access**: Admin, user, and custom role definitions
|
||
- **Permission System**: Granular permissions for different system functions
|
||
- **User Activity**: Login history and user action audit logs
|
||
- **Password Management**: Secure password policies and reset functionality
|
||
- **Session Management**: Active session monitoring and force logout
|
||
- **API Keys**: User API key generation and management
|
||
- **Role Assignment**: Dynamic role assignment and permission updates
|
||
|
||
#### Multi-Factor Authentication (MFA) — v1.6.0 (Issue #18)
|
||
|
||
MFA is **optional per account** and **default OFF**. Existing users keep their
|
||
single-factor (username/password) login unless they choose to enable it. The
|
||
feature is fully additive: nothing changes for accounts that don't opt in.
|
||
|
||
**For end-users**
|
||
|
||
- Open **Users** → find your own row → click **Enable MFA**.
|
||
- A wizard opens with three steps:
|
||
1. **Set up** — scan the QR code with Google Authenticator / Authy /
|
||
1Password / Microsoft Authenticator, or paste the displayed secret manually.
|
||
2. **Verify** — enter the current 6-digit code from your app.
|
||
3. **Backup codes** — save the 10 single-use recovery codes (format
|
||
`XXXX-YYYY`). They are shown only once. Use the **Copy all** /
|
||
**Download .txt** buttons.
|
||
- After enrollment your sign-in becomes two-step: username/password →
|
||
6-digit TOTP (or a backup code).
|
||
- To turn MFA off again, open your row's **Disable MFA** action and enter a
|
||
current TOTP or backup code.
|
||
|
||
**For admins**
|
||
|
||
- On any user with MFA enabled, the **Reset MFA** action wipes the user's
|
||
TOTP secret, backup codes, and pending challenges. A required *reason* is
|
||
written to the audit log. After reset the user logs in with their password
|
||
and may re-enroll.
|
||
|
||
**Emergency: reset MFA for every user**
|
||
|
||
Use the CLI helper when an authenticator outage / mass key loss happens.
|
||
Double confirmation is required; the action is irreversible.
|
||
|
||
```bash
|
||
API_URL=https://hap.example.com ADMIN_TOKEN=eyJ... \
|
||
./scripts/admin-mfa-reset-all.sh
|
||
# Prompts ask for: 'yes' → 'RESET ALL MFA' → reason
|
||
# Audit log: action='mfa.disabled.admin_bulk_reset'
|
||
```
|
||
|
||
**Configuration**
|
||
|
||
- Backend env var `MFA_ENCRYPTION_KEY` — a 44-char Fernet key used to encrypt
|
||
TOTP secrets at rest. Generate with:
|
||
```bash
|
||
python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
|
||
```
|
||
On Kubernetes the key lives in the `backend-secret` Secret
|
||
(`k8s/manifests/03-secrets.yaml`). The shipped manifest uses the placeholder
|
||
`mfa_encryption_key_replace_me`; replace it via your CI/CD pipeline (e.g.
|
||
`sed` step) before `kubectl apply`.
|
||
- Optional `MFA_ACCOUNT_LABEL_DOMAIN` — overrides the per-user otpauth label
|
||
domain so QR codes show e.g. `alice@hap.example.com` instead of the request
|
||
hostname.
|
||
|
||
**API endpoints (all additive)**
|
||
|
||
| Method | Path | Notes |
|
||
|---|---|---|
|
||
| `POST` | `/api/auth/login` | Returns `mfa_required:true`+`mfa_token` for MFA users; legacy shape otherwise |
|
||
| `POST` | `/api/auth/login/mfa-verify` | Submits TOTP or backup code; returns JWT |
|
||
| `GET` | `/api/mfa/status` | Self status (enabled / method / backup codes remaining) |
|
||
| `POST` | `/api/mfa/enroll/start` | Begin TOTP enrollment |
|
||
| `POST` | `/api/mfa/enroll/confirm` | Confirm enrollment, returns 10 backup codes once |
|
||
| `POST` | `/api/mfa/disable` | Self-disable (TOTP or backup required) |
|
||
| `POST` | `/api/mfa/backup-codes/regenerate` | Issue 10 fresh backup codes (TOTP only) |
|
||
| `GET` | `/api/mfa/admin/status/{id}` | Admin: any user's MFA status |
|
||
| `POST` | `/api/mfa/admin-reset/{id}` | Admin: reset a single user's MFA |
|
||
| `POST` | `/api/mfa/admin-reset-all` | Admin: emergency reset for all users |
|
||
|
||
**Rate limits (per-user, ingress-aware, operationally tunable)**
|
||
|
||
MFA endpoints are rate-limited via slowapi+Redis with a **user-aware key
|
||
function** (`backend/middleware/mfa_rate_limit_key.py`):
|
||
|
||
1. If the request carries a valid Bearer JWT → bucket is `user:<id>`.
|
||
Each operator gets an isolated bucket; an org-wide MFA rollout is no
|
||
longer bottlenecked by the shared ingress IP.
|
||
2. Else, if the TCP peer is in `MFA_TRUSTED_PROXY_CIDRS` → bucket is the
|
||
first `X-Forwarded-For` hop (real client IP behind the ingress).
|
||
3. Else → bucket is the TCP peer (slowapi default).
|
||
|
||
Defaults live in `backend/middleware/mfa_rate_limits.py` and are sized for
|
||
**enterprise-scale** rollouts (thousands of operators). Each one is
|
||
overridable via env var; an invalid string logs a `WARNING` and falls back
|
||
to the default without crashing.
|
||
|
||
| Endpoint | Env var | Default | Bucket |
|
||
|---|---|---|---|
|
||
| `POST /api/mfa/enroll/start` | `MFA_RATE_LIMIT_ENROLL_START` | `10/minute` | per user |
|
||
| `POST /api/mfa/enroll/confirm` | `MFA_RATE_LIMIT_ENROLL_CONFIRM` | `10/minute` | per user |
|
||
| `POST /api/mfa/disable` | `MFA_RATE_LIMIT_DISABLE` | `10/minute` | per user |
|
||
| `POST /api/mfa/backup-codes/regenerate` | `MFA_RATE_LIMIT_REGENERATE_BACKUP_CODES` | `5/hour` | per user |
|
||
| `POST /api/mfa/admin-reset/{id}` | `MFA_RATE_LIMIT_ADMIN_RESET` | `60/hour` | per admin |
|
||
| `POST /api/mfa/admin-reset-all` | `MFA_RATE_LIMIT_ADMIN_RESET_ALL` | `1/day` | per admin |
|
||
|
||
Limit string format follows slowapi: `<count>/<second|minute|hour|day>`.
|
||
|
||
**Trusted-proxy configuration**
|
||
`MFA_TRUSTED_PROXY_CIDRS` — comma-separated CIDR list, e.g.
|
||
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`. Empty (default) disables XFF
|
||
parsing — XFF from any peer is then ignored, which is the safe choice when
|
||
the topology is unknown. Set this when your backend sits behind a known
|
||
ingress / load balancer so anonymous flows still get per-real-IP buckets.
|
||
|
||
The pre-existing `/api/auth/login` rate-limiting policy is unchanged. On
|
||
Kubernetes, see commented overrides in
|
||
`k8s/manifests/07-configmaps.yaml::backend-config`.
|
||
|
||
### Settings - System Configuration
|
||
- **Theme Settings**: Light/dark mode toggle and UI customization
|
||
- **ACME / SSL Automation**: Configure ACME provider, directory URL, staging mode, auto-renewal, EAB credentials, and test CA connectivity
|
||
- **Notification Settings**: Alert preferences and notification channels
|
||
- **System Preferences**: Default timeouts, refresh intervals, and limits
|
||
- **Backup Settings**: Automated backup scheduling and retention policies
|
||
- **Integration Settings**: External system integrations and webhooks
|
||
- **Language Settings**: Multi-language support configuration
|
||
- **Performance Tuning**: System performance and optimization settings
|
||
|
||
|
||
## Getting Started - First-Time Usage
|
||
|
||
### Quick Start for New Users
|
||
|
||
This guide walks you through using HAProxy OpenManager for the first time, including importing your existing HAProxy configuration.
|
||
|
||
#### Setup Flow Diagram
|
||
|
||
```mermaid
|
||
graph TD
|
||
Start([🚀 Start Setup]) --> Step1[1️⃣ Create Agent Pool<br/>HAProxy Agent Pool Management]
|
||
Step1 --> Step2[2️⃣ Create Cluster<br/>HAProxy Cluster Management]
|
||
Step2 --> Step3[3️⃣ Generate Agent Script<br/>Agent Management]
|
||
Step3 --> Step4[4️⃣ Run Script on HAProxy Server<br/>SSH to server and execute]
|
||
Step4 --> Step5[5️⃣ Copy Backup Config<br/>haproxy.cfg.initial-backup]
|
||
Step5 --> Step6[6️⃣ Import Configuration<br/>Bulk HAProxy Config Import]
|
||
Step6 --> Step7[7️⃣ Apply Changes<br/>Apply Management]
|
||
Step7 --> Complete([✅ Setup Complete!])
|
||
|
||
style Start fill:#2d5016,stroke:#4caf50,stroke-width:3px,color:#fff
|
||
style Step1 fill:#7b4d1e,stroke:#ff9800,stroke-width:2px,color:#fff
|
||
style Step2 fill:#7b4d1e,stroke:#ff9800,stroke-width:2px,color:#fff
|
||
style Step3 fill:#1a4971,stroke:#2196f3,stroke-width:2px,color:#fff
|
||
style Step4 fill:#4a1a4a,stroke:#9c27b0,stroke-width:2px,color:#fff
|
||
style Step5 fill:#6b1e3f,stroke:#e91e63,stroke-width:2px,color:#fff
|
||
style Step6 fill:#1a4d43,stroke:#009688,stroke-width:2px,color:#fff
|
||
style Step7 fill:#6b5d1a,stroke:#fbc02d,stroke-width:2px,color:#fff
|
||
style Complete fill:#2d5016,stroke:#4caf50,stroke-width:3px,color:#fff
|
||
```
|
||
|
||
#### Detailed Step-by-Step Instructions
|
||
|
||
##### Step 1: Create Agent Pool 📦
|
||
|
||
Navigate to **HAProxy Agent Pool Management** page.
|
||
|
||
1. Click **"Create Pool"** button
|
||
2. Fill in pool details:
|
||
- **Pool Name**: `Production Pool` (or your preferred name)
|
||
- **Description**: `Production environment HAProxy servers`
|
||
3. Click **Save**
|
||
|
||
**Purpose**: Pools logically group your HAProxy agents for better organization.
|
||
|
||
---
|
||
|
||
##### Step 2: Create Cluster 🎯
|
||
|
||
Navigate to **HAProxy Cluster Management** page.
|
||
|
||
1. Click **"Create Cluster"** button
|
||
2. Fill in cluster details:
|
||
- **Cluster Name**: `Production Cluster`
|
||
- **Description**: `Main production HAProxy cluster`
|
||
- **Select Pool**: Choose the pool created in Step 1 (`Production Pool`)
|
||
3. Click **Save**
|
||
|
||
**Purpose**: A cluster represents a logical grouping of HAProxy configurations that will be deployed to agents.
|
||
|
||
---
|
||
|
||
##### Step 3: Generate Agent Installation Script 🤖
|
||
|
||
Navigate to **Agent Management** page.
|
||
|
||
1. Click **"Create Agent"** or **"Generate Script"** button
|
||
2. Configure agent details:
|
||
- **Agent Name**: `haproxy-prod-01` (unique identifier)
|
||
- **Select Pool**: Choose the same pool from Step 1 (`Production Pool`)
|
||
- **Platform**: Select `Linux` or `macOS`
|
||
- **Architecture**: Select `x86_64` or `ARM64`
|
||
3. Click **"Generate Install Script"**
|
||
4. Click **"Download Script"** to save the installation script
|
||
|
||
**Purpose**: This generates a customized installation script that will connect your HAProxy server to the management backend.
|
||
|
||
---
|
||
|
||
##### Step 4: Install Agent on HAProxy Server 🖥️
|
||
|
||
SSH into your HAProxy server and run the installation script.
|
||
|
||
```bash
|
||
# Copy the script to your HAProxy server
|
||
scp install-agent-haproxy-prod-01.sh root@your-haproxy-server:/tmp/
|
||
|
||
# SSH to the HAProxy server
|
||
ssh root@your-haproxy-server
|
||
|
||
# Make the script executable
|
||
chmod +x /tmp/install-agent-haproxy-prod-01.sh
|
||
|
||
# Run the installation script
|
||
sudo /tmp/install-agent-haproxy-prod-01.sh
|
||
```
|
||
|
||
**What the script does:**
|
||
- Installs required dependencies (Python, socat)
|
||
- Creates `haproxy-agent` systemd service
|
||
- Starts the agent (polls backend every 30 seconds)
|
||
- **Creates initial backup**: `/etc/haproxy/haproxy.cfg.initial-backup`
|
||
- Registers the agent with the backend
|
||
|
||
**Verify installation:**
|
||
```bash
|
||
# Check agent service status
|
||
sudo systemctl status haproxy-agent
|
||
|
||
# Check agent logs
|
||
sudo tail -f /var/log/haproxy-agent/agent.log
|
||
```
|
||
|
||
Return to the UI and verify the agent appears in **Agent Management** with status "Connected ✅".
|
||
|
||
---
|
||
|
||
##### Step 5: Copy Existing HAProxy Configuration 📄
|
||
|
||
You have **three options** to get your existing HAProxy configuration:
|
||
|
||
**Option A: Using the Initial Backup (Recommended)**
|
||
|
||
On your HAProxy server, the agent created a backup of your original configuration.
|
||
|
||
```bash
|
||
# Display the backup configuration
|
||
cat /etc/haproxy/haproxy.cfg.initial-backup
|
||
```
|
||
|
||
**Copy the entire contents** of this file to your clipboard.
|
||
|
||
---
|
||
|
||
**Option B: View Configuration via UI (Easiest)**
|
||
|
||
Navigate to **Configuration Management** page in the UI.
|
||
|
||
1. Select your agent from the **Agent Selector** dropdown
|
||
2. Click **"View Configuration"** button
|
||
3. The system will fetch the live configuration from the agent
|
||
4. **Copy the entire configuration** displayed in the viewer
|
||
5. You now have the configuration ready for import!
|
||
|
||
**Benefits**: No need to SSH to the server, get configuration directly from the UI.
|
||
|
||
---
|
||
|
||
**Option C: Current Configuration File**
|
||
|
||
If the backup doesn't exist, SSH to your server and copy the current configuration:
|
||
|
||
```bash
|
||
# Display current HAProxy configuration
|
||
cat /etc/haproxy/haproxy.cfg
|
||
```
|
||
|
||
**Copy the entire contents** to your clipboard.
|
||
|
||
---
|
||
|
||
##### Step 6: Import Configuration via Bulk Import 📥
|
||
|
||
Navigate to **Bulk HAProxy Config Import** page.
|
||
|
||
1. Select your cluster from the **Cluster Selector** dropdown (top of page)
|
||
2. **Paste** the configuration copied from Step 5 into the text area
|
||
3. Click **"Parse Configuration"** button
|
||
4. Review the parsed entities:
|
||
- Frontends detected
|
||
- Backends detected
|
||
- Servers detected
|
||
- SSL certificates (if any)
|
||
5. Click **"Import Configuration"**
|
||
|
||
**What happens:**
|
||
- The system parses your HAProxy config file
|
||
- Extracts frontends, backends, servers, and SSL bindings
|
||
- Creates database entities for each component
|
||
- Prepares configuration for deployment
|
||
|
||
**Result**: Your existing HAProxy configuration is now managed by HAProxy OpenManager! 🎉
|
||
|
||
---
|
||
|
||
##### Step 7: Apply Changes to Agent 🚀
|
||
|
||
Navigate to **Apply Management** page.
|
||
|
||
1. You'll see pending changes listed:
|
||
- Frontends to be created
|
||
- Backends to be created
|
||
- Servers to be added
|
||
2. Review the changes
|
||
3. Click **"Apply Changes"** button
|
||
4. Monitor deployment status:
|
||
```
|
||
Agent: haproxy-prod-01
|
||
Status: ✅ Configuration applied successfully
|
||
Version: v1.0.1
|
||
```
|
||
|
||
**What happens:**
|
||
- Backend marks configuration tasks as ready for deployment
|
||
- Agent polls and detects pending tasks (within 30 seconds)
|
||
- Agent downloads the new configuration
|
||
- Agent validates HAProxy configuration syntax
|
||
- Agent reloads HAProxy service (zero downtime)
|
||
- Agent reports success back to backend
|
||
|
||
**Verify deployment:**
|
||
- Check that agent status shows "Applied Successfully"
|
||
- Check HAProxy is running: `systemctl status haproxy`
|
||
- Check HAProxy stats: `http://your-server:8404/stats`
|
||
|
||
---
|
||
|
||
#### ✅ Setup Complete!
|
||
|
||
Your HAProxy server is now fully managed by HAProxy OpenManager. You can now:
|
||
|
||
- ✨ **Manage Frontends**: Add, edit, delete frontends via UI
|
||
- 🖥️ **Manage Backends**: Configure backend servers and health checks
|
||
- 🔒 **Upload SSL Certificates**: Centralized SSL management
|
||
- 🛡️ **Configure WAF Rules**: Add security rules
|
||
- 📊 **Monitor Performance**: View real-time statistics on dashboard
|
||
- 🔄 **Version Control**: Track all configuration changes with rollback capability
|
||
|
||
**Next Steps:**
|
||
- Add more agents to the same pool for multi-node clusters
|
||
- Configure SSL certificates in **SSL Management**
|
||
- Set up ACME Auto SSL in **SSL Management > ACME Automation** for automated Let's Encrypt certificates
|
||
- Add WAF rules in **WAF Management**
|
||
- Monitor your cluster in **Dashboard**
|
||
|
||
---
|
||
|
||
|
||
## Installation
|
||
|
||
Choose your preferred installation method:
|
||
|
||
| Method | Best For | Guide |
|
||
|--------|----------|-------|
|
||
| **Docker** | Quick setup, single-server, Docker Compose | [Docker Installation](#docker-installation) |
|
||
| **Kubernetes/OpenShift** | Production, HA, enterprise | [Kubernetes Installation](#kubernetesopenshift-installation) |
|
||
| **Standalone** | Bare-metal, VMs, LXC, no containers | [Standalone Guide](docs/STANDALONE_INSTALL.md) |
|
||
|
||
---
|
||
|
||
### Docker Installation
|
||
|
||
#### Prerequisites
|
||
|
||
- **Docker & Docker Compose** (v20.10+)
|
||
- **Git**
|
||
- **2GB+ RAM** for all services
|
||
|
||
#### Installation Steps
|
||
|
||
1. **Clone the Repository**
|
||
```bash
|
||
git clone https://github.com/taylanbakircioglu/haproxy-openmanager.git
|
||
cd haproxy-openmanager
|
||
```
|
||
|
||
2. **Start All Services**
|
||
```bash
|
||
docker-compose up -d
|
||
```
|
||
|
||
3. **Verify Installation**
|
||
```bash
|
||
# Check all services are running
|
||
docker-compose ps
|
||
|
||
# Check API health
|
||
curl http://localhost:8080/api/clusters
|
||
```
|
||
|
||
4. **Access the Application**
|
||
- **Web Interface**: http://localhost:8080
|
||
- **API Documentation**: http://localhost:8080/docs
|
||
- **HAProxy Stats**: http://localhost:8404/stats
|
||
|
||
### Default Credentials
|
||
|
||
- **Admin User**: `admin` / `admin123`
|
||
- **Regular User**: `user` / `user123`
|
||
|
||
> ⚠️ **Security Note**: Change default passwords immediately in production environments.
|
||
|
||
#### Services Overview
|
||
|
||
The application consists of 8 Docker containers:
|
||
|
||
```bash
|
||
# Core Application Services
|
||
haproxy-openmanager-frontend # React.js web interface
|
||
haproxy-openmanager-backend # FastAPI backend
|
||
haproxy-openmanager-db # PostgreSQL database
|
||
haproxy-openmanager-redis # Redis cache
|
||
haproxy-openmanager-nginx # Nginx reverse proxy
|
||
|
||
# HAProxy Test Instances
|
||
haproxy-instance # Local test instance
|
||
haproxy-remote1 # Remote test instance 1
|
||
haproxy-remote2 # Remote test instance 2
|
||
```
|
||
|
||
### Management Commands
|
||
|
||
```bash
|
||
# Start all services
|
||
docker-compose up -d
|
||
|
||
# View logs
|
||
docker-compose logs -f [service_name]
|
||
|
||
# Stop all services
|
||
docker-compose down
|
||
|
||
# Rebuild and restart
|
||
docker-compose down && docker-compose up --build -d
|
||
|
||
# Database reset (⚠️ Destroys all data)
|
||
docker-compose down
|
||
docker volume rm haproxy-openmanager_postgres_data
|
||
docker-compose up -d
|
||
```
|
||
|
||
### Volume Management
|
||
|
||
```bash
|
||
# List volumes
|
||
docker volume ls | grep haproxy
|
||
|
||
# Backup database
|
||
docker exec haproxy-openmanager-db pg_dump -U haproxy_user haproxy_openmanager > backup.sql
|
||
|
||
# Restore database
|
||
docker exec -i haproxy-openmanager-db psql -U haproxy_user haproxy_openmanager < backup.sql
|
||
```
|
||
|
||
### Kubernetes/OpenShift Installation
|
||
|
||
#### Kubernetes Deployment
|
||
|
||
For Kubernetes environments, use the provided manifests:
|
||
|
||
```bash
|
||
# Deploy to Kubernetes
|
||
kubectl apply -f k8s/manifests/
|
||
|
||
# Check deployment status
|
||
kubectl get pods -n haproxy-openmanager
|
||
|
||
# Access via port-forward
|
||
kubectl port-forward svc/nginx-service 8080:80 -n haproxy-openmanager
|
||
```
|
||
|
||
See [k8s/manifests/README.md](k8s/manifests/README.md) for detailed Kubernetes setup instructions.
|
||
|
||
### Local Development
|
||
|
||
#### Local Development Setup
|
||
|
||
1. **Backend Development**
|
||
```bash
|
||
cd backend
|
||
python -m venv venv
|
||
source venv/bin/activate # Linux/Mac
|
||
# or
|
||
venv\Scripts\activate # Windows
|
||
|
||
pip install -r requirements.txt
|
||
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
2. **Frontend Development**
|
||
```bash
|
||
cd frontend
|
||
npm install
|
||
npm start
|
||
```
|
||
|
||
3. **Database Setup**
|
||
```bash
|
||
# Start only database and redis
|
||
docker-compose up -d postgres redis
|
||
|
||
# Run migrations
|
||
python backend/migration.py
|
||
```
|
||
|
||
|
||
## Configuration
|
||
|
||
### Environment Variables
|
||
|
||
All configuration is managed through environment variables for maximum flexibility across different deployment environments.
|
||
|
||
**📄 Configuration Template**: `.env.template`
|
||
|
||
```bash
|
||
# 1. Copy the template
|
||
cp .env.template .env
|
||
|
||
# 2. Edit (replace placeholders with your actual values)
|
||
nano .env
|
||
|
||
# 3. Start
|
||
docker-compose up -d
|
||
```
|
||
|
||
**Important**:
|
||
- `.env.template` is only a template (committed to git)
|
||
- `.env` is the actual configuration (NOT committed - in .gitignore)
|
||
- When making changes, edit `.env`, not `.env.template`!
|
||
|
||
#### Backend Configuration
|
||
```bash
|
||
# Database
|
||
DATABASE_URL="postgresql://haproxy_user:haproxy_pass@postgres:5432/haproxy_openmanager"
|
||
REDIS_URL="redis://redis:6379"
|
||
|
||
# Security
|
||
SECRET_KEY="your-secret-key-change-this-in-production"
|
||
DEBUG="False" # Set to False in production
|
||
|
||
# Public URL Configuration (IMPORTANT for agent connectivity)
|
||
# This URL is embedded in agent installation scripts
|
||
PUBLIC_URL="https://your-haproxy-openmanager.example.com"
|
||
MANAGEMENT_BASE_URL="${PUBLIC_URL}" # Optional, defaults to PUBLIC_URL
|
||
|
||
# Examples:
|
||
# Development: PUBLIC_URL="http://localhost:8000"
|
||
# Production: PUBLIC_URL="https://haproxy-openmanager.example.com"
|
||
# Production: PUBLIC_URL="https://haproxy-openmanager.yourdomain.com"
|
||
|
||
# Logging
|
||
LOG_LEVEL="INFO" # DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||
|
||
# Agent Settings
|
||
AGENT_HEARTBEAT_TIMEOUT_SECONDS=15
|
||
AGENT_CONFIG_SYNC_INTERVAL_SECONDS=30
|
||
```
|
||
|
||
#### Frontend Configuration
|
||
|
||
The frontend is a Create-React-App single-page app served as a static bundle
|
||
(`serve -s build`). It uses **same-origin** (`window.location.host`) for all
|
||
`/api/*` calls — no env vars are needed in production. Routing is handled
|
||
entirely by the nginx reverse proxy in front of the frontend pod (Kubernetes
|
||
ingress + `nginx-config` ConfigMap, or `nginx/nginx.conf` for Docker Compose).
|
||
|
||
> ⚠️ **Do NOT set `REACT_APP_API_URL` in your CI/CD pipeline.** CRA inlines
|
||
> `REACT_APP_*` values into the bundle at **build time**, so any value baked
|
||
> in there overrides the runtime same-origin detection and breaks every
|
||
> deployment whose URL does not match the inlined string. Leave the variable
|
||
> unset; the bundle will resolve to whatever host the user is browsing.
|
||
|
||
```bash
|
||
# Optional, only when you intentionally need a cross-origin API
|
||
# (then CORS_ORIGINS on the backend must include the SPA's origin):
|
||
# REACT_APP_API_URL="https://api.example.com"
|
||
|
||
# Build settings
|
||
NODE_ENV="production"
|
||
GENERATE_SOURCEMAP="false"
|
||
```
|
||
|
||
#### Configuration Priority
|
||
|
||
1. **Development**: Use `.env` file in project root
|
||
2. **Docker Compose**: Environment variables in `docker-compose.yml`
|
||
3. **Kubernetes/OpenShift**: ConfigMaps (`k8s/manifests/07-configmaps.yaml`)
|
||
|
||
#### Why Parametric URLs?
|
||
|
||
All URLs are configurable via environment variables for enterprise deployment flexibility:
|
||
|
||
- ✅ **No hardcoded URLs** in source code
|
||
- ✅ **Deploy anywhere**: Development, staging, production
|
||
- ✅ **Multi-tenant support**: Each instance can have its own URL
|
||
- ✅ **Easy migration**: Change deployment location without code changes
|
||
|
||
### HAProxy Cluster Configuration
|
||
|
||
Add new HAProxy clusters through the web interface or directly via API:
|
||
|
||
```json
|
||
{
|
||
"name": "Production Cluster",
|
||
"description": "Production HAProxy cluster",
|
||
"host": "10.0.1.100",
|
||
"port": 8404,
|
||
"connection_type": "ssh",
|
||
"ssh_username": "haproxy",
|
||
"installation_type": "existing",
|
||
"deployment_type": "cluster",
|
||
"cluster_nodes": ["10.0.1.101", "10.0.1.102"],
|
||
"keepalive_ip": "10.0.1.10"
|
||
}
|
||
```
|
||
|
||
### Performance Tuning *(v1.8.6)*
|
||
|
||
The backend API defaults to a **single uvicorn worker process**, which uses one CPU core. Agents poll the API every 30 seconds (heartbeat, config, pending-requests, upgrade checks), so larger fleets add a constant baseline load. Two ways to scale:
|
||
|
||
- **Docker Compose — worker processes**: set `UVICORN_WORKERS` in your `.env` (default `1`). On a multi-core host, matching the core count (e.g. `UVICORN_WORKERS=2` on a 2-core machine) lets the API use all cores:
|
||
```bash
|
||
echo "UVICORN_WORKERS=2" >> .env && docker-compose up -d backend
|
||
```
|
||
- **Kubernetes/OpenShift — replicas**: the shipped manifests already include an HPA for the backend (2→10 replicas, `k8s/manifests/13-hpa.yaml`); raise `minReplicas`/`maxReplicas` as needed.
|
||
|
||
Both are safe: all background tasks (ACME completion, renewals, agent monitoring) are multi-replica safe by design (atomic claims via `FOR UPDATE SKIP LOCKED`, PostgreSQL advisory locks).
|
||
|
||
**Diagnosing slow requests**: every API response carries an `X-Response-Time` header, and the backend logs `Slow request detected` (WARNING) for any request taking longer than 1 second — check those log lines to pinpoint slow endpoints before tuning anything else.
|
||
|
||
## API Reference
|
||
|
||
### Authentication
|
||
```bash
|
||
# Login and get JWT token
|
||
POST /api/auth/login
|
||
Content-Type: application/json
|
||
{
|
||
"username": "admin",
|
||
"password": "admin123"
|
||
}
|
||
|
||
Response:
|
||
{
|
||
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||
"token_type": "bearer"
|
||
}
|
||
|
||
# Get current user
|
||
GET /api/auth/me
|
||
Authorization: Bearer <access_token>
|
||
```
|
||
|
||
### Pool Management
|
||
```bash
|
||
# List all pools
|
||
GET /api/pools
|
||
Authorization: Bearer <token>
|
||
|
||
# Create new pool
|
||
POST /api/pools
|
||
{
|
||
"name": "Production Pool",
|
||
"description": "Production environment pool"
|
||
}
|
||
|
||
# Get pool details
|
||
GET /api/pools/{pool_id}
|
||
```
|
||
|
||
### Agent Management
|
||
```bash
|
||
# List all agents
|
||
GET /api/agents
|
||
Authorization: Bearer <token>
|
||
|
||
# Create agent and get installation script
|
||
POST /api/agents
|
||
{
|
||
"name": "HAProxy-Prod-01",
|
||
"pool_id": 1,
|
||
"platform": "linux", # linux or macos
|
||
"architecture": "x86_64" # x86_64 or arm64
|
||
}
|
||
|
||
# Agent polls for tasks (called by agent service)
|
||
GET /api/agents/{agent_id}/tasks
|
||
Authorization: Bearer <agent_token>
|
||
|
||
# Agent reports task completion
|
||
POST /api/agents/tasks/{task_id}/complete
|
||
{
|
||
"status": "success",
|
||
"message": "Configuration applied successfully",
|
||
"logs": "HAProxy reloaded"
|
||
}
|
||
|
||
# Get agent status
|
||
GET /api/agents/{agent_id}/status
|
||
```
|
||
|
||
### Cluster Management
|
||
```bash
|
||
# List all clusters
|
||
GET /api/clusters
|
||
Authorization: Bearer <token>
|
||
|
||
# Create new cluster
|
||
POST /api/clusters
|
||
{
|
||
"name": "Production Cluster",
|
||
"description": "Production HAProxy cluster",
|
||
"pool_id": 1
|
||
}
|
||
|
||
# Get cluster details
|
||
GET /api/clusters/{cluster_id}
|
||
```
|
||
|
||
### Statistics & Monitoring
|
||
```bash
|
||
# Get dashboard overview
|
||
GET /api/dashboard/overview
|
||
|
||
# Get HAProxy statistics
|
||
GET /api/haproxy/stats?cluster_id=1
|
||
|
||
# Get backend server status
|
||
GET /api/backends?cluster_id=1
|
||
|
||
# Get frontend configurations
|
||
GET /api/frontends?cluster_id=1
|
||
```
|
||
|
||
### SSL CSR API (v1.9.0)
|
||
```bash
|
||
# Create a CSR (generates the private key server-side; response contains the
|
||
# CSR PEM — the private key is never returned by any endpoint)
|
||
POST /api/ssl/csrs
|
||
Authorization: Bearer <token>
|
||
{
|
||
"name": "www-example-com",
|
||
"common_name": "www.example.com",
|
||
"sans": ["api.example.com"],
|
||
"key_algorithm": "rsa-2048", # rsa-2048 | rsa-4096 | ecdsa-p256 | ecdsa-p384
|
||
"organization": "Example Corp",
|
||
"country": "TR"
|
||
}
|
||
|
||
# List CSRs (metadata only, no PEM)
|
||
GET /api/ssl/csrs
|
||
|
||
# CSR detail (includes the CSR PEM)
|
||
GET /api/ssl/csrs/{csr_id}
|
||
|
||
# Import the CA-signed certificate for a pending CSR
|
||
POST /api/ssl/csrs/{csr_id}/import
|
||
{
|
||
"certificate_content": "-----BEGIN CERTIFICATE-----...",
|
||
"chain_content": "-----BEGIN CERTIFICATE-----...", # optional
|
||
"usage_type": "frontend", # frontend | server
|
||
"is_global": false,
|
||
"cluster_ids": [1, 2]
|
||
}
|
||
|
||
# Delete a CSR (pending: permanently destroys the private key;
|
||
# completed: removes history only — the imported certificate is unaffected)
|
||
DELETE /api/ssl/csrs/{csr_id}
|
||
```
|
||
|
||
### ACME / Let's Encrypt API
|
||
```bash
|
||
# List ACME accounts
|
||
GET /api/letsencrypt/accounts
|
||
Authorization: Bearer <token>
|
||
|
||
# Register ACME account
|
||
POST /api/letsencrypt/accounts
|
||
{
|
||
"email": "admin@example.com",
|
||
"directory_url": "https://acme-v02.api.letsencrypt.org/directory",
|
||
"tos_agreed": true
|
||
}
|
||
|
||
# Deactivate ACME account
|
||
DELETE /api/letsencrypt/accounts/{account_id}
|
||
|
||
# Request certificate
|
||
POST /api/letsencrypt/certificates
|
||
{
|
||
"domains": ["example.com", "www.example.com"],
|
||
"account_id": 1,
|
||
"cluster_ids": [1, 2]
|
||
}
|
||
|
||
# List ACME orders
|
||
GET /api/letsencrypt/orders
|
||
|
||
# Get order details
|
||
GET /api/letsencrypt/orders/{order_id}
|
||
|
||
# Retry/resume order
|
||
POST /api/letsencrypt/orders/{order_id}/retry
|
||
|
||
# Cancel order
|
||
DELETE /api/letsencrypt/orders/{order_id}
|
||
|
||
# Renew certificate (create new order)
|
||
POST /api/letsencrypt/orders/{order_id}/renew
|
||
|
||
# Revoke certificate
|
||
POST /api/letsencrypt/certificates/{cert_id}/revoke
|
||
|
||
# Import CA chain certificates
|
||
POST /api/letsencrypt/import-ca-chain
|
||
|
||
# Get renewal schedule
|
||
GET /api/letsencrypt/renewal-schedule
|
||
|
||
# Check ACME prerequisites (setup status)
|
||
GET /api/letsencrypt/prerequisites
|
||
|
||
# ACME Settings
|
||
GET /api/settings/acme
|
||
PUT /api/settings/acme
|
||
GET /api/settings/acme/test-connection?directory_url=https://...
|
||
```
|
||
|
||
### Configuration Management
|
||
```bash
|
||
# Get HAProxy configuration
|
||
GET /api/haproxy/config?cluster_id=1
|
||
|
||
# Update configuration
|
||
PUT /api/haproxy/config?cluster_id=1
|
||
Content-Type: text/plain
|
||
|
||
# Validate configuration
|
||
POST /api/haproxy/config/validate
|
||
```
|
||
|
||
## Development
|
||
|
||
### Project Structure
|
||
|
||
```
|
||
haproxy-openmanager/
|
||
├── backend/ # FastAPI backend application
|
||
│ ├── main.py # Main application entry point
|
||
│ ├── config.py # Application configuration
|
||
│ ├── auth_middleware.py # Authentication middleware
|
||
│ ├── agent_notifications.py # Agent notification system
|
||
│ ├── database/ # Database layer
|
||
│ │ ├── connection.py # Database connection management
|
||
│ │ └── migrations.py # Database migration scripts
|
||
│ ├── models/ # SQLAlchemy ORM models
|
||
│ │ ├── user.py # User model
|
||
│ │ ├── cluster.py # Cluster model
|
||
│ │ ├── agent.py # Agent model
|
||
│ │ ├── frontend.py # Frontend configuration model
|
||
│ │ ├── backend.py # Backend configuration model
|
||
│ │ ├── ssl.py # SSL certificate model
|
||
│ │ └── waf.py # WAF rules model
|
||
│ ├── routers/ # API route handlers
|
||
│ │ ├── auth.py # Authentication endpoints
|
||
│ │ ├── user.py # User management
|
||
│ │ ├── cluster.py # Cluster management
|
||
│ │ ├── agent.py # Agent management
|
||
│ │ ├── frontend.py # Frontend configuration
|
||
│ │ ├── backend.py # Backend configuration
|
||
│ │ ├── ssl.py # SSL certificate management
|
||
│ │ ├── letsencrypt.py # ACME/Let's Encrypt endpoints
|
||
│ │ ├── settings.py # System settings (incl. ACME config)
|
||
│ │ ├── waf.py # WAF rules management
|
||
│ │ ├── dashboard.py # Dashboard statistics
|
||
│ │ ├── configuration.py # Configuration viewer
|
||
│ │ └── security.py # Security & token management
|
||
│ ├── services/ # Business logic services
|
||
│ │ ├── haproxy_config.py # HAProxy config generation
|
||
│ │ ├── acme_service.py # ACME protocol client (RFC 8555)
|
||
│ │ └── dashboard_stats_service.py # Stats aggregation
|
||
│ ├── middleware/ # Custom middleware
|
||
│ │ ├── activity_logger.py # Activity logging
|
||
│ │ ├── error_handler.py # Error handling
|
||
│ │ └── rate_limiter.py # API rate limiting
|
||
│ ├── utils/ # Utility functions
|
||
│ │ ├── auth.py # Authentication utilities
|
||
│ │ ├── haproxy_config_parser.py # Config parser
|
||
│ │ ├── haproxy_stats_parser.py # Stats parser
|
||
│ │ ├── haproxy_validator.py # Config validator
|
||
│ │ ├── ssl_parser.py # SSL certificate parser
|
||
│ │ ├── activity_log.py # Activity log utilities
|
||
│ │ ├── logging_config.py # Logging configuration
|
||
│ │ └── agent_scripts/ # Agent installation scripts
|
||
│ │ ├── linux_install.sh # Linux agent installer
|
||
│ │ └── macos_install.sh # macOS agent installer
|
||
│ ├── tests/ # Backend tests
|
||
│ │ ├── conftest.py # Pytest configuration
|
||
│ │ └── test_*.py # Test files
|
||
│ ├── requirements.txt # Python dependencies
|
||
│ └── Dockerfile # Backend Docker image
|
||
│
|
||
├── frontend/ # React.js frontend application
|
||
│ ├── src/
|
||
│ │ ├── components/ # React components
|
||
│ │ │ ├── Login.js # Login page
|
||
│ │ │ ├── DashboardV2.js # Main dashboard
|
||
│ │ │ ├── AgentManagement.js # Agent management
|
||
│ │ │ ├── ClusterManagement.js # Cluster management
|
||
│ │ │ ├── PoolManagement.js # Pool management
|
||
│ │ │ ├── FrontendManagement.js # Frontend config
|
||
│ │ │ ├── BackendServers.js # Backend config
|
||
│ │ │ ├── SSLManagement.js # SSL certificates
|
||
│ │ │ ├── ACMEAutomation.js # ACME Auto SSL UI
|
||
│ │ │ ├── WAFManagement.js # WAF rules
|
||
│ │ │ ├── ApplyManagement.js # Apply changes
|
||
│ │ │ ├── Configuration.js # Config viewer
|
||
│ │ │ ├── BulkConfigImport.js # Bulk import
|
||
│ │ │ ├── IPInventory.js # IP inventory & search
|
||
│ │ │ ├── VersionHistory.js # Version history
|
||
│ │ │ ├── UserManagement.js # User management
|
||
│ │ │ ├── Security.js # Security & tokens
|
||
│ │ │ ├── Settings.js # Settings page
|
||
│ │ │ ├── EntitySyncStatus.js # Sync status
|
||
│ │ │ └── dashboard/ # Dashboard components
|
||
│ │ │ └── *.js # Dashboard tabs & charts
|
||
│ │ ├── contexts/ # React contexts
|
||
│ │ │ ├── AuthContext.js # Authentication context
|
||
│ │ │ ├── ClusterContext.js # Cluster selection context
|
||
│ │ │ ├── ThemeContext.js # Theme (dark/light mode)
|
||
│ │ │ └── ProgressContext.js # Global progress indicator
|
||
│ │ ├── utils/ # Utility functions
|
||
│ │ │ ├── api.js # API client
|
||
│ │ │ ├── agentSync.js # Agent sync utilities
|
||
│ │ │ ├── colors.js # Color utilities
|
||
│ │ │ └── dashboardCache.js # Dashboard caching
|
||
│ │ ├── App.js # Main app component
|
||
│ │ ├── App.css # Global styles
|
||
│ │ └── index.js # React entry point
|
||
│ ├── public/
|
||
│ │ └── index.html # HTML template
|
||
│ ├── package.json # Node dependencies
|
||
│ └── Dockerfile # Frontend Docker image
|
||
│
|
||
├── docs/ # Documentation
|
||
│ └── screenshots/ # UI screenshots
|
||
│ ├── dashboard-overview.png
|
||
│ ├── frontend-management.png
|
||
│ ├── backend-configuration.png
|
||
│ ├── bulk-import.png
|
||
│ └── *.png # Other screenshots
|
||
│
|
||
├── k8s/ # Kubernetes/OpenShift manifests
|
||
│ └── manifests/
|
||
│ ├── 00-namespace.yaml # Namespace
|
||
│ ├── 01-service-accounts.yaml # Service accounts
|
||
│ ├── 02-rbac.yaml # RBAC roles
|
||
│ ├── 03-secrets.yaml # Secrets
|
||
│ ├── 04-storage.yaml # Persistent volumes
|
||
│ ├── 05-postgres.yaml # PostgreSQL deployment
|
||
│ ├── 06-redis.yaml # Redis deployment
|
||
│ ├── 07-configmaps.yaml # ConfigMaps
|
||
│ ├── 08-backend.yaml # Backend deployment
|
||
│ ├── 09-frontend.yaml # Frontend deployment
|
||
│ ├── 10-nginx.yaml # Nginx deployment
|
||
│ ├── 11-routes.yaml # OpenShift routes
|
||
│ ├── 12-ingress.yaml # Kubernetes ingress
|
||
│ ├── 13-hpa.yaml # Horizontal Pod Autoscaler
|
||
│ ├── deploy.sh # Deployment script
|
||
│ └── cleanup.sh # Cleanup script
|
||
│
|
||
├── nginx/ # Nginx reverse proxy
|
||
│ └── nginx.conf # Nginx configuration
|
||
│
|
||
├── haproxy/ # Sample HAProxy configs
|
||
│ ├── haproxy.cfg # Sample configuration
|
||
│ ├── haproxy-simple.cfg # Simple example
|
||
│ ├── haproxy-remote1.cfg # Remote example 1
|
||
│ └── haproxy-remote2.cfg # Remote example 2
|
||
│
|
||
├── scripts/ # Utility scripts
|
||
│ ├── check-agent-logs.sh # Check agent logs
|
||
│ ├── check-agent-stats.sh # Check agent stats
|
||
│ ├── check-haproxy-stats-socket.sh # Check stats socket
|
||
│ ├── cleanup-cluster-entities.sh # Cleanup entities
|
||
│ ├── cleanup-soft-deleted.sh # Cleanup soft deletes
|
||
│ ├── update-agent-version.sh # Update agent version
|
||
│ └── test-stats-parser.py # Test stats parser
|
||
│
|
||
├── utils/ # Uninstall utilities
|
||
│ ├── uninstall-agent-linux.sh # Linux uninstaller
|
||
│ └── uninstall-agent-macos.sh # macOS uninstaller
|
||
│
|
||
├── docker-compose.yml # Docker Compose configuration
|
||
├── docker-compose.localtest.yml # Local development/testing overrides
|
||
├── docker-compose.test.yml # Test environment
|
||
├── backend/version.json # Application version metadata (single source of truth)
|
||
├── build-images.sh # Build Docker images
|
||
├── pytest.ini # Pytest configuration
|
||
├── README.md # This file
|
||
├── CONFIG.md # Configuration documentation
|
||
├── TESTING.md # Testing documentation
|
||
├── UPGRADE_GUIDE.md # Upgrade guide
|
||
└── LICENSE # MIT License
|
||
```
|
||
|
||
### Adding New Features
|
||
|
||
1. **Backend API Endpoints**: Add to `backend/main.py`
|
||
2. **Frontend Components**: Add to `frontend/src/components/`
|
||
3. **Database Changes**: Update `backend/init.sql` and create migration
|
||
4. **HAProxy Features**: Extend `backend/haproxy_client.py`
|
||
|
||
|
||
## Troubleshooting
|
||
|
||
### Common Issues
|
||
|
||
#### 1. Connection Refused to HAProxy
|
||
```bash
|
||
# Check if HAProxy is running
|
||
curl http://localhost:8404/stats
|
||
|
||
# Check HAProxy configuration
|
||
docker exec haproxy-instance haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
|
||
|
||
# View HAProxy logs
|
||
docker logs haproxy-instance
|
||
```
|
||
|
||
#### 2. Database Connection Issues
|
||
```bash
|
||
# Check database status
|
||
docker exec haproxy-openmanager-db pg_isready -U haproxy_user
|
||
|
||
# Check database logs
|
||
docker logs haproxy-openmanager-db
|
||
|
||
# Reset database
|
||
docker-compose down
|
||
docker volume rm haproxy-openmanager_postgres_data
|
||
docker-compose up -d
|
||
```
|
||
|
||
#### 3. SSH Connection Issues
|
||
```bash
|
||
# Test SSH connectivity
|
||
ssh -i ~/.ssh/id_rsa user@remote-server
|
||
|
||
# Check SSH key permissions
|
||
chmod 600 ~/.ssh/id_rsa
|
||
chmod 644 ~/.ssh/id_rsa.pub
|
||
|
||
# Verify SSH service
|
||
systemctl status ssh # on remote server
|
||
```
|
||
|
||
#### 4. Frontend Build Issues
|
||
```bash
|
||
# Clear node modules and reinstall
|
||
cd frontend
|
||
rm -rf node_modules package-lock.json
|
||
npm install
|
||
|
||
# Check for port conflicts
|
||
lsof -i :3000
|
||
```
|
||
|
||
#### 5. Phantom Pending Changes / Orphan Config Versions
|
||
|
||
**Symptom:** Apply Management shows pending changes that can't be applied or rejected, or shows entities from wrong cluster.
|
||
|
||
**Root Cause:** Orphan config versions - versions that reference entities from different clusters or deleted entities (entity ID reuse bug).
|
||
|
||
**Solution (Automatic):**
|
||
- **Modern versions (v1.1.0+):** Orphan versions are automatically detected and cleaned when you:
|
||
- Open Apply Management page (orphans filtered out from display)
|
||
- Click "Apply Changes" (orphans deleted before apply)
|
||
- Click "Reject All" (orphans deleted before reject)
|
||
|
||
**Manual Cleanup (if needed):**
|
||
```sql
|
||
-- Connect to database
|
||
kubectl exec -it <postgres-pod> -- psql -U haproxy_user -d haproxy_openmanager
|
||
|
||
-- Find orphan versions (example for cluster_id=2)
|
||
SELECT cv.id, cv.version_name, cv.cluster_id,
|
||
CASE
|
||
WHEN cv.version_name ~ 'backend-[0-9]+-'
|
||
THEN (SELECT cluster_id FROM backends WHERE id = SUBSTRING(cv.version_name FROM 'backend-([0-9]+)-')::int)
|
||
ELSE NULL
|
||
END as entity_cluster
|
||
FROM config_versions cv
|
||
WHERE cv.cluster_id = 2 AND cv.status = 'PENDING';
|
||
|
||
-- Delete orphan versions (where entity_cluster != 2 or NULL)
|
||
DELETE FROM config_versions
|
||
WHERE id IN (SELECT id FROM ...);
|
||
```
|
||
|
||
**Prevention:**
|
||
- Always delete backends/frontends in the correct cluster
|
||
- Use Apply Changes workflow (don't skip)
|
||
- Keep backend/frontend names unique across clusters
|
||
|
||
#### 6. Apply Changes Stuck - HAProxy Configuration Validation Errors
|
||
|
||
**Symptom:** Pending changes in Apply Management remain stuck and are not applied by agents, or agents report configuration failures.
|
||
|
||
**Root Cause:** The generated HAProxy configuration has syntax errors or validation issues that prevent HAProxy from accepting the new config.
|
||
|
||
**Diagnosis - Check HAProxy Validation on Agent Server:**
|
||
|
||
```bash
|
||
# SSH to the agent server where changes are stuck
|
||
ssh user@agent-server
|
||
|
||
# Check agent logs for validation errors
|
||
sudo tail -f /var/log/haproxy-agent/agent.log
|
||
|
||
# The agent writes the new config to /tmp/haproxy-new-config.cfg before applying
|
||
# Manually validate the configuration file
|
||
sudo /usr/sbin/haproxy -c -f /tmp/haproxy-new-config.cfg
|
||
|
||
# If validation fails, you'll see detailed error messages like:
|
||
# [ALERT] parsing [/tmp/haproxy-new-config.cfg:45] : 'bind' : missing address, port or path
|
||
# [ALERT] parsing [/tmp/haproxy-new-config.cfg:78] : unknown keyword 'ssl-certificate'
|
||
```
|
||
|
||
**Common Validation Issues:**
|
||
|
||
1. **Invalid SSL Certificate Reference:**
|
||
```
|
||
[ALERT] 'bind' : unable to load SSL certificate '/etc/haproxy/certs/missing-cert.pem'
|
||
```
|
||
- Solution: Ensure SSL certificate was uploaded and synced to the agent
|
||
- Check: `ls -la /etc/haproxy/certs/`
|
||
|
||
2. **Backend Server Not Defined:**
|
||
```
|
||
[ALERT] 'use_backend' : unable to find required use_backend: 'backend-name'
|
||
```
|
||
- Solution: Ensure backend exists and is defined before the frontend that uses it
|
||
|
||
3. **Invalid ACL Syntax:**
|
||
```
|
||
[ALERT] parsing [config:42] : error detected while parsing ACL 'acl-name'
|
||
```
|
||
- Solution: Fix ACL syntax in Frontend Management
|
||
|
||
4. **Port Already in Use:**
|
||
```
|
||
[ALERT] Starting frontend GLOBAL: cannot bind socket
|
||
```
|
||
- Solution: Check if another service is using the same port
|
||
|
||
**Recovery Steps:**
|
||
|
||
1. **View Generated Config:**
|
||
```bash
|
||
# On agent server - view the new config that failed validation
|
||
sudo cat /tmp/haproxy-new-config.cfg
|
||
|
||
# Compare with current working config
|
||
sudo cat /etc/haproxy/haproxy.cfg
|
||
```
|
||
|
||
2. **Check Agent Logs:**
|
||
```bash
|
||
# See full error context and rollback messages
|
||
sudo tail -100 /var/log/haproxy-agent/agent.log | grep -A 10 "validation failed"
|
||
```
|
||
|
||
3. **Fix Issues in UI:**
|
||
- Go to Apply Management → View pending changes
|
||
- Identify the problematic entity (frontend, backend, SSL certificate)
|
||
- Edit or delete the problematic entity
|
||
- Reject current pending changes
|
||
- Re-apply after fixing
|
||
|
||
4. **Emergency Rollback:**
|
||
```bash
|
||
# On agent server - agent automatically keeps backup
|
||
sudo ls -la /etc/haproxy/haproxy.cfg.backup*
|
||
|
||
# Manually restore if needed (agent usually does this automatically on validation failure)
|
||
sudo cp /etc/haproxy/haproxy.cfg.backup.TIMESTAMP /etc/haproxy/haproxy.cfg
|
||
sudo systemctl reload haproxy
|
||
```
|
||
|
||
**Prevention:**
|
||
- Always test configurations incrementally (don't make too many changes at once)
|
||
- Use Configuration Viewer to preview generated HAProxy config before applying
|
||
- Ensure SSL certificates are uploaded before creating frontends that use them
|
||
- Keep backend names unique and descriptive
|
||
- Test ACL syntax before saving
|
||
|
||
#### 7. ACME Certificate Issues
|
||
|
||
**Symptom:** ACME certificate request stuck in "pending" or challenge validation fails.
|
||
|
||
**Diagnosis:**
|
||
```bash
|
||
# Check backend logs for ACME errors
|
||
kubectl logs deployment/backend -n haproxy-openmanager | grep "ACME"
|
||
|
||
# Verify ACME challenge endpoint is accessible through the managed HAProxy instance
|
||
curl http://your-domain/.well-known/acme-challenge/test
|
||
|
||
# Check if ACME is enabled on the cluster
|
||
curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/clusters | jq '.[].acme_enabled'
|
||
```
|
||
|
||
**Common Issues:**
|
||
|
||
1. **Challenge validation failed**: Ensure HAProxy cluster has `acme_enabled=true` and HTTP-mode frontends are configured
|
||
2. **DNS not pointing correctly**: The domain must resolve to the IP of the HAProxy instance(s) managed by the cluster's agents
|
||
3. **Rate limited**: Let's Encrypt has rate limits; use staging mode (`acme.staging_mode=true`) for testing
|
||
4. **Stuck orders**: Orders stuck > 24h are logged as warnings; check backend logs
|
||
5. **Firewall blocking**: Port 80 must be open for HTTP-01 challenges from the CA
|
||
|
||
**ACME Auto-Renewal Not Working:**
|
||
```bash
|
||
# Check if auto-renewal is enabled
|
||
curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/settings/acme
|
||
|
||
# Verify background task is running (check backend logs)
|
||
kubectl logs deployment/backend -n haproxy-openmanager | grep "ACME RENEWAL"
|
||
```
|
||
|
||
Settings to verify:
|
||
- `acme.auto_renew_enabled` must be `true`
|
||
- `acme.renew_before_days` controls when renewal triggers (default: 30 days)
|
||
- Certificate must have `auto_renew=true` and `source=letsencrypt`
|
||
|
||
### Debug Mode
|
||
|
||
Enable debug logging:
|
||
|
||
```bash
|
||
# Backend debug mode
|
||
export DEBUG=True
|
||
export LOG_LEVEL=DEBUG
|
||
|
||
# Frontend development mode
|
||
export NODE_ENV=development
|
||
export REACT_APP_DEBUG=true
|
||
```
|
||
|
||
### Log Locations
|
||
|
||
```bash
|
||
# Application logs
|
||
docker logs haproxy-openmanager-backend
|
||
docker logs haproxy-openmanager-frontend
|
||
|
||
# HAProxy logs
|
||
docker logs haproxy-instance
|
||
|
||
# Database logs
|
||
docker logs haproxy-openmanager-db
|
||
|
||
# System logs (on remote servers)
|
||
journalctl -u haproxy
|
||
journalctl -u keepalived
|
||
```
|
||
|
||
|
||
## Contributing
|
||
|
||
1. Fork the repository
|
||
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
||
3. Commit changes (`git commit -m 'Add amazing feature'`)
|
||
4. Push to branch (`git push origin feature/amazing-feature`)
|
||
5. Open a Pull Request
|
||
|
||
### How to Contribute
|
||
|
||
1. **Report Bugs**: Open an issue with detailed reproduction steps
|
||
2. **Suggest Features**: Share your ideas for new features or improvements
|
||
3. **Submit PRs**: Fix bugs, add features, or improve documentation
|
||
4. **Improve Docs**: Help make the documentation clearer and more comprehensive
|
||
5. **Share**: Star the project and share it with others
|
||
|
||
### Development Guidelines
|
||
|
||
- Follow PEP 8 for Python code
|
||
- Use ESLint/Prettier for JavaScript code
|
||
- Add tests for new features
|
||
- Update documentation for API changes
|
||
- Ensure Docker builds work correctly
|
||
- Follow existing code style and conventions
|
||
- Write clear commit messages
|
||
- Keep PRs focused on a single change
|
||
|
||
## Use Cases
|
||
|
||
1. **Enterprise HAProxy Management**: Manage multiple HAProxy clusters across different environments
|
||
2. **DevOps Automation**: Automated configuration deployment with version control
|
||
3. **Multi-Tenant Setups**: Separate pools and clusters for different teams/projects
|
||
4. **Monitoring & Analytics**: Real-time metrics and performance tracking
|
||
5. **Security Management**: Centralized SSL certificate and WAF rule management
|
||
|
||
|
||
## License
|
||
|
||
This project is licensed under the **GNU Affero General Public License v3.0 (or later)** — see the [LICENSE](LICENSE) file for the full text.
|
||
|
||
A short summary:
|
||
|
||
- You are free to use, modify, and distribute this software.
|
||
- If you run a modified version as a network service (e.g., SaaS), you must make the modified source available to its users (AGPL §13).
|
||
- Any redistribution or derivative work must remain under AGPL-3.0-or-later.
|
||
|
||
|
||
## Author
|
||
|
||
**Taylan Bakırcıoğlu**
|
||
Burgan Bank - DevOps / Product Group Manager
|
||
LinkedIn: [linkedin.com/in/taylanbakircioglu](https://www.linkedin.com/in/taylanbakircioglu/)
|
||
|
||
Developed with ❤️ for the HAProxy community
|
||
|
||
|
||
## Support
|
||
|
||
- **Documentation**: This README and inline code documentation
|
||
- **Issues**: [GitHub Issues](https://github.com/taylanbakircioglu/haproxy-openmanager/issues)
|
||
- **Pull Requests**: Contributions are welcome!
|
||
|
||
|
||
## Related Projects
|
||
|
||
- [HAProxy](https://www.haproxy.org/) - The load balancer being managed
|
||
- [FastAPI](https://fastapi.tiangolo.com/) - Backend framework
|
||
- [React.js](https://reactjs.org/) - Frontend framework
|
||
- [Ant Design](https://ant.design/) - UI component library
|
||
|
||
---
|
||
|
||
## Release Notes
|
||
|
||
- **v1.11.1** (2026-08-15) — **A node can no longer be hidden from adoption for good, and Config Import reaches a freshly installed agent**: the agent posts an unmanaged `keepalived.conf` for adoption and caches the hash of what it sent, so the file (which carries the VRRP password) is re-posted only when it changes. Delivery was judged by `curl`'s exit code, which is **0 for 5xx as well**, so a report the server *rejected* was recorded as delivered — and because a hand-maintained config does not change on its own, that node dropped out of *Unmanaged keepalived detected* permanently, curable only by deleting a cache file on the node by hand. Now: the report is cached only on a 2xx; `GET /agents/{name}/keepalived-config` tells the agent whether the server actually holds a discovery for it, so nodes stuck from earlier releases recover by themselves on the next poll; a **400/413/422** records the refusal so identical bytes are not re-posted forever (4xx and 5xx are never sampled out of the request log, so an unattended loop would write a row carrying the whole config every cycle), while **401 and 404 keep retrying** because in this system they mean a token rotation or an agent row briefly absent, not a bad payload. The same exit-code mistake in the *clear* path is fixed too, where it left a managed node still being offered for adoption. Separately, **Config Import silently did nothing on any agent that had never self-upgraded**: `check_config_requests` was defined in the installer and in the self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by `type`, so the operator asked a node for its `haproxy.cfg` and nothing arrived, with no error anywhere. And the cluster whose `keepalived_config_path` is handed to an agent is now resolved deterministically — a pool may hold several clusters, and the unordered join could return a different one between polls, pointing the agent at a file that does not exist. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema change.
|
||
- **v1.11.0** (2026-08-14) — **Unified request/response log with configurable retention**: until now the only record of what happened was `user_activity_logs`, which stores non-GET **2xx** operations with no bodies — so when something failed you could see *that* the count went up, never *what was sent or what came back*. This release adds one queryable timeline covering **both directions**: every inbound API call (**including GETs and including 4xx/5xx**) with the user, client IP, status, duration and — redacted and size-capped — the request and response bodies; and every **outbound** HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows **inherit the inbound request's id**, so one operator action and the CA/DNS calls it triggered read as a single trace — opening a failed *Request Certificate* shows the exact `POST /acme/new-order` and the CA's `429` body underneath it. Capture is a **pure-ASGI middleware that tees** the request and response streams rather than draining them, so no downstream handler is affected (notably the raw-body agent heartbeat), and rows are written by a **batching background writer** with a bounded queue, so the request path never waits on the database and a saturated logger drops rows visibly instead of blocking. Secrets never land: headers are an allowlist (`Authorization`/`Cookie` reduced to a presence marker), body keys and value shapes are redacted (passwords, tokens, API keys, private-key PEMs, JWTs), the **ACME JWS request body is never stored** (a stored `protected`+`signature` pair is a replayable credential — a summary is logged instead), DNS-provider errors record only the exception **type**, and the ACME HTTP-01 challenge endpoint is excluded so `key_authorization` is never captured. **Retention is operator-configurable** in *Settings → Request Log*: separate day counts for successful and failed rows (defaults 7 and 30) plus a hard row cap (500 000), whichever is reached first, pruned in **batches** under a Postgres advisory lock so a multi-million-row table cannot time out the delete or have every replica scan it at once. New **Request Log** page (`requestlog.read`) and retention/purge permission (`requestlog.manage`); `super_admin` and `security_admin` get both, `operator` gets read, `viewer` gets neither. **Successful agent polls are not logged** (`capture_agent_success`, default off; failures always are), which is what keeps the table's size a function of operator activity rather than of node count: measured at 2 424 bytes/row, a 200-node fleet would otherwise write 2.0M rows/day and reach the row cap in six hours, silently reducing the configured 7-day/30-day retention to a few hours for everything in the table. Cost is measured, not estimated: 27.7 µs per request on the hot path, 18.8 µs per row on the writer task, **0.096 % of one core at 500 nodes**. Adds one new table (`request_logs`) and its settings seed — SCHEMA_VERSION 11 → 12 (not 11: that number was taken by v1.10.4 while this was in review, and the version gate would have skipped the migration entirely on every existing install), auto-migrated, no existing table altered, no agent or rendered-config change. Kill switches: `REQUEST_LOG_ENABLED=false` (environment — the middleware is then never registered and costs nothing) or the `enabled` toggle in Settings (no restart).
|
||
- **v1.10.14** (2026-08-14) — **A converged node keeps acknowledging**: the deploy report is the server's only evidence that a member node applied its `keepalived.conf`, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return on every cycle and never reported again, so a **single lost report** — a backend restart, a 5xx, a network blip — left the VIP reading `SYNCING (0/n)` with an empty *Last ack* forever, while the node was demonstrably running the right config. Nothing would ever reconcile the two: the node was correct, the page was not, and the only way out was to change the rendered config so the agent wrote it again. The agent now re-asserts its state on the idempotent path too, which costs one request per node per ~2.5 minutes and touches nothing on the node — keepalived is not reloaded and the file is not rewritten. This is a long-standing gap from the original HA/VIP work, surfaced when acknowledgements were dropped for an unrelated reason in v1.10.12. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema or API change.
|
||
- **v1.10.13** (2026-08-14) — **Agent deploy acknowledgements were silently dropped** (regression in v1.10.12, fix it before or with that release): the takeover-retirement clause added to `POST /agents/{name}/keepalived-status` in v1.10.12 reused one query placeholder for both the assignment `last_deploy_hash=$n` and the comparison inside its `CASE`. PostgreSQL deduces a type per **use**, so the same placeholder came out as `text` in one and `character varying` in the other, and asyncpg rejected the statement with `AmbiguousParameterError`. The failure was not partial: the whole UPDATE never ran, so **no member ever recorded an acknowledgement**. Every VIP sat at `SYNCING (0/n)` with an empty *Last ack*, even after the nodes had deployed the config successfully, and teardown acknowledgements were lost the same way. The hash is now bound to its own placeholder, which is only ever compared against the column and therefore unambiguous. Verified against a real PostgreSQL: both statements execute, a matching hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash` both leave it in place, and every case records the acknowledgement. A test now asserts every `$n` in these statements is bound exactly once and that the count matches the arguments passed. Backend only: no schema, agent or API-shape change.
|
||
- **v1.10.12** (2026-08-14) — **A valid keepalived config is no longer rejected by its own warning**: before writing a rendered `keepalived.conf` the agent validates it with `keepalived -t` and, on failure, keeps the running config and does not restart keepalived. That fail-safe is right, but it treated **any** non-zero exit as invalid, and keepalived's config-test exit code does not separate fatal from benign. Measured on 2.2.8: a clean config exits 0, but `Truncating auth_pass to 8 characters` exits **5** and so does a missing `}` or an `Unknown keyword`. A VRRP password longer than eight characters was therefore enough to make every apply fail, including on nodes whose own running config produces the same warning and has been serving the VIP for weeks. The gate now judges the **output**: messages known to be benign are dropped and anything that remains still fails, so it fails **closed** and an unrecognised message is treated as fatal. Verified against real keepalived: a truncation warning passes while a missing brace, an unknown keyword and a `SECURITY VIOLATION` are all still refused. The agent also **reports what keepalived said** now, in the log and in the status the HA/VIP page shows; discarding it left a correct refusal with no way to act on it. Agent-script change: sync the script from Agent Management and let the agents upgrade for it to take effect. No schema or API change.
|
||
- **v1.10.11** (2026-08-14) — **The *Adoptable* tag names the problem that actually blocks adoption**: the tag and the disabled *Adopt* button were computed separately and could disagree. A pair blocked because its peer's `keepalived.conf` could not be parsed was labelled **MASTER missing** — technically true, since the unreadable node's `state MASTER` had not been counted, but it pointed the operator at the wrong node while the real reason sat in the button's own tooltip. Both now come from one ordered decision, so the label, its colour and the tooltip always describe the condition that stops the adoption; a group held up by an unreadable or unreachable peer reads **blocked by peer**, and two MASTERs is now distinct from none. Display only: what the endpoint accepts or refuses is unchanged. On the public repo this is the first artifact carrying v1.10.4 through v1.10.10: none was released separately, because VIP adoption did not work end to end until these fixes landed.
|
||
- **v1.10.10** (2026-08-14) — **Adoption blockers are listed once per instance**: with the instance-based panel a two-node pair reported the *same* problems about the *same* shared config twice, once per member, and the line numbers differ between the two files so plain de-duplication did not collapse them. Four issues on a pair read as eight, in both the *Adoptable* tooltip and the adopt dialog. They are now merged on the message text with the leading `line N:` ignored, so each distinct problem appears once. Display only: the endpoint already evaluated the combined set and its refusals are unchanged.
|
||
- **v1.10.9** (2026-08-14) — **Adoption refuses to strand a node or silently normalise a peer's settings**: v1.10.8 adopted the whole VRRP instance, but it could only *match* a node it was able to read, that was enabled, and that sat in the same pool. Each of those was a door a real member of the group left through silently — the nodes that remained were rewritten while the one that left kept serving the same address from an unmanaged config. Seen on a live pool: one node of a pair had an unclosed `vrrp_instance` block, so it parsed to nothing while its partner parsed cleanly. Instead of guarding each door, adoption now asks the question directly — is there **any** reported `keepalived.conf` that mentions this virtual address and is not among the nodes being taken over — and refuses naming the node and the reason (unparseable, agent disabled, different pool). Separately, four VIP-level fields (`prefix_length`, unicast/multicast mode, HAProxy tracking and the VRRP password) are stored once and re-rendered onto **every** member, so taking them from whichever node was clicked imposed its settings on the others; the prefix length is the sharpest, because the design refuses to *guess* a netmask for a live VIP and copying one node's netmask onto another is that same change by another name. Adoption now requires the nodes to agree on all four, and compares the VRRP secret by decrypting each node's token (Fernet is non-deterministic, so the ciphertexts cannot be compared). Finally, the takeover authorisation is genuinely one-shot: `takeover_expected_hash` was written at adoption and never cleared, so it stayed valid for that file content indefinitely — it is now retired the moment a member acknowledges our rendered config, gated on the acked hash matching so a failed deploy never drops it. No schema change, no agent change.
|
||
- **v1.10.8** (2026-08-13) — **VIP adoption takes the whole VRRP instance**: adoption used to take only the node whose row was clicked, which broke the exact case the feature exists for, a running HA pair. Adopting the **BACKUP** alone produced a VIP that could never be applied (*exactly one member must be MASTER*); adopting the **MASTER** alone left the peer unmanaged, and adopting it afterwards hit the VRID-collision guard with 409, so the pair could not be completed from the panel at all. Most serious, on a **unicast** instance the single-member render dropped the unicast block entirely — the renderer emits it only when it has peer addresses — so keepalived fell back to **multicast** on the adopted node while its peer stayed unicast: they stop seeing each other and **both** claim the VIP. Adoption now resolves the whole instance, keyed on `(virtual_router_id, virtual address)` exactly as keepalived groups nodes, and every participating node becomes a member with the role, priority and interface its own file declares and its own one-shot takeover hash. It refuses, with the reason, when the group does not have exactly one MASTER, when the nodes disagree on `advert_int`, when a declared unicast peer is not among the nodes being adopted, or when a node already belongs to a live VIP — a rule create/edit enforced and adoption did not. The panel now lists one row per **instance** instead of per node. Two further fixes: the Apply Management **View Change** diff did not recognise the `adopt` action, so it fell through to the generic HAProxy diff and rendered the cluster's entire `haproxy.cfg` as removed; and **rejecting** an adoption hid the node from the panel permanently, because `adopted_vip_id` is write-once, a VIP is only ever soft-deleted (so the column's `ON DELETE SET NULL` never fires) and the agent does not re-report an unchanged file — adoptability is now derived from whether the linked VIP is still active, which self-heals reject, undo-reject and approved teardown alike. No schema change, no agent change.
|
||
- **v1.10.7** (2026-08-13) — **HA / VIP follows the selected cluster**: the page ignored the cluster picker in the header. On a multi-cluster install both the VIP table and the new *Unmanaged keepalived detected* panel listed every cluster's nodes at once and did not change when the selection did, so the panel appeared to be stuck on one cluster's keepalived. Both lists now send `cluster_id`, resolved to that cluster's pool exactly as the Apply Management view already did. The API parameter is **optional**: a caller that omits it still receives the whole fleet, so nothing outside the page changes. This is a deliberate behaviour change for the VIP table, which was fleet-wide before. Backend and frontend only: no schema, no agent change.
|
||
- **v1.10.6** (2026-08-13) — **VIP adoption panel was unreachable**: v1.10.4's *Unmanaged keepalived detected* panel never appeared, even on a fleet where the agents had reported their configs correctly. `GET /discoveries` was declared **after** `GET /{vip_id}` in `routers/vip.py`, and FastAPI matches routes in declaration order, so every request for the discovery list was answered by the get-one-VIP handler, which takes `vip_id: int` and rejected `"discoveries"` with **422** before the real handler ran. Nothing surfaced the failure: the agents reported normally, the rows landed in `vip_discoveries`, and the HA/VIP page treats any non-OK response as "nothing to show" — so the whole feature was invisible with no error anywhere. The route is moved above the parameterised ones, and a static source scan now asserts that **no** literal path in **any** router is shadowed by an earlier parameterised one, so the class of bug cannot come back silently. Data reported under v1.10.4 is not lost: existing `vip_discoveries` rows appear as soon as the fixed backend is deployed, with no agent action needed. Backend-only fix. No schema, API-shape or agent change.
|
||
- **v1.10.5** (2026-08-09) — **HTTP-01 challenge backend on split deployments**: on a deployment where the HAProxy nodes and the management stack are on different hosts, HTTP-01 issuance could fail silently for weeks while DNS-01 kept working — the rendered config pointed `server _acme_mgmt` at an address that resolves **on the HAProxy node**, defaulting to loopback, and every diagnostic still reported success. The per-cluster `acme_backend_url` now has a UI field, changing it actually mints a config version, and the value is validated where it is written. Three adjacent bugs are fixed with it: a config-generation failure was returned as `# Error ...` text and then stored as an APPLIED version and pushed to agents as the cluster's whole `haproxy.cfg` (both call sites now refuse with 422); a nullable `frontends.mode` was interpolated raw and emitted `mode None`, which HAProxy rejects and which takes down the entire cluster config; and cluster creation silently dropped the ACME fields. `docker-compose.yml` now interpolates `PUBLIC_URL` / `MANAGEMENT_BASE_URL` instead of hardcoding them, with the old literals as defaults. Diagnostics read the response body so an SPA answering 200 is no longer counted as healthy, and every new condition is a warning rather than a failure so no install is locked on upgrade. No schema, API-shape or agent change.
|
||
- **v1.10.4** (2026-08-08) — **Adopt an existing keepalived VIP** (Issue #27 follow-up): on a fleet that already runs keepalived, the **HA / VIP** page came up empty, because the flow was one-way — VIPs were declared in OpenManager and pushed to the node, and nothing ever read what was already there. Agents now **report the `keepalived.conf` they find and do not own** (strictly read-only; the node is never touched), the page lists those nodes under *Unmanaged keepalived detected*, and **Adopt** turns one `vrrp_instance` into a managed VIP with the values from the file instead of retyping them. The heartbeat could not drive this: it carries the VIP address and a best-effort MASTER/BACKUP, while rendering a node's config needs **eleven** fields, and guessing them is not cosmetic — a wrong `virtual_router_id` puts the nodes in separate VRRP domains and a wrong `auth_pass` makes them reject each other, so both would claim the VIP. Because adoption **replaces** the operator's file with OpenManager's render, the parser reports every directive it cannot reproduce — a `notify_master` hook, an LVS `virtual_server` section, a `vrrp_sync_group`, a second address in one instance, a custom `track_script` — and **refuses** while any remain; the operator can waive that class explicitly, but a value that is simply *unknown* (an absent VRID or prefix length) can never be waived, only supplied. keepalived's own documented defaults (`state BACKUP`, `priority 100`, `advert_int 1`) are applied and shown as assumed. The agent's ownership guard is **not** weakened: adoption authorises exactly **one** takeover of exactly the file that was analysed, pinned to its hash, so a config edited between adoption and Apply is still refused. The adopted VIP is created **PENDING** like any other, so nothing reaches the node until it is applied from Apply Management. VRRP passwords are Fernet-encrypted at ingest and masked in the stored copy and the preview. Schema change: one new table `vip_discoveries` plus two additive columns (SCHEMA_VERSION 10 → 11, auto-migrated, no existing table altered) — **see the upgrade notes: this bump re-seeds the four built-in roles, and the Linux agent script must reach the nodes before discovery starts**.
|
||
- **v1.10.3** (2026-08-08) — **Multi-account ACME: the certificate wizard honours the account you pick**: with more than one ACME account registered, picking an **HTTP-01** account in *Request ACME Certificate* still produced a **DNS-01** request. Three faults compounded. (1) `Form.useWatch` reports only fields that are currently **rendered**, and the account `Select` lives on the *Configuration* step — so as soon as the wizard advanced to *Review* the watch read `undefined` and the wizard silently reverted to the default account, even though the value was still in the form store; the watches now pass `preserve: true`. The same fault disabled the **wildcard guard** on *Review*, the one step where Submit lives. (2) The UI and the backend disagreed on which account is the *default*: the backend takes the **newest** valid account (`ORDER BY created_at DESC`), the UI took the **oldest** entry of a list ordered by id — the opposite account whenever the two differ. The wizard now resolves the same one, and sends `account_id` **explicitly** so there is no guess left to disagree about. (3) `account_id` was read from the form store while `challenge_type` came from the reverted account object, so the request asked for DNS-01 validation on an HTTP-01 account and the API answered `The selected ACME account has no DNS provider configured for DNS-01.` — both are now derived from one resolved account. The *Review* step also showed the default account's address instead of the chosen one, and Submit stayed enabled for a deactivated account; both fixed. Frontend only — no schema, API-shape, agent or rendered-config changes, and single-account installations behave exactly as before.
|
||
- **v1.10.2** (2026-08-08) — **Dark mode fixes on Apply Management**: several panels on the Apply Management page were painted with light-mode colour literals, so in dark mode the **Pending Changes** box rendered as a cream panel with light text on it — measured contrast **1.03:1**, effectively unreadable, now **11.50:1**. The same bug affected the added/removed rows in the *View Change* diff (2.21:1 and 2.99:1, now 5.49:1 and 4.01:1), the ACME and pending-version panels, the VIP pending-delete row, and the agent-error recommendation box; all now derive from theme tokens. Separately, **static confirm dialogs came up white in dark mode**: in Ant Design 5 the static `Modal.confirm` / `message` / `notification` APIs render into their own detached root and never see the app's `ConfigProvider`, so they always used the light algorithm. Registering `ConfigProvider.config({ holderRender })` once at the app root fixes **every** static dialog in the application (12 components use them), not only this page. Light mode is byte-identical — each token resolves under the default algorithm to exactly the literal it replaced. Frontend only: no schema, API, environment or agent change.
|
||
- **v1.10.1** (2026-08-08) — **CSR private key encrypted at rest** (Issue #53): the private key of a **pending** CSR is now Fernet-encrypted in the database instead of stored as PEM. It is the one key in the system worth protecting this way — it sits idle for the entire signing window (days to weeks), is never transmitted to an agent, and is destroyed the moment the signed certificate is imported; `ssl_certificates.private_key_content` and the ACME order keys are unchanged, because agents must receive those in plaintext on every poll. The token replaces the PEM in the **same column**, so there is **no schema change and no `SCHEMA_VERSION` bump** (and therefore no re-seed of the built-in roles). CSRs created before this release keep a raw PEM and are still read transparently, so anything already out for signature imports normally with no data migration. The key derives from `SECRET_KEY` via HKDF with its own info string, independent of the VIP/MFA/DNS keys, and an optional `CSR_ENCRYPTION_KEY` enables independent rotation — rotating `SECRET_KEY` without it makes pending CSR keys unrecoverable, which now fails with an explicit "delete and re-create this CSR" error rather than a misleading key-mismatch. `.env.template` now documents all four per-purpose encryption keys. No API, UI or agent change.
|
||
- **v1.10.0** (2026-08-07) — **GoDaddy DNS provider for DNS-01** (Issue #35 follow-up): DNS-01 challenges can now be published and cleaned up automatically through **GoDaddy**, alongside the existing Manual and Cloudflare providers, so wildcard and internal-cluster certificates on GoDaddy-hosted zones **renew unattended**. Credentials are a **Production API Key + Secret** pair from `developer.godaddy.com/keys` (a **Personal Access Token** also works — paste it as the Key and leave the Secret blank, which is the forward path as GoDaddy retires `sso-key`); they are **verified against the GoDaddy API before being saved** and **encrypted at rest** (Fernet, the same path as Cloudflare), and are never returned by the API, logged, or written to an order event. GoDaddy's v1 API has **no per-value TXT write** — `PUT` replaces an entire RRset — so add/remove are read-modify-write with sibling values merged back, empty-`data` tombstones filtered out, and `DELETE` used for the last value (`PUT []` is rejected); this is what keeps the **apex + wildcard** case (two TXT values at one `_acme-challenge` name) working, and the record path is hard-gated so it can never collapse onto the zone-wide endpoint that would wipe SPF/DKIM/DMARC. Zone lookup probes the records API rather than the domain listing, so **delegated sub-zones** resolve and small accounts are not falsely rejected. Registry-only addition: one new provider module plus one registry line — no frontend change (the credential form is schema-driven). No schema, API-shape, agent, or rendered-config changes; Manual, Cloudflare and HTTP-01 are unaffected.
|
||
- **v1.9.0** (2026-08-04) — **CSR creation** (in-app key + CSR generation and signed-certificate import): a new **CSR tab** on the SSL Certificates page generates a private key and Certificate Signing Request server-side (RSA 2048/4096 or ECDSA P-256/P-384; full subject — O/OU/L/ST/C/email — plus DNS SANs with wildcard support), for certificates signed by an **external or corporate CA**. The operator downloads/copies the CSR PEM, has it signed, then imports the signed certificate (+ optional chain): the backend verifies the certificate against the stored key (hard gate), rejects expired certs, warns on SAN drift, and creates a normal SSL certificate entry (source `CSR`) that flows through the standard **PENDING → Apply Management → agent pull** pipeline. The private key **never leaves the server** — no CSR endpoint returns it, and after import the CSR row's key copy is destroyed (the key then lives only on the certificate, like every other key). Additive schema change: one new table `ssl_csrs` (SCHEMA_VERSION 9 → 10, auto-migrated, no existing table altered); key generation runs off the event loop and is rate-limited per user; existing `ssl.*` permissions govern all new endpoints. No agent or rendered-config changes.
|
||
- **v1.8.10** (2026-07-20) — **Security hardening** (GHSA-7rhv-c5pc-69r8, GHSA-3p5c-m5m4-mjpx, GHSA-3vh4): three advisory classes remediated, backend-only, no agent changes. (1) **RCE**: the agent script-template read/write endpoints now require the `agents.version` permission on top of authentication — a poisoned template is executed as root on every HAProxy node, so authentication alone was insufficient. (2) **Missing authentication**: operator/UI endpoints that were served without a JWT (dashboard stats, pool/cluster listings, agent inventory, WAF rules, config validate/optimize, SSL config-versions, health deep/agents/clusters) are now gated by a `require_authenticated_user` dependency, and agent data-plane endpoints that treated the `X-API-Key` header as *optional* (heartbeat, config, ssl-certificates, upgrade-status, pending-requests) now hard-reject a missing key. In every case the auth check was moved **ahead of** the handler's `try:` block so a 401 can no longer be rewritten into a 500 by the generic exception handler. (3) **SSRF**: a new `utils/ssrf_guard.py` (https-only, IPv4-pinned connector, all resolved addresses must be public, no redirects) protects the ACME directory fetch, the signed-request target and the ACME connection test, which accept operator- or DB-supplied URLs; the connection test also stopped reflecting arbitrary upstream JSON. Frontend dependency advisories patched in the same release. No schema, API-shape or rendered-config changes.
|
||
- **v1.8.9** (2026-07-13) — **ACL `-f` pattern-file support** (Issue #38 follow-up): ACL definitions that reference a host-side pattern file (`acl … -f /etc/haproxy/lists/blocked.lst`) are accepted on import and edit instead of being rejected. The referenced file lives on the HAProxy node and cannot be validated from the manager, so the manager emits an **advisory warning** rather than a hard rejection and lets the agent's `haproxy -c` check be the fail-safe gate (a broken reference fails validation on the node and the previous config is restored). Consistent with the SPOE handling introduced in v1.8.8.
|
||
- **v1.8.8** (2026-07-10) — **SPOE filter and frontend `log-format` preserved on import/edit** (Issue #38): importing an existing `haproxy.cfg` or editing a frontend silently dropped `filter spoe …` directives and custom `log-format` lines, so the next Apply pushed a config that had lost them. Both are now round-tripped through import and edit. As with `-f` pattern files, the SPOE engine config is a host-side file the manager cannot read, so it is preserved verbatim and reported as an advisory rather than validated centrally.
|
||
- **v1.8.7** (2026-07-09) — **Version reporting single-source fix**: the version shown in the UI (backend-sourced via `/api/version`) could lag behind the real release. The canonical version lived in the repo-root `version.json`, but the backend image is built from the `./backend` context, so that file did not reach the container in every pipeline; the backend then fell back to a hardcoded constant in `main.py` that had to be bumped by hand and had drifted (it reported 1.8.4 after 1.8.5/1.8.6 shipped). The version now lives in a single file, `backend/version.json`, baked into every image automatically, and `main.py` no longer carries a real version literal (its fallback is a neutral "unknown"). A new test enforces that the version stays single-source and cannot drift. No functional or API change.
|
||
- **v1.8.6** (2026-07-06) — **Performance: opt-in API workers + heartbeat micro-optimization** (Issue #35 follow-up): the backend container can now run multiple uvicorn worker processes via the new `UVICORN_WORKERS` environment variable (default **1** — behavior unchanged unless you opt in), letting the API use all cores on multi-core hosts; background tasks were already multi-replica safe, as exercised by the Kubernetes HPA deployment. The agent heartbeat handler now reads the agent's `status`/`version`/`upgrade_status` in one query instead of three (one round-trip per heartbeat, per agent, every 30s). Added a *Performance Tuning* section to the README (worker/replica scaling and how to use the `X-Response-Time` header and `Slow request detected` logs to pinpoint slow endpoints). Zero-risk release: no schema, API, or agent changes; defaults preserve existing behavior exactly.
|
||
- **v1.8.5** (2026-07-03) — **ACME completion-task SQL fix** (Issue #35 follow-up): the background order-completion task (`complete_pending_acme_orders`, runs every 60s) died on **every cycle** with `syntax error at or near ")"` — an extra closing parenthesis introduced in v1.8.0's bounded DNS-01 retry claim query. Because that query is the task's first database call, **no background ACME work ran at all from v1.8.0 through v1.8.4**: orders were never claimed for finalize/download, the DNS-01 TXT record was never published (so DNS-01 with an automated provider such as Cloudflare could never validate), Site Wizard staged orders never left `wizard_staged`, and DNS-01 retry/TXT-cleanup never executed. The stray parenthesis is removed and a regression test now scans all ACME modules' SQL for unbalanced parentheses (the unit suite mocks the database, which is why a raw-SQL syntax error could slip through). One-line backend query fix; no schema, API, or agent changes — fully backward compatible.
|
||
- **v1.8.4** (2026-06-27) — **Agent installer self-kill fix** (Issue #31): the Linux/macOS agent installer could abort during "pre-installation cleanup" (terminal showed `Killing processes matching: haproxy-agent` then `Killed`) when the install script's own filename contained "haproxy-agent". The cleanup killed processes by matching the bare string "haproxy-agent" against full command lines, which also matched the running installer (and a `sudo`/PAM ancestor the self-exclusion did not cover), so the installer terminated itself. Cleanup now targets only the installed agent (the `$INSTALL_DIR/haproxy-agent` binary and the agent service), never the bare string, and the UI now names the downloaded scripts `install-agent-<platform>.sh` / `uninstall-agent-<platform>.sh`. Installer-only change; the running agent and its privilege model (it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade) are unchanged.
|
||
- **v1.8.3** (2026-06-25) — **Agent heartbeat JSON fix** (Issue #31): a self-hosted agent could fail every heartbeat with `HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host, leaving a stray comma in the hand-built heartbeat JSON. The agent script now substitutes a valid placeholder when that block is empty so it can no longer emit a stray comma, and the backend heartbeat endpoint now parses valid payloads as-is and, only when a body fails to parse, tolerates that specific malformed pattern (a leading or doubled comma) so an already-deployed agent recovers on its next heartbeat after this build is deployed. Backend + agent-script only; healthy agents of every version are byte-for-byte unaffected.
|
||
- **v1.8.2** (2026-06-25) — **ACME nonce fix** (Issue #35 follow-up): the ACME client now scopes the anti-replay nonce **per certificate authority** so a nonce issued by one CA is never sent to another. This fixes ZeroSSL/Google account registration failing with `malformed: The Replay Nonce could not be base64url-decoded` (the client previously shared one nonce across CAs and only auto-retried on `badNonce`). Account registration now always uses a fresh nonce from the target CA, and the retry covers this case too. Backend-only; HTTP-01 and Let's Encrypt are unaffected.
|
||
- **v1.8.1** (2026-06-24) — **ACME DNS-01 fixes** (Issue #35 follow-up): Cloudflare API tokens are now sanitized so a pasted token with quotes/spaces no longer fails with "Invalid request headers"; ZeroSSL/Google **External Account Binding (EAB)** can be entered per-account in the register dialog and EAB-required failures show a clear message; and **Apply Management** now categorizes cluster ACME enable/disable changes under their own "ACME Challenge Routing" section and **Apply/Reject All** correctly process them (previously "Rejected 0 HA/VIP change(s)"), consistent with every other entity. Fully backward compatible.
|
||
- **v1.8.0** (2026-06-23) — **ACME DNS-01 challenge support** (Issue #35): Auto SSL can now validate via a **DNS TXT record** (`_acme-challenge.<domain>`) instead of HTTP-01 on port 80, enabling certificates for **internal/isolated clusters with no public ingress** and **wildcard** certificates (`*.example.com`). Pluggable **per-account DNS provider** (Manual + Cloudflare to start; credentials verified on save and **encrypted at rest**, never returned by the API or logged), the same **PENDING → APPLIED** pipeline, a **bounded automatic retry** on propagation lag, and a **DNS-01 event timeline** in the order detail. **Opt-in** via Settings → ACME (global switch, default off); **HTTP-01 is byte-for-byte unchanged**, with **zero agent or rendered-config changes**. Manual DNS-01 certificates cannot auto-renew unattended; the UI states this and disables auto-renew for them.
|
||
- **v1.7.8** (2026-06-07) — HA / VIP apply progress now shows **per-node** convergence: a multi-node VIP's apply popup reads "Syncing HA/VIP… 1/2 node(s) converged" (matching the HA/VIP table) instead of a coarse per-change count. Frontend-only.
|
||
- **v1.7.7** (2026-06-07) — HA / VIP apply-progress consistency: applying a VIP change (or approving a delete) used to flash the progress popup green instantly while the HA/VIP page still showed `SYNCING (0/1)` for a couple of minutes. The popup now **keeps showing "Syncing HA/VIP… X/Y node(s) converged"** until each member node reports the VIP `ACTIVE` (create/edit) or fully torn down (delete) — exactly like the HAProxy agent-sync widget — then completes green. It's a fire-and-forget background poll (the Apply button is released immediately), bounded at ~5 min so an offline node can't spin forever (then it completes with an informational "still converging — track on the HA/VIP page"). Frontend-only; no backend/agent/schema change.
|
||
- **v1.7.6** (2026-06-07) — HA / VIP UX + accuracy polish: (1) the on-prem/L2 cloud caveat is now a **subtle, collapsed-by-default "Network requirements" info link** instead of a prominent yellow warning. (2) The delete dialog is simplified — deletion is **always a graceful teardown** (stop & disable keepalived, remove our config, release the VIP, keep the package); the confusing "also uninstall the package" checkbox was removed (it was a no-op on any node whose keepalived predates the install marker, and package removal is better handled as a deliberate node-decommission step — the `purge_package` API remains for that). (3) The agent now reports keepalived **FAULT** state (e.g. when the chosen interface has no usable IPv4) instead of misreporting it as BACKUP, so a misconfigured VIP shows red/FAULT in the UI. (1)+(2) are frontend-only; (3) is an additive agent-script change — push it via **Agent Script Management → Reset to Defaults**, then **Upgrade**.
|
||
- **v1.7.5** (2026-06-07) — HA / VIP "View Change" fix: editing a VIP (e.g. priority + virtual IP) now shows a **real line diff — only the lines that actually changed** — in Apply Management's View Change, instead of rendering the whole `keepalived.conf` as "added". Also fixes a doubled `+ +` prefix (the VIP diff now stores lines without a +/- prefix, matching the standard config diff). View-only; no schema, agent, apply, or render change.
|
||
- **v1.7.4** (2026-06-06) — HAProxy config-generator robustness fix: a frontend that uses a stick counter (`track-sc<N>` or an `sc_*_rate(...)` fetch, e.g. a rate-limit `http-request deny if { sc_http_req_rate(0) gt N }`) but declares **no `stick-table`** caused HAProxy to fatally reject the whole cluster config with *"table '<frontend>' used but not configured"*. This happened where rate-limit directives had been baked into a frontend's stored `request_headers`/`options` (by an older version or a config import). The generator now **auto-injects a default `stick-table`** in that case. Purely additive — it only fires when a counter is used and no table exists (a config that was already invalid), so frontends that already declare a stick-table or don't rate-limit are byte-unchanged.
|
||
- **v1.7.3** (2026-06-06) — HA / VIP backward-compat fix: the v1.7.2 deletion-tracking list now only resurfaces VIPs deleted through the **new approval flow** (gated on `last_config_status='APPLIED'`), so a VIP soft-deleted under an earlier version's immediate-delete is no longer shown as `DELETING`. Display-only; no schema or agent change.
|
||
- **v1.7.2** (2026-06-06) — HA / VIP safety & visibility follow-up:
|
||
- **Approval-gated deletion (safety).** Deleting a *running* VIP from the UI no longer takes effect immediately — it is **staged for Apply Management** and the VIP **keeps running, untouched**, until you **Approve** it (Reject keeps it). The agent is told to tear keepalived down **only after approval**, so a misclick can never tear down a production VIP — an agent never deletes without an explicit human approval. The deletion is trackable through the standard Apply popup and a **DELETING** status on the HA/VIP tab. (A VIP that was never applied is removed at once — nothing is running to tear down.)
|
||
- **Diagnostics view** — a per-VIP search-icon modal showing each node's keepalived deploy state, the message it reported, last-ack time and live VRRP state (handy while a fresh install is SYNCING, ~30s), plus the exact node-side log commands.
|
||
- **Opt-in package uninstall** — the safe default keeps the keepalived package (stop & disable, remove our config, release the VIP); a default-off checkbox additionally uninstalls the package **only on nodes where OpenManager installed it** (an admin's pre-existing keepalived is never removed, tracked via an install marker).
|
||
- Adds additive `purge_on_teardown` + `pending_delete` columns; `SCHEMA_VERSION` 5 → 7 (idempotent; existing data/passwords unaffected). Agent script updated — push it via **Agent Script Management → Reset to Defaults**, then **Upgrade** agents.
|
||
- **v1.7.1** (2026-06-06) — HA / VIP follow-up: a VIP can now be created on a **single node** — a Keepalived-managed floating IP **without** failover (e.g. a one-box HAProxy that wants a stable address, or before a second node is added). Add a second node anytime for real VRRP failover. A single-node VIP renders a clean **multicast** config (no bare `unicast_src_ip`, which `keepalived -t` rejects); multi-node behaviour is unchanged. The Create-VIP form now auto-selects the first chosen node as **MASTER** and shows an in-UI notice that, on Apply, Keepalived is **installed automatically from the node's OS package repositories** (apt/dnf/yum/zypper/apk — the node must reach its repos / an internal mirror), while a hand-managed Keepalived is still left untouched. No schema change.
|
||
- **v1.7.0** (2026-06-05) — Feature (Issue #27): **HA / VIP (Keepalived) management** from the UI. A new "HA / VIP" tab lets you create a virtual IP, pick a per-node interface, and select which pool nodes participate (with MASTER/BACKUP roles + priorities); on Apply, the agent installs & configures Keepalived (unicast VRRP, cloud-safe default) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops, and the tab shows live MASTER/BACKUP per node. Fully **opt-in and backward compatible** — nodes/clusters without a VIP are untouched, and a node already running a hand-managed Keepalived is detected and never overwritten (reported as "externally managed"). Pending VIP changes can be **Rejected** to fully restore the last applied state. Keepalived is installed across the major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine), and live MASTER/BACKUP detection works across distros/init systems (journald + log files + portable interface-based detection). On-prem/L2 scope (VRRP); a clear in-UI notice covers the cloud caveat. Adds two new tables (`vip_instances`, `vip_members`) — `SCHEMA_VERSION` bumps to 3 (idempotent re-run; existing data and passwords unaffected). A new "HA / VIP" tab lets you create a virtual IP, pick a per-node interface, and select which pool nodes participate (with MASTER/BACKUP roles + priorities); on Apply, the agent installs & configures Keepalived (unicast VRRP, cloud-safe default) with a HAProxy health-check so the VIP fails over automatically when HAProxy drops, and the tab shows live MASTER/BACKUP per node. Fully **opt-in and backward compatible** — nodes/clusters without a VIP are untouched, and a node already running a hand-managed Keepalived is detected and never overwritten (reported as "externally managed"). Pending VIP changes can be **Rejected** to fully restore the last applied state. Keepalived is installed across the major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine), and live MASTER/BACKUP detection works across distros/init systems (journald + log files + portable interface-based detection). On-prem/L2 scope (VRRP); a clear in-UI notice covers the cloud caveat. Adds two new tables (`vip_instances`, `vip_members`) — `SCHEMA_VERSION` bumps to 3 (idempotent re-run; existing data and passwords unaffected).
|
||
- **v1.6.5** (2026-06-02) — Security: re-pinned the bundled nginx reverse-proxy image to `nginx:1.31.1-alpine` (mainline patched release) for the nginx "poolslip" advisory (fixed in mainline 1.31.1+ / stable 1.30.2+). Supersedes the v1.6.4 stable pin. No config, schema, or behavior changes.
|
||
- **v1.6.4** (2026-06-02) — Security: pinned the bundled nginx reverse-proxy image to a patched stable release (`nginx:1.30.2-alpine`) for the nginx "poolslip" advisory (mainline ≤ 1.31.0 affected; fixed in stable 1.30.2+). The product's nginx config uses no `rewrite` capture groups, so the config-level mitigation did not apply — the fix is the version pin. No config, schema, or behavior changes.
|
||
- **v1.6.3** (2026-06-01) — Bugfix: a backend **server** toggled OFF (`is_active=false`) disappeared from the UI with no way to reactivate it. `GET /api/backends` now honors `include_inactive` for servers (previously only backends), so disabled servers stay visible with an OFF switch + "Inactive" tag and can be re-enabled; soft-deleted (pending-delete) servers stay hidden. A Reject of a server toggle now correctly rolls back `is_active` (entity snapshot). Startup migrations are hardened for multiple replicas / rolling deploys (advisory lock + schema-version gate, so an already-current schema isn't re-migrated under a serving peer's load). Version is reported consistently across all layers. No schema changes; config generation unchanged (disabled servers stay commented out).
|
||
- **v1.6.2** (2026-05-30) — Bugfixes: (1) agents (which authenticate with their `X-API-Key` token) could not reach `GET /api/clusters` / `/api/clusters/{id}` after the v1.5.x cluster-read hardening, breaking agent assignment ("401: Authorization header missing"); these endpoints now accept either a user JWT or an agent token (anonymous access is still rejected). (2) The uninstall-script generator returned 400 for macOS agents (which report platform `darwin`); it now normalizes the platform the same way the install generator does. No UI or schema changes.
|
||
- **v1.6.1** (2026-05-21) — Security patch: bump `axios` to 1.16.x (prototype-pollution hardening, header-injection fix, keep-alive memory leak fix) and `fast-uri` to 3.1.2 (GHSA-v39h-62p7-jpjc). No functional changes.
|
||
|
||
For full release notes and the list of features delivered in each version (v1.5.x Site Wizard + ACME Diagnostic Panel, v1.4.0 ACME stability + enterprise audit, v1.3.0, ...) see the [GitHub Releases](https://github.com/taylanbakircioglu/haproxy-openmanager/releases) page.
|
||
|
||
---
|
||
|
||
**Made with ❤️ for the HAProxy community**
|