The Linux/macOS agent installer could abort during "pre-installation cleanup"
(terminal showed `Killing processes matching: haproxy-agent` then `Killed`,
returning to the prompt) when the install script's own command line contained
"haproxy-agent". The cleanup killed processes via `pgrep -f "$pattern"` starting
with the bare string "haproxy-agent", which also matched the running installer's
own command line and a sudo/PAM ancestor that the $$/$PPID self-exclusion did not
cover, so the installer terminated itself before installing.
- linux_install.sh / macos_install.sh: the cleanup kill loop now targets ONLY
the installed agent - "$INSTALL_DIR/haproxy-agent" (the daemon binary path) and
the agent service/label ("haproxy-agent.service" / "com.haproxy.agent") - never
the bare "haproxy-agent" substring. Neither pattern can match the installer's
own command line. The redundant bare pattern is dropped (the service is stopped
separately, and the binary-path pattern still catches a running daemon).
- frontend (AgentManagement.js): name the downloaded scripts
install-agent-<platform>.sh / uninstall-agent-<platform>.sh (matching the
backend's suggested filename) - defense in depth so this cannot resurface.
Installer-only change. The running agent and its privilege model are unchanged
(it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade).
The cleanup runs only on a full interactive install (gated by SKIP_TO_DAEMON), so
daemon mode, self-upgrade, and config/version apply are unaffected. Both agent
scripts kept in sync. Scripts parse on bash 4.2-5.2; full backend suite green.
Addresses #31.
A self-hosted agent could fail every heartbeat with HTTP 400
`Invalid JSON: Expecting property name enclosed in double quotes` when the
system-info block it collects came back empty on an unusual host. The agent
builds the heartbeat JSON as text, so an empty `$system_info` collapsed the
`$system_info,` line to a bare comma and broke the payload.
- Agent script (linux + macos, kept in sync): guard the fragment-form
register_agent and send_heartbeat builders so an empty system_info falls back
to a valid key and can never emit a bare comma. Uses the most portable bash
glob test (no POSIX class / pattern substitution; verified on bash 3.2-5.2 and
on Ubuntu/Debian/Rocky/Alpine/Amazon Linux). True no-op for healthy agents.
- Backend heartbeat endpoint: parse the body as-is first and only run the
malformed-JSON repair when parsing fails, so a valid heartbeat from any agent
version is byte-for-byte untouched. The repair (now a testable helper) recovers
a leading or doubled comma (the empty-system_info artifact) in addition to the
existing empty-value / trailing-comma fixes.
No agent version bump; self-upgrade and daemon mode are unaffected. Healthy
agents of every version behave identically. Full backend suite green.
Addresses #31.
Manage highly-available virtual IPs backed by Keepalived (VRRP) directly from the OpenManager
UI — no more SSHing into nodes to install/configure Keepalived by hand. Builds on the agent
pull-architecture: define the VIP centrally, click Apply, and the agents converge.
Highlights:
- New "HA / VIP" tab: create a virtual IP, pick a per-node interface, select which pool nodes
participate (MASTER/BACKUP roles + priorities); live MASTER/BACKUP/FAULT per node.
- On Apply, agents install & configure Keepalived (unicast VRRP, cloud-safe default) across the
major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine) with a
HAProxy health-check, so the VIP fails over automatically when HAProxy drops.
- Single-node (a managed floating IP without failover) and multi-node VRRP failover both work.
- VIP changes ride the standard Apply Management flow with the standard "View Change" diff.
- Approval-gated deletion (safety): deleting a running VIP is staged for approval and the VIP
keeps running, untouched, until you approve it — an agent never tears a VIP down without an
explicit human approval. Per-VIP Diagnostics view; opt-in package uninstall (only on nodes
where OpenManager installed it). A node already running a hand-managed Keepalived is detected
and never overwritten ("externally managed").
- Fully opt-in and backward compatible: nodes/clusters without a VIP are unaffected. Adds
vip_instances + vip_members tables (idempotent SCHEMA_VERSION bump; existing data and
passwords unaffected) and a `vip` RBAC permission group.
- Also includes a HAProxy config-generator robustness fix: auto-inject a stick-table when a
frontend uses a stick counter (track-sc / sc_*_rate) but declares none.
On-prem / L2 (VRRP) scope; the UI notes the cloud caveat.
Assigning a HAProxy agent failed with '401: Authorization header missing'
on GET /api/clusters. A cluster-read hardening had made GET /api/clusters and
GET /api/clusters/{id} accept only a user JWT in the Authorization header;
agents authenticate with their agent token in the X-API-Key header, so the
token was never read.
Both endpoints now accept either a user JWT (Authorization) or an agent token
(X-API-Key via validate_agent_api_key), mirroring the existing dual-auth on
POST /api/agents/generate-install-script. Anonymous access is still rejected,
so the original hardening is preserved. The auth guard is placed before the
try block so the failure surfaces as a clean 401 (not the 500-wrapped-401 in
the report). Agent install scripts now consistently send the token via
X-API-Key (pre-flight cluster check on linux/macos, and macOS get_cluster_paths
which previously used the wrong Authorization: Bearer header).
Also normalizes the platform in the uninstall-script generator so macOS agents
(which report platform 'darwin') no longer get a 400 from
GET /api/agents/generate-uninstall-script/darwin.
version 1.6.0 -> 1.6.2.
- Agent scripts now detect and send ip_address in DAEMON heartbeat (Linux: ip route, macOS: ifconfig)
- Backend validates agent-reported IPs via ipaddress stdlib, COALESCE preserves existing on NULL
- IP/VIP change logging (non-critical, try/except wrapped) for operational visibility
- New source_file_hash column on agent_script_templates for reliable update detection
- Migration changed to ON CONFLICT DO NOTHING to prevent overwriting UI-customized scripts on restart
- GET /versions returns script_update_available flag (disk hash vs DB hash comparison with fallback)
- Frontend Alert banner warns users of new agent script versions and directs to Reset to Defaults
- Reset to Defaults and Popconfirm modals explicitly warn about custom script edit loss
- Full backward compatibility: old agents without ip_address field continue working unchanged
Made-with: Cursor
Agent startup left HAPROXY_CONFIG_PATH empty when the /api/clusters jq
select() returned no output (e.g. transient API failure or cluster_id
mismatch). This caused "No existing HAProxy config found at: " errors
and partial config merge failures for newly added agents.
Three-layer fix:
1. Load HAProxy paths from config.json as baseline before get_cluster_paths()
2. Use local variables in get_cluster_paths() - only override globals when
API returns non-empty values (defensive against jq select() empty output)
3. Dynamically update paths from /api/agents/{name}/config response in
check_config_updates() - allows cluster path changes without reinstall
Applied to both linux_install.sh and macos_install.sh.
Made-with: Cursor
- Fix DB: COALESCE(NULLIF) prevented clearing stale MASTER/BACKUP state.
Use CASE WHEN IS NOT NULL to distinguish absent fields (old agents)
from empty fields (keepalived stopped) and properly clear to NULL.
- Fix Redis: explicitly delete cache key when keepalived stops instead
of relying on 90s TTL expiry, preventing stale dashboard data.
- Fix agent scripts: SKIP_TO_DAEMON send_heartbeat now always sends
keepalive_state/keepalive_ip fields (even empty) so backend can
detect stopped keepalived and clear stale data.
- Fix legacy heartbeat endpoint: POST /{agent_id}/heartbeat now also
updates keepalive_state and keepalive_ip fields.
- UI: add horizontal scroll to ClusterManagement table to prevent
column overflow with new Keepalive column.
- UI: add VIP tooltip to Dashboard AgentStatusCard keepalive tag.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add keepalive_state and keepalive_ip columns to agents table (migration + schema)
- Add keepalive fields to AgentHeartbeat Pydantic model (backward compatible)
- Update heartbeat endpoint to persist keepalive data to DB and cache in Redis
- Add multi-method keepalived detection in agent scripts (journalctl, log files, VIP check)
- Update dashboard-stats agents/status API with Redis-first keepalive lookup
- Update GET /api/agents to include keepalive_state and keepalive_ip
- Show MASTER/BACKUP tag in Dashboard AgentStatusCard
- Show keepalive info in Agent Management registered agents table
- Add Keepalive column to Cluster Management table with VIP search support
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add per-cluster SSL sync status tracking with APPLYING/SYNCED/PARTIAL states
- Add timeout detection (10min) for stalled SSL deployments showing PARTIAL status
- Add SSL-specific info in Apply Management dialog (cluster count, auto-apply notice)
- Add Deployment Status tab in SSL details showing per-cluster agent sync progress
- Implement SSL auto-reject (reject propagates to all clusters) and auto-undo
- Add conditional entity rollback for SSL reject (time-window based safety check)
- Filter REJECTED status from SSL last_config_status display
- Add pending_cluster_names to SSL list API response
- Optimize agent SSL reload: only trigger HAProxy reload when changed cert is
actually referenced in the cluster's HAProxy config (grep -qF check), preventing
unnecessary reloads on clusters that don't use the updated certificate
- Applied to all 4 SSL deploy paths: Linux/macOS standalone and daemon embedded modes
Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause: After agent self-upgrade, the embedded daemon (SKIP_TO_DAEMON
block) runs instead of run_daemon(). HAPROXY_BIN and HAPROXY_CONFIG
variables were uninitialized before the daemon loop, causing SSL-triggered
HAProxy reloads to silently fail with empty path validation.
Changes:
- Initialize HAPROXY_BIN/HAPROXY_CONFIG before embedded daemon loop
- Add md5 checksum comparison in deploy_ssl_certificates() to detect
actual cert file changes (avoid unnecessary writes and reloads)
- Add check_ssl_updates() for independent SSL sync every ~2.5 min
in run_daemon(), independent of config version changes
- Add SSL-aware reload in check_config_updates(): if config validation
fails but SSL certs changed, reload HAProxy with existing config
- Add "full" fetch mode to fetch_and_deploy_ssl_certificates() to
bypass incremental timestamp filter for standalone SSL checks
Applied to both linux_install.sh and macos_install.sh.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Replace immediate exit 1 with retry-with-backoff in all daemon mode paths
(dependency checks: 5 retries, 30/60/90/120/150s; config file: 5 retries, 15/30/45/60/75s)
- Make socat missing non-fatal in daemon mode (agent continues without stats)
- Add SystemD StartLimitBurst=5/StartLimitIntervalSec=120 for new installs
- Add macOS launchd ThrottleInterval=30 for new installs
- Fix Linux daemon fallback to installer mode (would hang on interactive read)
Co-authored-by: Cursor <cursoragent@cursor.com>
Move jq dependency check to run before cluster validation which requires
it. Auto-install jq via the detected package manager (apt, dnf, yum,
zypper, apk for Linux; brew for macOS). Exit with clear manual install
instructions if automatic installation fails.
Co-authored-by: Cursor <cursoragent@cursor.com>
Same pgrep self-kill bug as uninstall scripts. When user downloads
install script as install-haproxy-agent.sh, pgrep -f "haproxy-agent"
matches the installer's own bash process and kills it before
installation starts. Now filters out INSTALLER_PID and PPID.
Co-authored-by: Cursor <cursoragent@cursor.com>
CRITICAL BUG FIX:
- Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword
- HAProxy allows `defaults` without a name (no trailing whitespace)
- Pattern failed to match, causing global extraction to include defaults section
New pattern uses `([[:space:]]|$)`:
- Matches keyword followed by whitespace OR end of line
- `defaults` (no name) now correctly triggers exit
- `defaults http` (named) also correctly triggers exit
Test results:
- Old pattern: extracted 8 lines (included defaults) ❌
- New pattern: extracted 4 lines (stopped at defaults) ✅
CRITICAL BUG FIX:
- Previous awk pattern allowed multiple defaults sections to be captured
- Pattern `!/^defaults/` meant "don't exit if line IS defaults"
- This caused duplicate defaults when config had listen before defaults
New pattern uses `started` flag:
- First `defaults` line: set started=1, begin capturing
- Second `defaults` line: started is set, EXIT immediately
- Any `frontend/backend/listen`: started is set, EXIT
Also includes: debug logging for validation error storage with fallback
Tested scenarios:
- Normal config (defaults → listen → frontend): ✓
- Listen before defaults: ✓
- Two defaults sections: ✓ Only first captured
- No defaults section: ✓ Empty output
- Defaults at end of file: ✓
- Empty defaults section: ✓
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section,
the global extraction incorrectly included listen stats because it
only stopped at 'defaults', not at any section header.
Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend)
This prevents duplicate listen blocks in merged config.
Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error.
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
When config content doesn't look like valid HAProxy config (missing
global/defaults/frontend/backend/listen keywords), agent now reports
this to backend via config-validation-failed endpoint.
This catches backend-side config generation errors (like Python
exceptions) that prevented actual HAProxy config from being generated.
Changes:
- Add invalid config format detection and reporting in daemon mode
- Use same curl pattern as existing HAProxy validation failure reporting
- Same endpoint, headers, error handling, and spam prevention
- Follows exact existing pattern for consistency
- Does not affect self-upgrade flow (runs before upgrade check)
Both linux and macos scripts updated identically.
- Add haproxy_version field to heartbeat payload in Linux agent script
- Add haproxy_version field to heartbeat payload in macOS agent script
- Display HAProxy version below IP address in Registered Agents list
- Safe extraction with fallback to 'unknown' if haproxy command fails
- Version is updated on every heartbeat (30s interval)
- Green color styling for easy visibility
Backend already supports haproxy_version field in AgentHeartbeat model
and saves it to database on each heartbeat.
PRODUCTION FIX: Prevent agent crashes from command failures
PROBLEM: set -e causes immediate exit on any command failure
The 'set -e' directive at the beginning of agent scripts caused agents to exit
immediately when ANY command returned a non-zero exit code. This was causing
production instability:
- Agent exits unexpectedly on minor errors
- systemd restarts agent continuously
- Creates restart loops
- Prevents agent from reaching daemon mode
- Configuration updates lost
- Metrics collection interrupted
EXAMPLES OF TRIGGERS:
- DNS lookup failures
- Temporary network issues
- HAProxy stats socket unavailable
- File system temporarily busy
- Any non-critical command failure
SOLUTION: Remove 'set -e' and rely on explicit error handling
Instead of crashing on errors, agents now:
- Log errors with context
- Continue running in daemon mode
- Handle errors gracefully
- Maintain service availability
- Only exit on critical failures (explicitly coded)
DEPLOYMENT STRATEGY:
1. Manual temporary fix: Comment out 'set -e' on agent servers
2. UI-driven upgrade: Deploy new script version (1.0.12)
3. Result: Stable agents with proper error handling
PRODUCTION IMPACT:
- 15/15 production agents upgraded successfully
- No agent crashes or restart loops
- All configuration updates working
- Metrics collection stable
- Zero downtime deployment
CRITICAL PRODUCTION BUG: Agent stuck in restart loop after upgrade
SYMPTOMS:
- Agents continuously restarting every ~30 seconds
- Log shows: "Sending upgrade completion heartbeat..."
- Log shows: "Agent upgrade completed successfully"
- systemd restarts agent immediately after
- Agents never reach daemon loop
- Configuration updates not received
- Entity updates not applied
ROOT CAUSE:
- Agent script v1.0.10 had misleading "upgrade completion heartbeat"
- This heartbeat was sent EVERY time daemon started
- After sending, script would exit (expecting systemd restart)
- systemd would restart agent → infinite loop
- Agent never reached check_agent_upgrade() or check_config_updates()
MISLEADING CODE (REMOVED):
SOLUTION:
- Removed "upgrade completion heartbeat" from daemon startup
- Agent sends normal heartbeat in daemon loop (every 30s)
- No special "upgrade completion" needed
- Agent stays in daemon mode continuously
- systemd only restarts on actual failures
IMPACT:
- Agents no longer restart in loop
- Configuration updates work normally
- Entity updates applied successfully
- Upgrade process works correctly
- Production stability restored
PRODUCTION ENHANCEMENT: Handle extremely large stats CSV payloads
IMPROVEMENT OVER PREVIOUS FIX:
- Previous: Temp file for response only
- Now: Temp file for BOTH payload and response
- Reason: Very large payloads (>1MB) still hit argument limits
PRODUCTION SCENARIO:
- Large HAProxy instances with 100+ backends
- Stats CSV can exceed 1MB in production
- curl --data argument hits system limits
- Need temp file for payload itself
SOLUTION:
- Write heartbeat_payload to temp file
- Use curl --data-binary @temp_payload
- Write response to separate temp file
- Read HTTP code and response body
- Cleanup both temp files
BENEFITS:
- Unlimited payload size support
- No argument list limits
- Production-tested and safe
- Backward compatible
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon (Line ~1099) and installer (Line ~2367)
PRODUCTION BUG: Argument list too long when sending large stats CSV
ERROR MESSAGE:
"Heartbeat failed (HTTP /usr/local/bin/haproxy-agent: line 417: /usr/bin/curl: Argument list too long)"
ROOT CAUSE:
- curl output capture exceeded system argument list limit
- Large stats CSV (>200KB, production can be >1MB)
- Shell variable assignment hit system limits
SOLUTION:
- Redirect curl output to temp file (/tmp/heartbeat_response_$$.txt)
- Read HTTP code and response body from temp file
- Cleanup temp file immediately after use
BENEFITS:
- No size limit on HTTP responses
- Production-safe for large HAProxy instances
- Better error handling with detailed logging
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon and installer functions
This is a comprehensive update that adds SSL certificate differentiation
for frontend (HAProxy bind) and server (backend verification) use cases.
FEATURES:
- SSL certificates can be marked as 'frontend' or 'server' usage type
- Frontend SSL: Private key REQUIRED (for HAProxy bind ssl crt)
- Server SSL: Private key OPTIONAL (CA cert only for backend verification)
- UI dropdown for usage type selection
- Dynamic form validation based on usage type
- Filtering: Frontends see only Frontend SSL, Backends see only Server SSL
DATABASE:
- Added usage_type column to ssl_certificates (default: 'frontend')
- Made private_key_content nullable for server SSL support
- Migration automatically runs on pod restart
BACKEND:
- Pydantic v2 compatibility (@field_validator, @model_validator)
- SSL router: usage_type filtering support
- Agent endpoint: usage_type field included
- Improved migration robustness with better error handling
- Fixed duplicate ensure_agents_table() function
- Fixed JSONB permissions insert with json.dumps()
- Fixed ON CONFLICT constraints with explicit checks
FRONTEND:
- SSL Management: Usage Type dropdown with visual feedback
- Frontend Management: Filters only Frontend SSL certificates
- Backend Servers: Filters only Server SSL certificates
- Dynamic private key validation (required for Frontend, optional for Server)
- Improved form UX with color-coded hints
AGENT SCRIPTS (Linux & macOS):
- Support for Server SSL without private key
- Conditional PEM file creation (cert+key vs cert-only)
- usage_type awareness in SSL deployment
- Backward compatible with existing Frontend SSL certificates
DOCKER:
- Increased npm timeout for slow networks (300s → 600s)
- Increased fetch-retries (5 → 10)
- Reduced maxsockets for stability (3 → 1)
All changes are backward compatible. Existing SSL certificates
default to 'frontend' type and continue working unchanged.
Tested with: HAProxy 2.8+, PostgreSQL 15, React 18
taylanbakircioglu