- Fix DB: COALESCE(NULLIF) prevented clearing stale MASTER/BACKUP state.
Use CASE WHEN IS NOT NULL to distinguish absent fields (old agents)
from empty fields (keepalived stopped) and properly clear to NULL.
- Fix Redis: explicitly delete cache key when keepalived stops instead
of relying on 90s TTL expiry, preventing stale dashboard data.
- Fix agent scripts: SKIP_TO_DAEMON send_heartbeat now always sends
keepalive_state/keepalive_ip fields (even empty) so backend can
detect stopped keepalived and clear stale data.
- Fix legacy heartbeat endpoint: POST /{agent_id}/heartbeat now also
updates keepalive_state and keepalive_ip fields.
- UI: add horizontal scroll to ClusterManagement table to prevent
column overflow with new Keepalive column.
- UI: add VIP tooltip to Dashboard AgentStatusCard keepalive tag.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add keepalive_state and keepalive_ip columns to agents table (migration + schema)
- Add keepalive fields to AgentHeartbeat Pydantic model (backward compatible)
- Update heartbeat endpoint to persist keepalive data to DB and cache in Redis
- Add multi-method keepalived detection in agent scripts (journalctl, log files, VIP check)
- Update dashboard-stats agents/status API with Redis-first keepalive lookup
- Update GET /api/agents to include keepalive_state and keepalive_ip
- Show MASTER/BACKUP tag in Dashboard AgentStatusCard
- Show keepalive info in Agent Management registered agents table
- Add Keepalive column to Cluster Management table with VIP search support
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add per-cluster SSL sync status tracking with APPLYING/SYNCED/PARTIAL states
- Add timeout detection (10min) for stalled SSL deployments showing PARTIAL status
- Add SSL-specific info in Apply Management dialog (cluster count, auto-apply notice)
- Add Deployment Status tab in SSL details showing per-cluster agent sync progress
- Implement SSL auto-reject (reject propagates to all clusters) and auto-undo
- Add conditional entity rollback for SSL reject (time-window based safety check)
- Filter REJECTED status from SSL last_config_status display
- Add pending_cluster_names to SSL list API response
- Optimize agent SSL reload: only trigger HAProxy reload when changed cert is
actually referenced in the cluster's HAProxy config (grep -qF check), preventing
unnecessary reloads on clusters that don't use the updated certificate
- Applied to all 4 SSL deploy paths: Linux/macOS standalone and daemon embedded modes
Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause: After agent self-upgrade, the embedded daemon (SKIP_TO_DAEMON
block) runs instead of run_daemon(). HAPROXY_BIN and HAPROXY_CONFIG
variables were uninitialized before the daemon loop, causing SSL-triggered
HAProxy reloads to silently fail with empty path validation.
Changes:
- Initialize HAPROXY_BIN/HAPROXY_CONFIG before embedded daemon loop
- Add md5 checksum comparison in deploy_ssl_certificates() to detect
actual cert file changes (avoid unnecessary writes and reloads)
- Add check_ssl_updates() for independent SSL sync every ~2.5 min
in run_daemon(), independent of config version changes
- Add SSL-aware reload in check_config_updates(): if config validation
fails but SSL certs changed, reload HAProxy with existing config
- Add "full" fetch mode to fetch_and_deploy_ssl_certificates() to
bypass incremental timestamp filter for standalone SSL checks
Applied to both linux_install.sh and macos_install.sh.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Replace immediate exit 1 with retry-with-backoff in all daemon mode paths
(dependency checks: 5 retries, 30/60/90/120/150s; config file: 5 retries, 15/30/45/60/75s)
- Make socat missing non-fatal in daemon mode (agent continues without stats)
- Add SystemD StartLimitBurst=5/StartLimitIntervalSec=120 for new installs
- Add macOS launchd ThrottleInterval=30 for new installs
- Fix Linux daemon fallback to installer mode (would hang on interactive read)
Co-authored-by: Cursor <cursoragent@cursor.com>
Move jq dependency check to run before cluster validation which requires
it. Auto-install jq via the detected package manager (apt, dnf, yum,
zypper, apk for Linux; brew for macOS). Exit with clear manual install
instructions if automatic installation fails.
Co-authored-by: Cursor <cursoragent@cursor.com>
Same pgrep self-kill bug as uninstall scripts. When user downloads
install script as install-haproxy-agent.sh, pgrep -f "haproxy-agent"
matches the installer's own bash process and kills it before
installation starts. Now filters out INSTALLER_PID and PPID.
Co-authored-by: Cursor <cursoragent@cursor.com>
Instead of a hardcoded PROTECTED_PATHS list (varies per environment):
- safe_rm() blocks any path not containing "haproxy-agent"
- Process killer verifies ps output contains "haproxy-agent"
- Pre/post HAProxy integrity check via md5 hash comparison of config
and service status diff (was running -> still running?)
Co-authored-by: Cursor <cursoragent@cursor.com>
pgrep -f "haproxy-agent" was matching the uninstall script's own
process (uninstall-haproxy-agent-linux.sh contains "haproxy-agent"),
causing the script to terminate itself at step 1/7 before reaching
file cleanup. Now filters out $$ (own PID) and $PPID from kill lists.
Co-authored-by: Cursor <cursoragent@cursor.com>
CRITICAL BUG FIX:
- Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword
- HAProxy allows `defaults` without a name (no trailing whitespace)
- Pattern failed to match, causing global extraction to include defaults section
New pattern uses `([[:space:]]|$)`:
- Matches keyword followed by whitespace OR end of line
- `defaults` (no name) now correctly triggers exit
- `defaults http` (named) also correctly triggers exit
Test results:
- Old pattern: extracted 8 lines (included defaults) ❌
- New pattern: extracted 4 lines (stopped at defaults) ✅
CRITICAL BUG FIX:
- Previous awk pattern allowed multiple defaults sections to be captured
- Pattern `!/^defaults/` meant "don't exit if line IS defaults"
- This caused duplicate defaults when config had listen before defaults
New pattern uses `started` flag:
- First `defaults` line: set started=1, begin capturing
- Second `defaults` line: started is set, EXIT immediately
- Any `frontend/backend/listen`: started is set, EXIT
Also includes: debug logging for validation error storage with fallback
Tested scenarios:
- Normal config (defaults → listen → frontend): ✓
- Listen before defaults: ✓
- Two defaults sections: ✓ Only first captured
- No defaults section: ✓ Empty output
- Defaults at end of file: ✓
- Empty defaults section: ✓
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section,
the global extraction incorrectly included listen stats because it
only stopped at 'defaults', not at any section header.
Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend)
This prevents duplicate listen blocks in merged config.
Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error.
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
When config content doesn't look like valid HAProxy config (missing
global/defaults/frontend/backend/listen keywords), agent now reports
this to backend via config-validation-failed endpoint.
This catches backend-side config generation errors (like Python
exceptions) that prevented actual HAProxy config from being generated.
Changes:
- Add invalid config format detection and reporting in daemon mode
- Use same curl pattern as existing HAProxy validation failure reporting
- Same endpoint, headers, error handling, and spam prevention
- Follows exact existing pattern for consistency
- Does not affect self-upgrade flow (runs before upgrade check)
Both linux and macos scripts updated identically.
- Copy uninstall-agent-linux.sh and uninstall-agent-macos.sh to
backend/utils/agent_scripts/ (same location as install scripts)
- Update generate-uninstall-script endpoint to use the same path
pattern as working install script endpoint
- Add container-specific fallback paths for robustness
- Add debug logging to track which path is used
Fixes 404 error when fetching uninstall script in production
where /utils/ directory at project root is not deployed.
- Add haproxy_version field to heartbeat payload in Linux agent script
- Add haproxy_version field to heartbeat payload in macOS agent script
- Display HAProxy version below IP address in Registered Agents list
- Safe extraction with fallback to 'unknown' if haproxy command fails
- Version is updated on every heartbeat (30s interval)
- Green color styling for easy visibility
Backend already supports haproxy_version field in AgentHeartbeat model
and saves it to database on each heartbeat.
PRODUCTION FIX: Prevent agent crashes from command failures
PROBLEM: set -e causes immediate exit on any command failure
The 'set -e' directive at the beginning of agent scripts caused agents to exit
immediately when ANY command returned a non-zero exit code. This was causing
production instability:
- Agent exits unexpectedly on minor errors
- systemd restarts agent continuously
- Creates restart loops
- Prevents agent from reaching daemon mode
- Configuration updates lost
- Metrics collection interrupted
EXAMPLES OF TRIGGERS:
- DNS lookup failures
- Temporary network issues
- HAProxy stats socket unavailable
- File system temporarily busy
- Any non-critical command failure
SOLUTION: Remove 'set -e' and rely on explicit error handling
Instead of crashing on errors, agents now:
- Log errors with context
- Continue running in daemon mode
- Handle errors gracefully
- Maintain service availability
- Only exit on critical failures (explicitly coded)
DEPLOYMENT STRATEGY:
1. Manual temporary fix: Comment out 'set -e' on agent servers
2. UI-driven upgrade: Deploy new script version (1.0.12)
3. Result: Stable agents with proper error handling
PRODUCTION IMPACT:
- 15/15 production agents upgraded successfully
- No agent crashes or restart loops
- All configuration updates working
- Metrics collection stable
- Zero downtime deployment
CRITICAL PRODUCTION BUG: Agent stuck in restart loop after upgrade
SYMPTOMS:
- Agents continuously restarting every ~30 seconds
- Log shows: "Sending upgrade completion heartbeat..."
- Log shows: "Agent upgrade completed successfully"
- systemd restarts agent immediately after
- Agents never reach daemon loop
- Configuration updates not received
- Entity updates not applied
ROOT CAUSE:
- Agent script v1.0.10 had misleading "upgrade completion heartbeat"
- This heartbeat was sent EVERY time daemon started
- After sending, script would exit (expecting systemd restart)
- systemd would restart agent → infinite loop
- Agent never reached check_agent_upgrade() or check_config_updates()
MISLEADING CODE (REMOVED):
SOLUTION:
- Removed "upgrade completion heartbeat" from daemon startup
- Agent sends normal heartbeat in daemon loop (every 30s)
- No special "upgrade completion" needed
- Agent stays in daemon mode continuously
- systemd only restarts on actual failures
IMPACT:
- Agents no longer restart in loop
- Configuration updates work normally
- Entity updates applied successfully
- Upgrade process works correctly
- Production stability restored
PRODUCTION ENHANCEMENT: Handle extremely large stats CSV payloads
IMPROVEMENT OVER PREVIOUS FIX:
- Previous: Temp file for response only
- Now: Temp file for BOTH payload and response
- Reason: Very large payloads (>1MB) still hit argument limits
PRODUCTION SCENARIO:
- Large HAProxy instances with 100+ backends
- Stats CSV can exceed 1MB in production
- curl --data argument hits system limits
- Need temp file for payload itself
SOLUTION:
- Write heartbeat_payload to temp file
- Use curl --data-binary @temp_payload
- Write response to separate temp file
- Read HTTP code and response body
- Cleanup both temp files
BENEFITS:
- Unlimited payload size support
- No argument list limits
- Production-tested and safe
- Backward compatible
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon (Line ~1099) and installer (Line ~2367)
PRODUCTION BUG: Argument list too long when sending large stats CSV
ERROR MESSAGE:
"Heartbeat failed (HTTP /usr/local/bin/haproxy-agent: line 417: /usr/bin/curl: Argument list too long)"
ROOT CAUSE:
- curl output capture exceeded system argument list limit
- Large stats CSV (>200KB, production can be >1MB)
- Shell variable assignment hit system limits
SOLUTION:
- Redirect curl output to temp file (/tmp/heartbeat_response_$$.txt)
- Read HTTP code and response body from temp file
- Cleanup temp file immediately after use
BENEFITS:
- No size limit on HTTP responses
- Production-safe for large HAProxy instances
- Better error handling with detailed logging
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon and installer functions
This is a comprehensive update that adds SSL certificate differentiation
for frontend (HAProxy bind) and server (backend verification) use cases.
FEATURES:
- SSL certificates can be marked as 'frontend' or 'server' usage type
- Frontend SSL: Private key REQUIRED (for HAProxy bind ssl crt)
- Server SSL: Private key OPTIONAL (CA cert only for backend verification)
- UI dropdown for usage type selection
- Dynamic form validation based on usage type
- Filtering: Frontends see only Frontend SSL, Backends see only Server SSL
DATABASE:
- Added usage_type column to ssl_certificates (default: 'frontend')
- Made private_key_content nullable for server SSL support
- Migration automatically runs on pod restart
BACKEND:
- Pydantic v2 compatibility (@field_validator, @model_validator)
- SSL router: usage_type filtering support
- Agent endpoint: usage_type field included
- Improved migration robustness with better error handling
- Fixed duplicate ensure_agents_table() function
- Fixed JSONB permissions insert with json.dumps()
- Fixed ON CONFLICT constraints with explicit checks
FRONTEND:
- SSL Management: Usage Type dropdown with visual feedback
- Frontend Management: Filters only Frontend SSL certificates
- Backend Servers: Filters only Server SSL certificates
- Dynamic private key validation (required for Frontend, optional for Server)
- Improved form UX with color-coded hints
AGENT SCRIPTS (Linux & macOS):
- Support for Server SSL without private key
- Conditional PEM file creation (cert+key vs cert-only)
- usage_type awareness in SSL deployment
- Backward compatible with existing Frontend SSL certificates
DOCKER:
- Increased npm timeout for slow networks (300s → 600s)
- Increased fetch-retries (5 → 10)
- Reduced maxsockets for stability (3 → 1)
All changes are backward compatible. Existing SSL certificates
default to 'frontend' type and continue working unchanged.
Tested with: HAProxy 2.8+, PostgreSQL 15, React 18
taylanbakircioglu