pgrep -f "haproxy-agent" was matching the uninstall script's own
process (uninstall-haproxy-agent-linux.sh contains "haproxy-agent"),
causing the script to terminate itself at step 1/7 before reaching
file cleanup. Now filters out $$ (own PID) and $PPID from kill lists.
Co-authored-by: Cursor <cursoragent@cursor.com>
CRITICAL BUG FIX:
- Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword
- HAProxy allows `defaults` without a name (no trailing whitespace)
- Pattern failed to match, causing global extraction to include defaults section
New pattern uses `([[:space:]]|$)`:
- Matches keyword followed by whitespace OR end of line
- `defaults` (no name) now correctly triggers exit
- `defaults http` (named) also correctly triggers exit
Test results:
- Old pattern: extracted 8 lines (included defaults) ❌
- New pattern: extracted 4 lines (stopped at defaults) ✅
CRITICAL BUG FIX:
- Previous awk pattern allowed multiple defaults sections to be captured
- Pattern `!/^defaults/` meant "don't exit if line IS defaults"
- This caused duplicate defaults when config had listen before defaults
New pattern uses `started` flag:
- First `defaults` line: set started=1, begin capturing
- Second `defaults` line: started is set, EXIT immediately
- Any `frontend/backend/listen`: started is set, EXIT
Also includes: debug logging for validation error storage with fallback
Tested scenarios:
- Normal config (defaults → listen → frontend): ✓
- Listen before defaults: ✓
- Two defaults sections: ✓ Only first captured
- No defaults section: ✓ Empty output
- Defaults at end of file: ✓
- Empty defaults section: ✓
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section,
the global extraction incorrectly included listen stats because it
only stopped at 'defaults', not at any section header.
Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend)
This prevents duplicate listen blocks in merged config.
Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error.
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
When config content doesn't look like valid HAProxy config (missing
global/defaults/frontend/backend/listen keywords), agent now reports
this to backend via config-validation-failed endpoint.
This catches backend-side config generation errors (like Python
exceptions) that prevented actual HAProxy config from being generated.
Changes:
- Add invalid config format detection and reporting in daemon mode
- Use same curl pattern as existing HAProxy validation failure reporting
- Same endpoint, headers, error handling, and spam prevention
- Follows exact existing pattern for consistency
- Does not affect self-upgrade flow (runs before upgrade check)
Both linux and macos scripts updated identically.
- Copy uninstall-agent-linux.sh and uninstall-agent-macos.sh to
backend/utils/agent_scripts/ (same location as install scripts)
- Update generate-uninstall-script endpoint to use the same path
pattern as working install script endpoint
- Add container-specific fallback paths for robustness
- Add debug logging to track which path is used
Fixes 404 error when fetching uninstall script in production
where /utils/ directory at project root is not deployed.
- Add haproxy_version field to heartbeat payload in Linux agent script
- Add haproxy_version field to heartbeat payload in macOS agent script
- Display HAProxy version below IP address in Registered Agents list
- Safe extraction with fallback to 'unknown' if haproxy command fails
- Version is updated on every heartbeat (30s interval)
- Green color styling for easy visibility
Backend already supports haproxy_version field in AgentHeartbeat model
and saves it to database on each heartbeat.
PRODUCTION FIX: Prevent agent crashes from command failures
PROBLEM: set -e causes immediate exit on any command failure
The 'set -e' directive at the beginning of agent scripts caused agents to exit
immediately when ANY command returned a non-zero exit code. This was causing
production instability:
- Agent exits unexpectedly on minor errors
- systemd restarts agent continuously
- Creates restart loops
- Prevents agent from reaching daemon mode
- Configuration updates lost
- Metrics collection interrupted
EXAMPLES OF TRIGGERS:
- DNS lookup failures
- Temporary network issues
- HAProxy stats socket unavailable
- File system temporarily busy
- Any non-critical command failure
SOLUTION: Remove 'set -e' and rely on explicit error handling
Instead of crashing on errors, agents now:
- Log errors with context
- Continue running in daemon mode
- Handle errors gracefully
- Maintain service availability
- Only exit on critical failures (explicitly coded)
DEPLOYMENT STRATEGY:
1. Manual temporary fix: Comment out 'set -e' on agent servers
2. UI-driven upgrade: Deploy new script version (1.0.12)
3. Result: Stable agents with proper error handling
PRODUCTION IMPACT:
- 15/15 production agents upgraded successfully
- No agent crashes or restart loops
- All configuration updates working
- Metrics collection stable
- Zero downtime deployment
CRITICAL PRODUCTION BUG: Agent stuck in restart loop after upgrade
SYMPTOMS:
- Agents continuously restarting every ~30 seconds
- Log shows: "Sending upgrade completion heartbeat..."
- Log shows: "Agent upgrade completed successfully"
- systemd restarts agent immediately after
- Agents never reach daemon loop
- Configuration updates not received
- Entity updates not applied
ROOT CAUSE:
- Agent script v1.0.10 had misleading "upgrade completion heartbeat"
- This heartbeat was sent EVERY time daemon started
- After sending, script would exit (expecting systemd restart)
- systemd would restart agent → infinite loop
- Agent never reached check_agent_upgrade() or check_config_updates()
MISLEADING CODE (REMOVED):
SOLUTION:
- Removed "upgrade completion heartbeat" from daemon startup
- Agent sends normal heartbeat in daemon loop (every 30s)
- No special "upgrade completion" needed
- Agent stays in daemon mode continuously
- systemd only restarts on actual failures
IMPACT:
- Agents no longer restart in loop
- Configuration updates work normally
- Entity updates applied successfully
- Upgrade process works correctly
- Production stability restored
PRODUCTION ENHANCEMENT: Handle extremely large stats CSV payloads
IMPROVEMENT OVER PREVIOUS FIX:
- Previous: Temp file for response only
- Now: Temp file for BOTH payload and response
- Reason: Very large payloads (>1MB) still hit argument limits
PRODUCTION SCENARIO:
- Large HAProxy instances with 100+ backends
- Stats CSV can exceed 1MB in production
- curl --data argument hits system limits
- Need temp file for payload itself
SOLUTION:
- Write heartbeat_payload to temp file
- Use curl --data-binary @temp_payload
- Write response to separate temp file
- Read HTTP code and response body
- Cleanup both temp files
BENEFITS:
- Unlimited payload size support
- No argument list limits
- Production-tested and safe
- Backward compatible
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon (Line ~1099) and installer (Line ~2367)
PRODUCTION BUG: Argument list too long when sending large stats CSV
ERROR MESSAGE:
"Heartbeat failed (HTTP /usr/local/bin/haproxy-agent: line 417: /usr/bin/curl: Argument list too long)"
ROOT CAUSE:
- curl output capture exceeded system argument list limit
- Large stats CSV (>200KB, production can be >1MB)
- Shell variable assignment hit system limits
SOLUTION:
- Redirect curl output to temp file (/tmp/heartbeat_response_$$.txt)
- Read HTTP code and response body from temp file
- Cleanup temp file immediately after use
BENEFITS:
- No size limit on HTTP responses
- Production-safe for large HAProxy instances
- Better error handling with detailed logging
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon and installer functions
This is a comprehensive update that adds SSL certificate differentiation
for frontend (HAProxy bind) and server (backend verification) use cases.
FEATURES:
- SSL certificates can be marked as 'frontend' or 'server' usage type
- Frontend SSL: Private key REQUIRED (for HAProxy bind ssl crt)
- Server SSL: Private key OPTIONAL (CA cert only for backend verification)
- UI dropdown for usage type selection
- Dynamic form validation based on usage type
- Filtering: Frontends see only Frontend SSL, Backends see only Server SSL
DATABASE:
- Added usage_type column to ssl_certificates (default: 'frontend')
- Made private_key_content nullable for server SSL support
- Migration automatically runs on pod restart
BACKEND:
- Pydantic v2 compatibility (@field_validator, @model_validator)
- SSL router: usage_type filtering support
- Agent endpoint: usage_type field included
- Improved migration robustness with better error handling
- Fixed duplicate ensure_agents_table() function
- Fixed JSONB permissions insert with json.dumps()
- Fixed ON CONFLICT constraints with explicit checks
FRONTEND:
- SSL Management: Usage Type dropdown with visual feedback
- Frontend Management: Filters only Frontend SSL certificates
- Backend Servers: Filters only Server SSL certificates
- Dynamic private key validation (required for Frontend, optional for Server)
- Improved form UX with color-coded hints
AGENT SCRIPTS (Linux & macOS):
- Support for Server SSL without private key
- Conditional PEM file creation (cert+key vs cert-only)
- usage_type awareness in SSL deployment
- Backward compatible with existing Frontend SSL certificates
DOCKER:
- Increased npm timeout for slow networks (300s → 600s)
- Increased fetch-retries (5 → 10)
- Reduced maxsockets for stability (3 → 1)
All changes are backward compatible. Existing SSL certificates
default to 'frontend' type and continue working unchanged.
Tested with: HAProxy 2.8+, PostgreSQL 15, React 18
taylanbakircioglu