pgrep -f "haproxy-agent" was matching the uninstall script's own
process (uninstall-haproxy-agent-linux.sh contains "haproxy-agent"),
causing the script to terminate itself at step 1/7 before reaching
file cleanup. Now filters out $$ (own PID) and $PPID from kill lists.
Co-authored-by: Cursor <cursoragent@cursor.com>
Disabled agents were blocking apply sync progress indefinitely because
they were counted in total_agents but could never report as synced.
Backend:
- agent-sync endpoint: disabled agents excluded from total/synced/unsynced
counts, added disabled_agents and total_agents_including_disabled fields
- SSL cert agent-sync endpoint: same disabled agent exclusion
- Both endpoints still return disabled agents in the list with
sync_excluded: true for UI display
Frontend:
- ApplyManagement: cluster sync complete logic handles 0 enabled agents,
all updateEntityCounts calls pass disabled count, agent table shows
OFF/Excluded tags for disabled agents
- GlobalProgress: shows "(X off)" indicator, visible even when all
agents are disabled
- ProgressContext: agentCounts state supports disabled field
- agentSync utility: verifyRealAgentSync treats null/0-agent sync_status
as synced (nothing to wait for)
Co-authored-by: Cursor <cursoragent@cursor.com>
Fixed TypeError "t.children.toLowerCase is not a function" when searching
in Default Backend dropdown after editing a frontend.
Root cause: option.children was a React element array (multiple JSX parts),
not a string, so toLowerCase() failed.
Solution: Added label prop to Option and use optionLabelProp="label" pattern
(consistent with SSL Certificates Select in same file).
Changes:
- Added optionLabelProp="label" to Select component
- Added label={backendLabel} to each Option
- Changed filterOption to use option.label instead of option.children
- Users can now search by backend name, server count, or "No servers"
CRITICAL BUG FIX:
- Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword
- HAProxy allows `defaults` without a name (no trailing whitespace)
- Pattern failed to match, causing global extraction to include defaults section
New pattern uses `([[:space:]]|$)`:
- Matches keyword followed by whitespace OR end of line
- `defaults` (no name) now correctly triggers exit
- `defaults http` (named) also correctly triggers exit
Test results:
- Old pattern: extracted 8 lines (included defaults) ❌
- New pattern: extracted 4 lines (stopped at defaults) ✅
CRITICAL BUG FIX:
- Previous awk pattern allowed multiple defaults sections to be captured
- Pattern `!/^defaults/` meant "don't exit if line IS defaults"
- This caused duplicate defaults when config had listen before defaults
New pattern uses `started` flag:
- First `defaults` line: set started=1, begin capturing
- Second `defaults` line: started is set, EXIT immediately
- Any `frontend/backend/listen`: started is set, EXIT
Also includes: debug logging for validation error storage with fallback
Tested scenarios:
- Normal config (defaults → listen → frontend): ✓
- Listen before defaults: ✓
- Two defaults sections: ✓ Only first captured
- No defaults section: ✓ Empty output
- Defaults at end of file: ✓
- Empty defaults section: ✓
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section,
the global extraction incorrectly included listen stats because it
only stopped at 'defaults', not at any section header.
Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend)
This prevents duplicate listen blocks in merged config.
Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error.
SECURITY FIX (agent.py):
- Ensure ONLY APPLIED versions are sent to agents
- Fixed fallback query that could return PENDING versions
- Agents will never receive unapproved configurations
UX FIX (cluster.py):
- Clear validation_error when creating new version via Apply
- Prevents stale validation errors from showing after re-apply
UX FIX (agent.py):
- Clear validation_error when agent successfully applies config
- Clear last_validation_error on agent record after success
Query agent preserved_listen_blocks dynamically and skip
conflicting entities to prevent HAProxy validation errors.
Falls back to static reserved list if no agent data available.
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
- Updated SUGGESTION_TEMPLATES in haproxy_error_parser.py
- Fixed fallback error messages in cluster.py
- Product language should be English throughout
Bulk Import Fix (config.py):
- Check for existing servers (including soft-deleted) before INSERT
- If server exists: UPDATE and reactivate instead of INSERT
- Create snapshot for ALL updated servers (enables reject/rollback)
- Add backend_name to UPDATE for field consistency
- Track updated/reactivated servers in response
- Prevents duplicate key constraint violation on re-import
Frontend Deletion Fix (frontend.py):
- Add hard delete support for already soft-deleted frontends
- Follows same pattern as backend.py hard delete
- Deletes WAF associations and config versions before frontend
- Works with existing orphan cleanup in Apply/Reject flows
Both fixes are safe because:
- Orphan version cleanup handles hard-deleted entities in Apply/Reject
- Consistent with existing backend.py patterns
- Snapshot-based rollback fully supported
When config content doesn't look like valid HAProxy config (missing
global/defaults/frontend/backend/listen keywords), agent now reports
this to backend via config-validation-failed endpoint.
This catches backend-side config generation errors (like Python
exceptions) that prevented actual HAProxy config from being generated.
Changes:
- Add invalid config format detection and reporting in daemon mode
- Use same curl pattern as existing HAProxy validation failure reporting
- Same endpoint, headers, error handling, and spam prevention
- Follows exact existing pattern for consistency
- Does not affect self-upgrade flow (runs before upgrade check)
Both linux and macos scripts updated identically.
Remove redundant local 'import json' statements that caused:
"cannot access local variable 'json' where it is not associated with a value"
Root cause: Local import inside try block made 'json' a local variable,
but except clause referenced json.JSONDecodeError before assignment.
Changes:
- Line 197: Remove local import, use global json (line 5)
- Line 514: Remove 'import json as _json', use global json
- Use ValueError instead of json.JSONDecodeError (equivalent, JSONDecodeError
is a subclass of ValueError)
This was causing config generation to fail completely, returning error
message as config content instead of actual HAProxy configuration.
- Copy uninstall-agent-linux.sh and uninstall-agent-macos.sh to
backend/utils/agent_scripts/ (same location as install scripts)
- Update generate-uninstall-script endpoint to use the same path
pattern as working install script endpoint
- Add container-specific fallback paths for robustness
- Add debug logging to track which path is used
Fixes 404 error when fetching uninstall script in production
where /utils/ directory at project root is not deployed.
- Add haproxy_error_parser.py: Parses HAProxy validation errors with
confidence scoring, extracts entity type/name, line number, error type
- Add ValidationErrorModal.js: Rich modal with parsed error summary,
quick fix suggestions, and manual troubleshooting guide
- Update cluster.py: Integrate error parser into agent-sync and
config-versions endpoints with graceful fallback
- Update ApplyManagement.js: Add validation error banner with quick
navigation buttons and error detail modal
- Update FrontendManagement.js & BackendServers.js: Handle URL params
for deep-linking to entity edit forms with field highlighting
Enables users to see actionable validation failure details directly
in the UI without needing server access for debugging.
- Add new API endpoint to serve uninstall scripts by platform
- Display uninstall script alongside install script in setup wizard
- Add dedicated delete agent modal with 2-step workflow
- Modern UI with gradient banners, platform icons, and info cards
- Enhanced uninstall scripts to clean all agent temp/backup files
- HAProxy service and config remain untouched during uninstall
- Add 'random' and 'first' options to backend balance method selector
- Add balance method validation in config parser with warning for unknown methods
- Update API documentation with all supported balance algorithms
- Fix token-agent relationship not updating when agent config changes
- Agent's api_key now syncs with DB on heartbeat when using different token
- Change Security page badge color from red to blue for better UX
- Add HAProxy Version column to cluster list table
- Show version number with agent names directly (no hover required)
- Fetch agents for each cluster to get haproxy_version info
- Show warning icon (yellow) when agents have different HAProxy versions
- Green color for consistent versions, yellow for mismatched versions
- Agent names displayed below each version in smaller gray text
- UI-only change, no backend modifications
- Safe implementation using existing /api/agents endpoint
When ssl_enabled is true for a backend server but:
- No ssl_verify option is explicitly set AND
- No CA file (ssl_certificate_id) is specified
HAProxy 2.8+ defaults to 'verify required' which fails without a CA.
Now auto-adding 'verify none' in this case with a warning log.
Users can override this in UI by setting SSL Verification explicitly.
This fix is safe for:
- Bulk Import: Parser already sets ssl_verify='none' when removing ca-file
- Version Diff: Generated config correctly shows 'verify none'
- Restore: DB values unchanged, config regeneration applies same fix
Fixes: 'verify is enabled by default but no CA file specified' error
- Add haproxy_version field to heartbeat payload in Linux agent script
- Add haproxy_version field to heartbeat payload in macOS agent script
- Display HAProxy version below IP address in Registered Agents list
- Safe extraction with fallback to 'unknown' if haproxy command fails
- Version is updated on every heartbeat (30s interval)
- Green color styling for easy visibility
Backend already supports haproxy_version field in AgentHeartbeat model
and saves it to database on each heartbeat.
CRITICAL DATA LOSS BUG FIX
Problem:
- User updates existing entity via bulk import
- User clicks Reject
- EXPECTED: Entity reverts to old values
- ACTUAL: Entity is COMPLETELY DELETED!
Root Cause:
Reject logic tracked ALL bulk import entities for force deletion.
Did not distinguish between CREATE and UPDATE operations.
Fix:
Added operation check - only track CREATE operations for force deletion.
UPDATE operations are rolled back, not deleted.
Impact:
- CREATE scenarios: No change in behavior
- UPDATE scenarios: Data loss prevented
- Mixed CREATE+UPDATE: Only CREATE entities deleted
- Rollback failures: Safety mechanism preserved
Files Changed:
- backend/routers/cluster.py: Line 4494 added operation check
CRITICAL BUG FIX: Backend server SSL advanced options not in generated HAProxy config
Problem:
- Database has ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers columns ✅
- Config generation code tries to write them to HAProxy config (line 680-687) ✅
- BUT SELECT statement did NOT include these fields ❌
- Result: server.get('ssl_sni') always returned None
Impact:
- User sets 'TLS Min Version: TLSv1.2' for backend server in UI
- Value saved to database correctly
- BUT generated HAProxy config missing 'ssl-min-ver TLSv1.2'
- Agent deploys incomplete config, SSL settings lost!
Solution:
- Added 4 server SSL advanced options to SELECT query (line 616):
- ssl_sni (for SNI hostname)
- ssl_min_ver (minimum TLS version)
- ssl_max_ver (maximum TLS version)
- ssl_ciphers (cipher suite override)
Config Generation Logic:
- Line 680-687: Code already writes these fields to HAProxy config
- Line 616: Now SELECT actually retrieves the values from database
- Example output: 'server backend1 10.0.0.1:443 ssl sni example.com ssl-min-ver TLSv1.2'
Testing:
- After deployment, bulk import a config with backend server SSL options
- Check generated HAProxy config on agent
- Should now see: 'server X ssl ssl-min-ver TLSv1.2 ciphers ...'
Files Changed:
- backend/services/haproxy_config.py: Line 616 SELECT statement
CRITICAL BUG FIX: SSL advanced options were not being returned by GET API
Problem:
- Database has ssl_alpn, ssl_npn, ssl_ciphers, etc. columns ✅
- Response builder tries to access them (Line 341-347) ✅
- BUT SELECT statements did NOT include these fields ❌
- Result: f.get('ssl_alpn') returned None for all frontends
Impact:
- Frontend Edit modal always showed empty SSL advanced options fields
- User edits would overwrite existing values with NULL
- Data loss on every frontend edit!
Solution:
- Added all 7 SSL advanced options to ALL 6 SELECT queries:
1. cluster_id filter (line 174)
2. cluster_id fallback (line 190)
3. global include_inactive=True (line 219)
4. global include_inactive=False (line 231)
5. global fallback include_inactive=True (line 246)
6. global fallback include_inactive=False (line 258)
Testing:
- Added debug console.log for SSL advanced options (line 551-559)
- After deployment, check browser console for 'SSL ADVANCED OPTIONS DEBUG'
- Should now show: ssl_alpn: 'h2,http/1.1' etc.
Files Changed:
- backend/routers/frontend.py: All 6 SELECT statements
- frontend/src/components/FrontendManagement.js: Debug logging
Critical additions:
1. Change Detection (backend/routers/config.py)
- Added SSL advanced options to bulk import change detection logic
- Detects changes in ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites
- Detects changes in ssl_min_ver, ssl_max_ver, ssl_strict_sni
- Changes will now appear in version diff
2. Pydantic Validators (backend/models/frontend.py, backend/models/backend.py)
- TLS version validator: Only allows valid versions (SSLv3, TLSv1.0-1.3)
- ALPN protocol validator: Only allows h2, http/1.1, http/1.0, h2c, spdy/*
- NPN protocol validator: Only allows http/1.1, http/1.0, spdy/*
- Prevents invalid values from being saved to database
- HAProxy validation will not fail due to invalid SSL options
Impact:
- Bulk import will correctly detect SSL option changes
- Apply Management diff will show SSL changes
- User cannot enter invalid TLS versions or protocols
- Improved UX with early validation errors
Previous fixes in this series:
- Frontend GET API: Added SSL fields to response
- Bulk Import UPDATE: Added SSL fields to UPDATE statement
- Backend Server GET API: Added SSL fields to response
Test: Bulk import with ALPN change → Should see change in version diff
Critical fixes for SSL advanced options (alpn, npn, ciphers, ciphersuites, min-ver, max-ver, strict-sni):
1. Frontend GET API (backend/routers/frontend.py)
- Added all 7 SSL advanced option fields to response
- Frontend Edit modal will now display these fields
- Prevents NULL overwrite when user edits frontend
2. Bulk Import Frontend UPDATE (backend/routers/config.py)
- Added all 7 SSL advanced option fields to UPDATE statement
- Previously skipped with comment 'MVP: DON'T update SSL settings'
- Now bulk import re-runs preserve SSL settings
3. Backend Server GET API (backend/routers/backend.py)
- Added 4 server SSL fields (ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers)
- SQL SELECT query updated
- Response object updated
- Backend Server Edit modal will now display these fields
Root cause: GET APIs were not returning SSL advanced options, causing:
- UI forms to show empty fields
- User edits to overwrite with NULL
- Data loss on subsequent updates
All other endpoints (POST, PUT, INSERT) were already correct.
Impact:
- No more accidental data loss when editing frontends/servers
- Bulk import now preserves SSL advanced options
- UI will correctly display all SSL parameters
Test: Deploy backend, run bulk import, verify ssl_alpn appears in database
- Previously auto-heal only ran when agent sent server_statuses (which is empty on first heartbeat)
- Now auto-heal runs on EVERY heartbeat, ensuring pool_id is fixed immediately
- This fixes agents being stuck with NULL pool_id when haproxy config is empty
- Agents will now appear in UI correctly even before applying first config
- Modified auto-registration when using placeholder token to query pool_id from cluster_id in heartbeat
- Previously pool_id was copied from placeholder agent (which was NULL)
- Now correctly fetches pool_id from haproxy_clusters table using cluster_id
- Added extensive debug logging for troubleshooting
- Fixes issue where agents were registered with NULL pool_id and not appearing in UI
PROBLEM:
- Agents registered with pool_id = NULL
- UI doesn't show agents without pool assignment
- Auto-healing only works when server_statuses present
- New agents with empty HAProxy config never get pool_id
ROOT CAUSE:
- Agent auto-register INSERT excluded pool_id column
- Auto-healing (line 1417-1422) only triggers inside 'if server_statuses:' block
- Fresh agents send no server_statuses → auto-healing never runs
- Agent sits with pool_id = NULL indefinitely
ANALYSIS:
- Previous commit 1f9d8c2 added auto-healing for edge case (cluster before pool)
- But auto-healing has conditional prerequisite: server_statuses must exist
- New agents: empty HAProxy config → no server_statuses → no healing
- This was working before because agents had config from start
SOLUTION:
- Extract pool_id from cluster_id during registration
- Query haproxy_clusters for pool_id using heartbeat's cluster_id
- Insert agent with pool_id immediately (no wait for auto-heal)
- Two auto-register paths fixed: with API key + without API key
EDGE CASES HANDLED:
1. cluster_id missing → pool_id = NULL → auto-healing fallback ✅
2. cluster deleted → pool_id = NULL → auto-healing fallback ✅
3. cluster pool_id NULL → pool_id = NULL → auto-healing works ✅
4. Old agent script → no cluster_id → auto-healing works ✅
COMPATIBILITY:
- Auto-healing preserved (line 1417-1422 untouched)
- Two mechanisms work together harmoniously
- Backward compatible with existing agents
- No breaking changes
IMPACT:
- New agents visible in UI immediately
- Faster registration (no wait for auto-heal)
- Reduced heartbeat cycles to full functionality
- Better user experience
TESTING REQUIRED:
1. Delete agent from database
2. Install fresh agent with cluster_id in config
3. Verify pool_id populated on registration
4. Verify agent appears in UI immediately
5. Verify existing agents unaffected
PROBLEM:
- Config generation was producing invalid syntax: use_backend ["..."]
- HAProxy validation failing on agents
- Old code had insufficient type checking for JSONB fields
ROOT CAUSE:
- Missing ELSE branch when use_backend_rules wasn't a list
- No handling for unexpected types (tuple, Record, etc.)
- No debug logging to track type issues
FIX:
- Added comprehensive type checking (str, list, tuple, other)
- Added debug logging to track type and value
- Added fallback parsing for string representation of lists
- Enhanced validation to skip invalid entries
- Prevents generation of invalid HAProxy syntax
IMPACT:
- Fixes validation failure for cluster 7 (demo-cluster)
- Enables proper parsing of JSONB use_backend_rules
- Backward compatible with legacy string format
TESTED:
- Database validation: use_backend_rules is proper JSONB array
- String elements correctly extracted and written
- Invalid types caught and logged
PRODUCTION STABILITY FIX - Detailed Logging for Malformed Agent Payloads
PROBLEM:
- demo-agent1 and demo-agent2 sending malformed JSON
- Error: JSON decode error at body position 322
- No visibility into WHAT is malformed or WHY
- Impossible to debug without raw payload inspection
ROOT CAUSE:
- FastAPI consumes request body before error handler
- Pydantic validation fails but does not show raw input
- Agent script might be generating invalid JSON
- No logging of actual problematic payload
SOLUTION - ENHANCED ERROR HANDLING:
1. MIDDLEWARE ENHANCEMENT (error_handler.py):
- Extract RAW body in validation error handler
- Parse agent name from JSON (even if malformed)
- Log first 500 chars of problematic payload
- Add body size to error details
- Special handling for /heartbeat endpoint
- Detailed logging for json_invalid errors
2. HEARTBEAT ENDPOINT (agent.py):
- Added Request parameter for raw body access
- Enhanced docstring with troubleshooting info
BENEFITS:
- Instant visibility into malformed JSON
- Agent name logged even on parse failure
- Exact payload position + preview
- No performance impact (only on errors)
- Backward compatible (does not change API)
NEXT STEPS (After Deploy):
1. Check logs for CRITICAL JSON PARSE ERROR
2. Identify exact field causing parse failure
3. Fix agent script if needed
4. Or fix backend to be more tolerant
This enables root cause analysis without SSH access to agent servers
🔴 PRODUCTION CRITICAL FIX - Agent HTTP 422 Validation Error
PROBLEM:
- Production agents sending heartbeat with flat system_info fields
- Backend Pydantic model was strict and rejecting unknown fields
- Agents going offline with 'Validation error in request data' (HTTP 422)
ROOT CAUSE:
- Legacy agents embed system_info as flat key-value pairs in heartbeat JSON
- Backend expected only defined fields, rejected extra fields
- No backward compatibility for agent format variations
SOLUTION - BACKEND ONLY (NO AGENT CHANGES):
✅ Added 'extra = "allow"' to AgentHeartbeat Pydantic Config
✅ Backend now accepts both formats:
- Flat format: operating_system, kernel_version, etc. (legacy agents)
- Nested format: system_info: {...} (future agents)
✅ Updated comments to clarify backward compatibility
IMPACT:
- ✅ ZERO CHANGES to production agent scripts
- ✅ Existing agents will work immediately after backend deploy
- ✅ Forward compatible with future agent upgrades
- ✅ Tolerant to agent format variations
SAFETY:
- Minimal change (3 lines)
- Pydantic still validates required fields
- Extra fields ignored silently (no breaking changes)
- Production agents continue without restart or upgrade
DEPLOYMENT:
1. Deploy backend (this commit)
2. Agents come online automatically (no action needed)
3. Agent upgrades can happen later (when convenient)
This fix ensures production stability without touching agent scripts.
CRITICAL FIX: All API endpoints now handle SSL advanced options
✅ FRONTEND ENDPOINTS FIXED:
- create_frontend() - INSERT statement now includes all 7 SSL advanced fields
- update_frontend() - UPDATE statement now includes all 7 SSL advanced fields
- Bulk import frontend INSERT - All SSL params included
✅ BACKEND SERVER ENDPOINTS FIXED:
- add_server_to_backend() - INSERT now includes 4 SSL server advanced fields
- update_server() - allowed_fields list now includes SSL params (dynamic update)
- Bulk import server INSERT - All SSL params included
FIXED FIELDS:
Frontend: ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni
Server: ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers
IMPACT:
- Bulk import now correctly saves parsed SSL parameters to database
- Frontend edit/create now accepts SSL params from UI
- Server edit/create now accepts SSL params from UI
- All CRUD operations fully support SSL advanced options
NEXT: Frontend UI components (FrontendManagement.js & BackendServers.js)
CRITICAL FIXES:
1. Agent Heartbeat Validation Error (HTTP 422)
- Added missing 'system_info' field to AgentHeartbeat model
- Agents were sending system_info but backend model didn't accept it
- No agent script update needed - agents already send this field
2. Bulk Import SSL Parser Enhancement
- Fixed parsing of multiple SSL certificates with alpn/npn parameters
- Example: 'bind :443 ssl crt cert1.pem crt cert2.pem crt cert3.pem alpn h2,http/1.1'
- Old parser stopped at first whitespace after crt path
- New parser extracts all crt paths even with alpn/npn/ciphers after them
- Added user-friendly warning for SSL parameters (alpn, npn, ciphers) that won't be imported
TECHNICAL DETAILS:
- backend/models/agent.py: Added system_info: Optional[Dict[str, Any]]
- backend/utils/haproxy_config_parser.py: Enhanced SSL bind parsing logic
* Parse bind line by splitting and iterating through parts
* Extract all crt paths before hitting SSL parameters
* Detect and warn about alpn, npn, ciphers, ciphersuites parameters
* Inform user these advanced options should be configured manually
USER IMPACT:
- Agents will come online after backend deployment (no reinstall needed)
- Bulk import will correctly parse configs with multiple SSL certs + alpn
- Clear warnings shown in UI about SSL parameters not imported
taylanbakircioglu