Manage highly-available virtual IPs backed by Keepalived (VRRP) directly from the OpenManager
UI — no more SSHing into nodes to install/configure Keepalived by hand. Builds on the agent
pull-architecture: define the VIP centrally, click Apply, and the agents converge.
Highlights:
- New "HA / VIP" tab: create a virtual IP, pick a per-node interface, select which pool nodes
participate (MASTER/BACKUP roles + priorities); live MASTER/BACKUP/FAULT per node.
- On Apply, agents install & configure Keepalived (unicast VRRP, cloud-safe default) across the
major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine) with a
HAProxy health-check, so the VIP fails over automatically when HAProxy drops.
- Single-node (a managed floating IP without failover) and multi-node VRRP failover both work.
- VIP changes ride the standard Apply Management flow with the standard "View Change" diff.
- Approval-gated deletion (safety): deleting a running VIP is staged for approval and the VIP
keeps running, untouched, until you approve it — an agent never tears a VIP down without an
explicit human approval. Per-VIP Diagnostics view; opt-in package uninstall (only on nodes
where OpenManager installed it). A node already running a hand-managed Keepalived is detected
and never overwritten ("externally managed").
- Fully opt-in and backward compatible: nodes/clusters without a VIP are unaffected. Adds
vip_instances + vip_members tables (idempotent SCHEMA_VERSION bump; existing data and
passwords unaffected) and a `vip` RBAC permission group.
- Also includes a HAProxy config-generator robustness fix: auto-inject a stick-table when a
frontend uses a stick counter (track-sc / sc_*_rate) but declares none.
On-prem / L2 (VRRP) scope; the UI notes the cloud caveat.
Assigning a HAProxy agent failed with '401: Authorization header missing'
on GET /api/clusters. A cluster-read hardening had made GET /api/clusters and
GET /api/clusters/{id} accept only a user JWT in the Authorization header;
agents authenticate with their agent token in the X-API-Key header, so the
token was never read.
Both endpoints now accept either a user JWT (Authorization) or an agent token
(X-API-Key via validate_agent_api_key), mirroring the existing dual-auth on
POST /api/agents/generate-install-script. Anonymous access is still rejected,
so the original hardening is preserved. The auth guard is placed before the
try block so the failure surfaces as a clean 401 (not the 500-wrapped-401 in
the report). Agent install scripts now consistently send the token via
X-API-Key (pre-flight cluster check on linux/macos, and macOS get_cluster_paths
which previously used the wrong Authorization: Bearer header).
Also normalizes the platform in the uninstall-script generator so macOS agents
(which report platform 'darwin') no longer get a 400 from
GET /api/agents/generate-uninstall-script/darwin.
version 1.6.0 -> 1.6.2.
Closes#13, Closes#14.
This release squashes the v1.4.0 → v1.5.0 development line. v1.4.0
shipped the ACME stability & enterprise audit (Issues #10/#11/#12).
v1.5.0 builds on that foundation with two co-equal headline features
plus a 22-round audit campaign hardening the prior configuration
surface. License remains MIT for v1.5.0 (relicense to AGPL-3.0
lands in v1.5.2).
------------------------------------------------------------------
HEADLINE FEATURE A — ACME Diagnostic Panel (Issue #13)
------------------------------------------------------------------
A live pre-flight + post-failure diagnostic surface for every ACME
order, reachable from the ACME Automation page. The panel exists
to make ACME failures legible to operators who do NOT have shell
access to the API host.
Endpoints (`backend/routers/acme_diagnostics.py`):
POST /api/letsencrypt/orders/{order_id}/diagnostics
Run the full 5-check suite (DNS / port-80 / routing /
account / agents) and humanize the order's `error_detail`
(>=11 RFC-8555 problem types, backwards compatible with
legacy plain-string failures).
POST /api/letsencrypt/orders/{order_id}/diagnostics/
{check_id}/rerun
Re-run a single check in place — used by the "Re-run"
button on every row of the modal's pre-flight table.
GET /api/letsencrypt/orders/{order_id}/events
Merged event timeline combining the typed
`acme_order_events` rows with correlated
`user_activity_logs` entries (resource_type =
'letsencrypt_order' AND resource_id = order_id). The
diagnostic modal auto-tails this timeline every 5 seconds
while open.
Service-level checks (`backend/services/acme_diagnostics.py`):
* DNS resolution via stdlib socket.gethostbyname_ex through
run_in_executor (intentionally avoiding an aiodns runtime
dep for v1.5.0).
* Port-80 HEAD probe, target locked to the order's domains,
success on HTTP 200 OR 404, warns on egress timeout
(corp egress policies routinely blackhole outbound 80 —
fail-hard would be too noisy).
* SSRF guard: probe refuses non-public IPs and surfaces the
skip in the diagnostic result; IPv4-mapped IPv6 normalisation
closes the `::ffff:169.254.169.254` cloud-metadata vector.
* HAProxy routing presence check: matches the order's
cluster_ids to a port-80 HTTP frontend.
* ACME account validity check against `letsencrypt_accounts`.
* Agent presence check (>=1 active agent in target cluster).
* Every sub-check wrapped in a wall-clock timeout to bound
impact on the API event loop.
RBAC: ssl.read for run, ssl.read for events. Per-user 5/min rate
limit on both run and rerun, backed by the (user_id, action,
created_at DESC) composite index.
Frontend (`frontend/src/components/ACMEAutomation.js`):
* "Diagnose" button on every order row + the existing
"stuck order" warning row.
* Modal with two tabs:
- Pre-flight Checks (Antd Table with status pills + Re-run
buttons + humanized error banner)
- Event Log (Antd Timeline with auto-tail polling, scroll-
to-bottom, pause-on-hover)
* Correlation IDs surfaced in error banners and individual
check fail details for backend-log lookup.
------------------------------------------------------------------
HEADLINE FEATURE B — Site Setup Wizard (Issue #14)
------------------------------------------------------------------
A single guided flow that creates a Backend + Servers + HTTP
Frontend (and optional HTTPS Frontend) in one atomic transaction.
Endpoints (`backend/routers/site_wizard.py`):
POST /api/site-wizard/preview — diff-preview the changeset
POST /api/site-wizard/create — atomic execute
POST /api/site-wizard/reject — clean rollback (including
any wizard_staged ACME
orders)
GET /api/site-wizard/drafts — draft persistence
PUT /api/site-wizard/drafts/{id} — save/update
DELETE /api/site-wizard/drafts/{id}
Feature surface:
* One screen captures both backend (mode + servers) AND
frontend (http + optional https + SSL mode) inputs.
* SSL modes: ACME (new order, HTTP-01 only for v1.5.0),
Upload (existing PEM), Existing (link to a stored cert),
or None.
* ACME-staged path: wizard_staged_until watermark on the
`letsencrypt_orders` row defers finalisation until agent
confirmation; per-mode reject cleanly cancels and rolls
back the staged order.
* Live diff preview against the cluster's current generated
config (renderer-evolution noise stripped — track-sc<N>
dedup, per-server cookie strip, defaults-cookie
inheritance, listen-block flattening).
* Draft persistence with PEM stripped at save time (private
keys never round-trip through the drafts table).
* Per-cluster multi-tenancy: drafts and wizard_staged orders
are isolated to the creating user's cluster scope.
Frontend (`frontend/src/components/SiteWizard.js`):
* 4-step Antd Steps flow: Backend → Frontend → SSL → Review.
* Render the live diff preview inline before commit.
* Antd Form-level validation mirrors backend Pydantic
validators (numeric bounds, HAProxy reserved keywords, ALPN
consistency, IPv6 scope-id, domain regex, server name
dedup).
------------------------------------------------------------------
AUDIT CAMPAIGN — Rounds 1 → 22 (Bulgu #1 → #82)
------------------------------------------------------------------
v1.5.0 includes 22 adversarial review passes. Each round produced
its own commit set in the corporate development line; this squash
collapses those into the v1.5.0 release artefact. Highlights:
Round 1-4 Site Wizard core: dry-run parity, single-line
value injection guard, ACL -f pattern-file block,
SSL parity, timeout regex, form-state pin.
Round 5-7 defaults-cookie inheritance, server-named-cookie
guard, fe/be mode mismatch, duplicate server
names, health_check_uri + server_address
validators.
Round 8-10 cookie_name / cookie_options newline-injection
guard, dry-run parity (round 9), TCP-mode HTTP-only
feature blockers.
Round 11 SSL name path traversal + health-check >= 1.
Round 12-13 SSL & ACME deep dive (Bulgu #23-#32).
Round 14 single-line value injection (Bulgu #33).
Round 15-17 ACME multi-tenant UX, numeric bounds, HAProxy
reserved keywords, ALPN/TLS consistency,
all-backup, multi-domain & multi-user enterprise
edges, drain/HSTS/post-completion (Bulgu
#34-#53).
Round 18-21 concurrency, agent state, TCP-mode HTTP-only,
list size caps, IPv6 scope-id, preview account
validation, TCP backend + balance uri reject
(Bulgu #54-#61).
Round 22 FE error visibility + 3x stale-data lockouts,
referential integrity + cascade safety,
authentication & authorization, multi-cluster
isolation, apply_pending_changes concurrency,
script injection + bulk import multi-tenancy,
prefix-stripped signature comparison
(Bulgu #62-#82).
------------------------------------------------------------------
NO CORPORATE-SPECIFIC ARTIFACTS
------------------------------------------------------------------
This squash deliberately sanitises corporate hostnames, container
registry references, and TLS secret names into generic
placeholders (`your-registry.example.com/your-org`,
`haproxy-openmanager*.example.com`, `wildcard-tls`,
`taylanbakircioglu/haproxy-openmanager-*`) so the public artefact
contains no internal infrastructure detail. Pilot / development
history that retained those values stays in the corporate fork
and is NOT part of this commit.
- Agent scripts now detect and send ip_address in DAEMON heartbeat (Linux: ip route, macOS: ifconfig)
- Backend validates agent-reported IPs via ipaddress stdlib, COALESCE preserves existing on NULL
- IP/VIP change logging (non-critical, try/except wrapped) for operational visibility
- New source_file_hash column on agent_script_templates for reliable update detection
- Migration changed to ON CONFLICT DO NOTHING to prevent overwriting UI-customized scripts on restart
- GET /versions returns script_update_available flag (disk hash vs DB hash comparison with fallback)
- Frontend Alert banner warns users of new agent script versions and directs to Reset to Defaults
- Reset to Defaults and Popconfirm modals explicitly warn about custom script edit loss
- Full backward compatibility: old agents without ip_address field continue working unchanged
Made-with: Cursor
Bulk import accepted dots in frontend/backend/server names but UI and
backend validators rejected them with ^[a-zA-Z0-9_-]+$. After import,
entities with dots could not be edited. HAProxy itself allows dots in
section names, so the regex is expanded to ^[a-zA-Z0-9_.-]+$ across
all 12 validation points (5 React form rules, 1 ACL char-strip,
3 Pydantic validators, 1 WAF validator, 2 config-validator warnings).
When a backend block in haproxy.cfg does not specify explicit timeout
values, the bulk import was injecting hardcoded defaults (connect 10s,
server 60s, queue 60s) into the database. These then appeared in the
generated config and overrode the agent's defaults section. Now, only
explicitly declared timeouts are stored; omitted ones remain NULL so
the agent's existing defaults section stays in effect.
- Step 3 (Enable ACME on Cluster) now shows a process icon instead of
a misleading green checkmark when ACME is enabled but not yet applied.
Per-cluster "(pending apply)" annotation for multi-cluster setups.
- Step 4 button and all /apply-management navigation buttons now say
"Apply Changes" instead of "Configure" for clearer guidance.
- Setup Guide auto-selects the correct cluster before navigating to
Apply Management, showing pending cluster names in alerts.
- Pending ACME disable changes are now correctly detected in Step 4
even when acme_enabled is already FALSE in the database.
- Entity snapshot rollback for cluster ACME settings: reject correctly
restores acme_enabled/acme_backend_url to pre-change values.
- Deduplication logic prevents "last wins" bug when multiple ACME
toggles are rejected in sequence.
- Connection leak prevention with try/finally around conn2 in ACME
config version creation.
- Step 4 branching uses boolean has_enabled instead of fragile string
truthiness check.
Made-with: Cursor
Agent startup left HAPROXY_CONFIG_PATH empty when the /api/clusters jq
select() returned no output (e.g. transient API failure or cluster_id
mismatch). This caused "No existing HAProxy config found at: " errors
and partial config merge failures for newly added agents.
Three-layer fix:
1. Load HAProxy paths from config.json as baseline before get_cluster_paths()
2. Use local variables in get_cluster_paths() - only override globals when
API returns non-empty values (defensive against jq select() empty output)
3. Dynamically update paths from /api/agents/{name}/config response in
check_config_updates() - allows cluster path changes without reinstall
Applied to both linux_install.sh and macos_install.sh.
Made-with: Cursor
- Fix DB: COALESCE(NULLIF) prevented clearing stale MASTER/BACKUP state.
Use CASE WHEN IS NOT NULL to distinguish absent fields (old agents)
from empty fields (keepalived stopped) and properly clear to NULL.
- Fix Redis: explicitly delete cache key when keepalived stops instead
of relying on 90s TTL expiry, preventing stale dashboard data.
- Fix agent scripts: SKIP_TO_DAEMON send_heartbeat now always sends
keepalive_state/keepalive_ip fields (even empty) so backend can
detect stopped keepalived and clear stale data.
- Fix legacy heartbeat endpoint: POST /{agent_id}/heartbeat now also
updates keepalive_state and keepalive_ip fields.
- UI: add horizontal scroll to ClusterManagement table to prevent
column overflow with new Keepalive column.
- UI: add VIP tooltip to Dashboard AgentStatusCard keepalive tag.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add keepalive_state and keepalive_ip columns to agents table (migration + schema)
- Add keepalive fields to AgentHeartbeat Pydantic model (backward compatible)
- Update heartbeat endpoint to persist keepalive data to DB and cache in Redis
- Add multi-method keepalived detection in agent scripts (journalctl, log files, VIP check)
- Update dashboard-stats agents/status API with Redis-first keepalive lookup
- Update GET /api/agents to include keepalive_state and keepalive_ip
- Show MASTER/BACKUP tag in Dashboard AgentStatusCard
- Show keepalive info in Agent Management registered agents table
- Add Keepalive column to Cluster Management table with VIP search support
Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause: SSL update passes expiry_date as Python datetime object in
new_values. json.dumps() fails on datetime, causing save_entity_snapshot
to return {} (empty). Entity snapshot is never saved in config_version
metadata, so reject/rollback can never find it to restore old values.
Fix: Apply same JSON serialization to new_values as old_values. Only
affects SSL certificates (only entity with datetime in new_values).
All other entity types (frontend, backend, server, waf) are unaffected
as their new_values contain only JSON-serializable types.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add per-cluster SSL sync status tracking with APPLYING/SYNCED/PARTIAL states
- Add timeout detection (10min) for stalled SSL deployments showing PARTIAL status
- Add SSL-specific info in Apply Management dialog (cluster count, auto-apply notice)
- Add Deployment Status tab in SSL details showing per-cluster agent sync progress
- Implement SSL auto-reject (reject propagates to all clusters) and auto-undo
- Add conditional entity rollback for SSL reject (time-window based safety check)
- Filter REJECTED status from SSL last_config_status display
- Add pending_cluster_names to SSL list API response
- Optimize agent SSL reload: only trigger HAProxy reload when changed cert is
actually referenced in the cluster's HAProxy config (grep -qF check), preventing
unnecessary reloads on clusters that don't use the updated certificate
- Applied to all 4 SSL deploy paths: Linux/macOS standalone and daemon embedded modes
Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause: After agent self-upgrade, the embedded daemon (SKIP_TO_DAEMON
block) runs instead of run_daemon(). HAPROXY_BIN and HAPROXY_CONFIG
variables were uninitialized before the daemon loop, causing SSL-triggered
HAProxy reloads to silently fail with empty path validation.
Changes:
- Initialize HAPROXY_BIN/HAPROXY_CONFIG before embedded daemon loop
- Add md5 checksum comparison in deploy_ssl_certificates() to detect
actual cert file changes (avoid unnecessary writes and reloads)
- Add check_ssl_updates() for independent SSL sync every ~2.5 min
in run_daemon(), independent of config version changes
- Add SSL-aware reload in check_config_updates(): if config validation
fails but SSL certs changed, reload HAProxy with existing config
- Add "full" fetch mode to fetch_and_deploy_ssl_certificates() to
bypass incremental timestamp filter for standalone SSL checks
Applied to both linux_install.sh and macos_install.sh.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Replace immediate exit 1 with retry-with-backoff in all daemon mode paths
(dependency checks: 5 retries, 30/60/90/120/150s; config file: 5 retries, 15/30/45/60/75s)
- Make socat missing non-fatal in daemon mode (agent continues without stats)
- Add SystemD StartLimitBurst=5/StartLimitIntervalSec=120 for new installs
- Add macOS launchd ThrottleInterval=30 for new installs
- Fix Linux daemon fallback to installer mode (would hang on interactive read)
Co-authored-by: Cursor <cursoragent@cursor.com>
Move jq dependency check to run before cluster validation which requires
it. Auto-install jq via the detected package manager (apt, dnf, yum,
zypper, apk for Linux; brew for macOS). Exit with clear manual install
instructions if automatic installation fails.
Co-authored-by: Cursor <cursoragent@cursor.com>
Same pgrep self-kill bug as uninstall scripts. When user downloads
install script as install-haproxy-agent.sh, pgrep -f "haproxy-agent"
matches the installer's own bash process and kills it before
installation starts. Now filters out INSTALLER_PID and PPID.
Co-authored-by: Cursor <cursoragent@cursor.com>
Instead of a hardcoded PROTECTED_PATHS list (varies per environment):
- safe_rm() blocks any path not containing "haproxy-agent"
- Process killer verifies ps output contains "haproxy-agent"
- Pre/post HAProxy integrity check via md5 hash comparison of config
and service status diff (was running -> still running?)
Co-authored-by: Cursor <cursoragent@cursor.com>
pgrep -f "haproxy-agent" was matching the uninstall script's own
process (uninstall-haproxy-agent-linux.sh contains "haproxy-agent"),
causing the script to terminate itself at step 1/7 before reaching
file cleanup. Now filters out $$ (own PID) and $PPID from kill lists.
Co-authored-by: Cursor <cursoragent@cursor.com>
CRITICAL BUG FIX:
- Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword
- HAProxy allows `defaults` without a name (no trailing whitespace)
- Pattern failed to match, causing global extraction to include defaults section
New pattern uses `([[:space:]]|$)`:
- Matches keyword followed by whitespace OR end of line
- `defaults` (no name) now correctly triggers exit
- `defaults http` (named) also correctly triggers exit
Test results:
- Old pattern: extracted 8 lines (included defaults) ❌
- New pattern: extracted 4 lines (stopped at defaults) ✅
CRITICAL BUG FIX:
- Previous awk pattern allowed multiple defaults sections to be captured
- Pattern `!/^defaults/` meant "don't exit if line IS defaults"
- This caused duplicate defaults when config had listen before defaults
New pattern uses `started` flag:
- First `defaults` line: set started=1, begin capturing
- Second `defaults` line: started is set, EXIT immediately
- Any `frontend/backend/listen`: started is set, EXIT
Also includes: debug logging for validation error storage with fallback
Tested scenarios:
- Normal config (defaults → listen → frontend): ✓
- Listen before defaults: ✓
- Two defaults sections: ✓ Only first captured
- No defaults section: ✓ Empty output
- Defaults at end of file: ✓
- Empty defaults section: ✓
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section,
the global extraction incorrectly included listen stats because it
only stopped at 'defaults', not at any section header.
Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend)
This prevents duplicate listen blocks in merged config.
Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error.
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
- Updated SUGGESTION_TEMPLATES in haproxy_error_parser.py
- Fixed fallback error messages in cluster.py
- Product language should be English throughout
When config content doesn't look like valid HAProxy config (missing
global/defaults/frontend/backend/listen keywords), agent now reports
this to backend via config-validation-failed endpoint.
This catches backend-side config generation errors (like Python
exceptions) that prevented actual HAProxy config from being generated.
Changes:
- Add invalid config format detection and reporting in daemon mode
- Use same curl pattern as existing HAProxy validation failure reporting
- Same endpoint, headers, error handling, and spam prevention
- Follows exact existing pattern for consistency
- Does not affect self-upgrade flow (runs before upgrade check)
Both linux and macos scripts updated identically.
- Copy uninstall-agent-linux.sh and uninstall-agent-macos.sh to
backend/utils/agent_scripts/ (same location as install scripts)
- Update generate-uninstall-script endpoint to use the same path
pattern as working install script endpoint
- Add container-specific fallback paths for robustness
- Add debug logging to track which path is used
Fixes 404 error when fetching uninstall script in production
where /utils/ directory at project root is not deployed.
- Add haproxy_error_parser.py: Parses HAProxy validation errors with
confidence scoring, extracts entity type/name, line number, error type
- Add ValidationErrorModal.js: Rich modal with parsed error summary,
quick fix suggestions, and manual troubleshooting guide
- Update cluster.py: Integrate error parser into agent-sync and
config-versions endpoints with graceful fallback
- Update ApplyManagement.js: Add validation error banner with quick
navigation buttons and error detail modal
- Update FrontendManagement.js & BackendServers.js: Handle URL params
for deep-linking to entity edit forms with field highlighting
Enables users to see actionable validation failure details directly
in the UI without needing server access for debugging.
- Add 'random' and 'first' options to backend balance method selector
- Add balance method validation in config parser with warning for unknown methods
- Update API documentation with all supported balance algorithms
- Add haproxy_version field to heartbeat payload in Linux agent script
- Add haproxy_version field to heartbeat payload in macOS agent script
- Display HAProxy version below IP address in Registered Agents list
- Safe extraction with fallback to 'unknown' if haproxy command fails
- Version is updated on every heartbeat (30s interval)
- Green color styling for easy visibility
Backend already supports haproxy_version field in AgentHeartbeat model
and saves it to database on each heartbeat.
CRITICAL FIXES:
1. Agent Heartbeat Validation Error (HTTP 422)
- Added missing 'system_info' field to AgentHeartbeat model
- Agents were sending system_info but backend model didn't accept it
- No agent script update needed - agents already send this field
2. Bulk Import SSL Parser Enhancement
- Fixed parsing of multiple SSL certificates with alpn/npn parameters
- Example: 'bind :443 ssl crt cert1.pem crt cert2.pem crt cert3.pem alpn h2,http/1.1'
- Old parser stopped at first whitespace after crt path
- New parser extracts all crt paths even with alpn/npn/ciphers after them
- Added user-friendly warning for SSL parameters (alpn, npn, ciphers) that won't be imported
TECHNICAL DETAILS:
- backend/models/agent.py: Added system_info: Optional[Dict[str, Any]]
- backend/utils/haproxy_config_parser.py: Enhanced SSL bind parsing logic
* Parse bind line by splitting and iterating through parts
* Extract all crt paths before hitting SSL parameters
* Detect and warn about alpn, npn, ciphers, ciphersuites parameters
* Inform user these advanced options should be configured manually
USER IMPACT:
- Agents will come online after backend deployment (no reinstall needed)
- Bulk import will correctly parse configs with multiple SSL certs + alpn
- Clear warnings shown in UI about SSL parameters not imported
PRODUCTION FIX: Prevent agent crashes from command failures
PROBLEM: set -e causes immediate exit on any command failure
The 'set -e' directive at the beginning of agent scripts caused agents to exit
immediately when ANY command returned a non-zero exit code. This was causing
production instability:
- Agent exits unexpectedly on minor errors
- systemd restarts agent continuously
- Creates restart loops
- Prevents agent from reaching daemon mode
- Configuration updates lost
- Metrics collection interrupted
EXAMPLES OF TRIGGERS:
- DNS lookup failures
- Temporary network issues
- HAProxy stats socket unavailable
- File system temporarily busy
- Any non-critical command failure
SOLUTION: Remove 'set -e' and rely on explicit error handling
Instead of crashing on errors, agents now:
- Log errors with context
- Continue running in daemon mode
- Handle errors gracefully
- Maintain service availability
- Only exit on critical failures (explicitly coded)
DEPLOYMENT STRATEGY:
1. Manual temporary fix: Comment out 'set -e' on agent servers
2. UI-driven upgrade: Deploy new script version (1.0.12)
3. Result: Stable agents with proper error handling
PRODUCTION IMPACT:
- 15/15 production agents upgraded successfully
- No agent crashes or restart loops
- All configuration updates working
- Metrics collection stable
- Zero downtime deployment
CRITICAL PRODUCTION BUG: Agent stuck in restart loop after upgrade
SYMPTOMS:
- Agents continuously restarting every ~30 seconds
- Log shows: "Sending upgrade completion heartbeat..."
- Log shows: "Agent upgrade completed successfully"
- systemd restarts agent immediately after
- Agents never reach daemon loop
- Configuration updates not received
- Entity updates not applied
ROOT CAUSE:
- Agent script v1.0.10 had misleading "upgrade completion heartbeat"
- This heartbeat was sent EVERY time daemon started
- After sending, script would exit (expecting systemd restart)
- systemd would restart agent → infinite loop
- Agent never reached check_agent_upgrade() or check_config_updates()
MISLEADING CODE (REMOVED):
SOLUTION:
- Removed "upgrade completion heartbeat" from daemon startup
- Agent sends normal heartbeat in daemon loop (every 30s)
- No special "upgrade completion" needed
- Agent stays in daemon mode continuously
- systemd only restarts on actual failures
IMPACT:
- Agents no longer restart in loop
- Configuration updates work normally
- Entity updates applied successfully
- Upgrade process works correctly
- Production stability restored
PRODUCTION ENHANCEMENT: Handle extremely large stats CSV payloads
IMPROVEMENT OVER PREVIOUS FIX:
- Previous: Temp file for response only
- Now: Temp file for BOTH payload and response
- Reason: Very large payloads (>1MB) still hit argument limits
PRODUCTION SCENARIO:
- Large HAProxy instances with 100+ backends
- Stats CSV can exceed 1MB in production
- curl --data argument hits system limits
- Need temp file for payload itself
SOLUTION:
- Write heartbeat_payload to temp file
- Use curl --data-binary @temp_payload
- Write response to separate temp file
- Read HTTP code and response body
- Cleanup both temp files
BENEFITS:
- Unlimited payload size support
- No argument list limits
- Production-tested and safe
- Backward compatible
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon (Line ~1099) and installer (Line ~2367)
PRODUCTION BUG: Argument list too long when sending large stats CSV
ERROR MESSAGE:
"Heartbeat failed (HTTP /usr/local/bin/haproxy-agent: line 417: /usr/bin/curl: Argument list too long)"
ROOT CAUSE:
- curl output capture exceeded system argument list limit
- Large stats CSV (>200KB, production can be >1MB)
- Shell variable assignment hit system limits
SOLUTION:
- Redirect curl output to temp file (/tmp/heartbeat_response_$$.txt)
- Read HTTP code and response body from temp file
- Cleanup temp file immediately after use
BENEFITS:
- No size limit on HTTP responses
- Production-safe for large HAProxy instances
- Better error handling with detailed logging
FILES CHANGED:
- backend/utils/agent_scripts/linux_install.sh
- backend/utils/agent_scripts/macos_install.sh
- Updated both embedded daemon and installer functions
Problem 1: Apply affects all entities (should only affect changed ones)
Problem 2: Reject rollback not working (entity stays at new value)
Added detailed logging:
- Snapshot creation: JSON test result, field count
- Frontend update: metadata keys, entity_snapshot presence
- Reject: metadata parsing, entity_snapshot detection
- Rollback: entity data, operation type, old_values
- _rollback_update: Before/after values, UPDATE query result
- Verify: Post-rollback database state
This will help identify:
- Is snapshot being created?
- Is metadata being saved to database?
- Is metadata being parsed during reject?
- Is rollback function being called?
- Is UPDATE query executing?
- What are the actual values being restored?
Log locations to check:
kubectl logs deployment/haproxy-openmanager-backend -n haproxy-openmanager | grep 'SNAPSHOT\|ROLLBACK\|REJECT'
taylanbakircioglu