34 Commits

Author SHA1 Message Date
taylanbakircioglu 9e5185c458 fix(security): post-review hardening — agent-inventory regression, coverage gaps, SSRF newNonce
Follow-up to the RCE/missing-auth/SSRF remediation, from a thorough multi-lens
review (3 agents + a black-box audit of all 201 routes). Backend-only; no
agent-script changes.

Regression fix (introduced by the previous commit):
- GET /api/agents was made JWT-only, but deployed agents call it WITH X-API-Key
  (not a JWT) to read their applied_config_version and avoid re-applying config on
  restart. It now accepts EITHER a valid operator JWT OR a valid agent X-API-Key,
  so agents no longer get 401 (which caused a spurious HAProxy reload every restart).

Completeness (GHSA-3p5c siblings the first pass missed — same data class, now JWT):
- dashboard.py: GET /api/haproxy-cluster-pools/{id}/agents (full agent inventory —
  a direct anonymous bypass of the GET /api/agents lockdown), /api/pools,
  /api/haproxy-cluster-pools, /api/dashboard/stats, /api/dashboard/overview
  (auth was optional -> leaked stats/names/health/alerts anonymously),
  /api/haproxy/stats.
- waf.py: GET /api/waf/rules. health.py: GET /api/health/errors.
- agent.py: GET /api/agents/generate-uninstall-script/{platform} (agent-management
  endpoint; was anonymous) now requires JWT or agent key, like generate-install-script.
- config.py: POST /api/config/{validate,optimize,templates/{id}/generate} were
  optional-auth (logging only) and run a HAProxy validator on caller input; now
  require a JWT. (bulk-create, parse-bulk, diff and configuration/request were
  already mandatory-auth — verified.)
  All newly-gated endpoints are frontend-only (axios sends the JWT) or unused;
  agents never call them.

SSRF (GHSA-3vh4) gap:
- acme_service._get_nonce fetched directory['newNonce'] (from the attacker-
  influenceable directory JSON) with a bare session, http allowed, dual-stack, and
  BEFORE the guarded _signed_request POST. Now guarded (assert_public_url +
  safe_connector + no redirects + timeout), matching the other ACME sinks.

Correctness:
- Three agent webhooks (config-applied, config-validation-failed, config-sync)
  swallowed their auth 401 into a 200 error body via a bare `except Exception`.
  Added `except HTTPException: raise` so the 401/403 propagates.

Audit result (live black-box, all 201 routes probed unauthenticated): no data
leak and no unauthenticated mutation anywhere; every sensitive route returns
401/403 (a pre-existing group of read handlers wraps the 401 into a 500 via a
broad except — no data is exposed; left as-is, documented as cosmetic).

Verified: full pytest tests/ (1145 passed, 0 failed; +16 regression tests) + live
localtest stack smoke — agent-key GET /api/agents=200, anonymous=401, all newly
gated endpoints reject anonymous and admit JWT, the 3 webhooks return 401.
2026-07-20 13:43:32 +03:00
taylanbakircioglu c79391cd13 feat(acl): accept HAProxy -f pattern-file references with advisory warnings (v1.8.9, Issue #38)
The manual Frontend editor, wizard and visual ACL builder hard-rejected the ACL
`-f <file>` flag while bulk import accepted it. Worse, a frontend imported with
an `-f` ACL could not be edited at all (422) until the ACL was dropped.

The original guard predated the fail-safe apply flow: the agent runs `haproxy -c`
before every reload, so a missing pattern file is rejected safely and the previous
config keeps running. Pattern files are operator-managed host files — the same
policy adopted for SPOE filter configs in v1.8.8.

- models: remove the 5 `-f` hard rejects (frontend acl/redirect/use_backend
  validators + wizard string/dict-redirect guards); `$(`/backtick and X!X
  contradiction guards unchanged
- routers/frontend: `_pattern_file_warnings` helper; non-blocking warning on
  create + update responses listing referenced pattern files (empty when no
  rule uses `-f` — zero noise)
- routers/config: bulk-import preview advisory listing pattern files per
  frontend (cluster config-dir aware, next to the SPOE advisories)
- React: remove the FrontendManagement submit gate and SiteWizard step gate;
  ACLRuleBuilder renders informational notes instead of errors and re-adds
  `-f (pattern file on host)` to the flag dropdown; create path now renders
  server warnings like update
- tests: 4 reject-pins inverted to accept-pins; new test_acl_pattern_file_allow.py
  (accept/guards-kept/zero-noise/advisory); full suite green (1094 passed)
2026-07-14 00:27:11 +03:00
taylanbakircioglu 9e2ea04777 feat(haproxy): preserve SPOE filter + frontend log-format on import/edit (v1.8.8, Issue #38)
Bulk import / manual edit silently dropped `filter spoe engine ...` (Coraza WAF)
and frontend `log-format` because the parser recognised only a fixed directive
set. The regenerated config then missed the SPOE engine, so HAProxy failed with
"unable to find SPOE engine 'coraza' used by the send-spoe-group".

- parser: capture `filter` + `log-format`/`log-format-sd` into new ParsedFrontend fields
- db: additive nullable `log_format` + `filters` TEXT columns on frontends (SCHEMA_VERSION 8->9)
- generator: new `filter` bucket flushed before http-request rules so `filter` precedes
  `send-spoe-group`; `log-format` kept in prelude
- bulk import: preview dict, change-detection, persist (create + merge-update); cluster-aware
  SPOE pre-flight advisories (missing-filter + host-prerequisite) surfaced in the UI
- manual CRUD: full round-trip (get/create/update) incl. React form fields (no null-wipe)
- reject/rollback: restore the new columns; restore path + wizard helper kept in parity
- backend `option spop-check` recognised (suppresses spurious warning for coraza-spoa)
- tests: test_spoe_filter_import.py; full suite green (1079 passed)
2026-07-10 18:34:36 +03:00
taylanbakircioglu 02b1cb2bca feat: v1.5.0 — Site Wizard (Issue #14) + ACME Diagnostic Panel (Issue #13)
Closes #13, Closes #14.

This release squashes the v1.4.0 → v1.5.0 development line. v1.4.0
shipped the ACME stability & enterprise audit (Issues #10/#11/#12).
v1.5.0 builds on that foundation with two co-equal headline features
plus a 22-round audit campaign hardening the prior configuration
surface. License remains MIT for v1.5.0 (relicense to AGPL-3.0
lands in v1.5.2).

------------------------------------------------------------------
HEADLINE FEATURE A — ACME Diagnostic Panel (Issue #13)
------------------------------------------------------------------
A live pre-flight + post-failure diagnostic surface for every ACME
order, reachable from the ACME Automation page. The panel exists
to make ACME failures legible to operators who do NOT have shell
access to the API host.

Endpoints (`backend/routers/acme_diagnostics.py`):
  POST /api/letsencrypt/orders/{order_id}/diagnostics
       Run the full 5-check suite (DNS / port-80 / routing /
       account / agents) and humanize the order's `error_detail`
       (>=11 RFC-8555 problem types, backwards compatible with
       legacy plain-string failures).
  POST /api/letsencrypt/orders/{order_id}/diagnostics/
                                {check_id}/rerun
       Re-run a single check in place — used by the "Re-run"
       button on every row of the modal's pre-flight table.
  GET  /api/letsencrypt/orders/{order_id}/events
       Merged event timeline combining the typed
       `acme_order_events` rows with correlated
       `user_activity_logs` entries (resource_type =
       'letsencrypt_order' AND resource_id = order_id). The
       diagnostic modal auto-tails this timeline every 5 seconds
       while open.

Service-level checks (`backend/services/acme_diagnostics.py`):
  * DNS resolution via stdlib socket.gethostbyname_ex through
    run_in_executor (intentionally avoiding an aiodns runtime
    dep for v1.5.0).
  * Port-80 HEAD probe, target locked to the order's domains,
    success on HTTP 200 OR 404, warns on egress timeout
    (corp egress policies routinely blackhole outbound 80 —
    fail-hard would be too noisy).
  * SSRF guard: probe refuses non-public IPs and surfaces the
    skip in the diagnostic result; IPv4-mapped IPv6 normalisation
    closes the `::ffff:169.254.169.254` cloud-metadata vector.
  * HAProxy routing presence check: matches the order's
    cluster_ids to a port-80 HTTP frontend.
  * ACME account validity check against `letsencrypt_accounts`.
  * Agent presence check (>=1 active agent in target cluster).
  * Every sub-check wrapped in a wall-clock timeout to bound
    impact on the API event loop.

RBAC: ssl.read for run, ssl.read for events. Per-user 5/min rate
limit on both run and rerun, backed by the (user_id, action,
created_at DESC) composite index.

Frontend (`frontend/src/components/ACMEAutomation.js`):
  * "Diagnose" button on every order row + the existing
    "stuck order" warning row.
  * Modal with two tabs:
    - Pre-flight Checks (Antd Table with status pills + Re-run
      buttons + humanized error banner)
    - Event Log (Antd Timeline with auto-tail polling, scroll-
      to-bottom, pause-on-hover)
  * Correlation IDs surfaced in error banners and individual
    check fail details for backend-log lookup.

------------------------------------------------------------------
HEADLINE FEATURE B — Site Setup Wizard (Issue #14)
------------------------------------------------------------------
A single guided flow that creates a Backend + Servers + HTTP
Frontend (and optional HTTPS Frontend) in one atomic transaction.

Endpoints (`backend/routers/site_wizard.py`):
  POST /api/site-wizard/preview     — diff-preview the changeset
  POST /api/site-wizard/create      — atomic execute
  POST /api/site-wizard/reject      — clean rollback (including
                                       any wizard_staged ACME
                                       orders)
  GET  /api/site-wizard/drafts      — draft persistence
  PUT  /api/site-wizard/drafts/{id} — save/update
  DELETE /api/site-wizard/drafts/{id}

Feature surface:
  * One screen captures both backend (mode + servers) AND
    frontend (http + optional https + SSL mode) inputs.
  * SSL modes: ACME (new order, HTTP-01 only for v1.5.0),
    Upload (existing PEM), Existing (link to a stored cert),
    or None.
  * ACME-staged path: wizard_staged_until watermark on the
    `letsencrypt_orders` row defers finalisation until agent
    confirmation; per-mode reject cleanly cancels and rolls
    back the staged order.
  * Live diff preview against the cluster's current generated
    config (renderer-evolution noise stripped — track-sc<N>
    dedup, per-server cookie strip, defaults-cookie
    inheritance, listen-block flattening).
  * Draft persistence with PEM stripped at save time (private
    keys never round-trip through the drafts table).
  * Per-cluster multi-tenancy: drafts and wizard_staged orders
    are isolated to the creating user's cluster scope.

Frontend (`frontend/src/components/SiteWizard.js`):
  * 4-step Antd Steps flow: Backend → Frontend → SSL → Review.
  * Render the live diff preview inline before commit.
  * Antd Form-level validation mirrors backend Pydantic
    validators (numeric bounds, HAProxy reserved keywords, ALPN
    consistency, IPv6 scope-id, domain regex, server name
    dedup).

------------------------------------------------------------------
AUDIT CAMPAIGN — Rounds 1 → 22 (Bulgu #1#82)
------------------------------------------------------------------
v1.5.0 includes 22 adversarial review passes. Each round produced
its own commit set in the corporate development line; this squash
collapses those into the v1.5.0 release artefact. Highlights:

  Round 1-4   Site Wizard core: dry-run parity, single-line
              value injection guard, ACL -f pattern-file block,
              SSL parity, timeout regex, form-state pin.
  Round 5-7   defaults-cookie inheritance, server-named-cookie
              guard, fe/be mode mismatch, duplicate server
              names, health_check_uri + server_address
              validators.
  Round 8-10  cookie_name / cookie_options newline-injection
              guard, dry-run parity (round 9), TCP-mode HTTP-only
              feature blockers.
  Round 11    SSL name path traversal + health-check >= 1.
  Round 12-13 SSL & ACME deep dive (Bulgu #23-#32).
  Round 14    single-line value injection (Bulgu #33).
  Round 15-17 ACME multi-tenant UX, numeric bounds, HAProxy
              reserved keywords, ALPN/TLS consistency,
              all-backup, multi-domain & multi-user enterprise
              edges, drain/HSTS/post-completion (Bulgu
              #34-#53).
  Round 18-21 concurrency, agent state, TCP-mode HTTP-only,
              list size caps, IPv6 scope-id, preview account
              validation, TCP backend + balance uri reject
              (Bulgu #54-#61).
  Round 22    FE error visibility + 3x stale-data lockouts,
              referential integrity + cascade safety,
              authentication & authorization, multi-cluster
              isolation, apply_pending_changes concurrency,
              script injection + bulk import multi-tenancy,
              prefix-stripped signature comparison
              (Bulgu #62-#82).

------------------------------------------------------------------
NO CORPORATE-SPECIFIC ARTIFACTS
------------------------------------------------------------------
This squash deliberately sanitises corporate hostnames, container
registry references, and TLS secret names into generic
placeholders (`your-registry.example.com/your-org`,
`haproxy-openmanager*.example.com`, `wildcard-tls`,
`taylanbakircioglu/haproxy-openmanager-*`) so the public artefact
contains no internal infrastructure detail. Pilot / development
history that retained those values stays in the corporate fork
and is NOT part of this commit.
2026-05-14 00:04:19 +03:00
taylanbakircioglu cd4a94beb1 feat: agent IP/VIP live update + script update detection banner
- Agent scripts now detect and send ip_address in DAEMON heartbeat (Linux: ip route, macOS: ifconfig)
- Backend validates agent-reported IPs via ipaddress stdlib, COALESCE preserves existing on NULL
- IP/VIP change logging (non-critical, try/except wrapped) for operational visibility
- New source_file_hash column on agent_script_templates for reliable update detection
- Migration changed to ON CONFLICT DO NOTHING to prevent overwriting UI-customized scripts on restart
- GET /versions returns script_update_available flag (disk hash vs DB hash comparison with fallback)
- Frontend Alert banner warns users of new agent script versions and directs to Reset to Defaults
- Reset to Defaults and Popconfirm modals explicitly warn about custom script edit loss
- Full backward compatibility: old agents without ip_address field continue working unchanged

Made-with: Cursor
2026-04-17 10:50:51 +03:00
taylanbakircioglu 61024e24cb fix: strip rate-limit and WAF auto-generated content from bulk import comparison
Made-with: Cursor
2026-04-17 10:45:12 +03:00
taylanbakircioglu 6a349e22c6 fix: strip auto-generated ACME content from bulk import preview
Made-with: Cursor
2026-04-17 10:43:30 +03:00
taylanbakircioglu f37f3afd71 feat: bulk import change detection, multi-select delete, auto-content stripping
- Server-level change detection in bulk import (field-by-field comparison
  for 17 server attributes with UPDATE/NO CHANGES status and tooltip)
- Multi-select delete for backends and frontends with dependency checks
- Dashboard "Backends Summary" address column for servers
- Fix unique constraint violation on bulk-create for existing servers
  (natural key lookup matching DB constraint instead of backend_id FK)
- ORDER BY is_active DESC on all entity lookups to prefer active records
- Strip auto-generated content (ACME, rate-limit, WAF) from bulk import
  comparison to eliminate false positive changes on re-import
- Fix toolbar overflow with Space wrap prop
- Frontend bulk delete modal clarity (selected vs deletable count)
- Version bump to 1.3.0

Made-with: Cursor
2026-04-14 20:21:07 +03:00
taylanbakircioglu 7fffaddaf8 fix: bulk import should not inject hardcoded timeout defaults
When a backend block in haproxy.cfg does not specify explicit timeout
values, the bulk import was injecting hardcoded defaults (connect 10s,
server 60s, queue 60s) into the database. These then appeared in the
generated config and overrode the agent's defaults section. Now, only
explicitly declared timeouts are stored; omitted ones remain NULL so
the agent's existing defaults section stays in effect.
2026-04-13 05:52:31 +03:00
taylanbakircioglu 93d7ad8fdb feat: add ACME Auto SSL with Let's Encrypt integration (v1.1.0)
Add automated SSL certificate management via ACME protocol (RFC 8555):
- Full ACME client implementation (account registration, HTTP-01 challenges, certificate issuance/renewal)
- Configurable ACME providers (Let's Encrypt, ZeroSSL, custom CA) via Settings UI
- Auto-renewal scheduler with PENDING -> Apply -> APPLIED flow alignment
- ACME account management (register, deactivate) from UI
- Certificate request wizard with domain validation and cluster targeting
- Zero changes to HAProxy agent scripts - challenges routed through existing architecture
- Comprehensive security hardening (no private key exposure in API responses)
- Full backward compatibility with existing SSL, Apply, Rollback, and Restore workflows
- Updated README, API documentation, and Kubernetes deployment notes
- Version management embedded in code (v1.1.0)
- UI messaging improvements for agent-pull architecture accuracy

Made-with: Cursor
2026-04-02 00:25:50 +03:00
taylanbakircioglu 095c682fa3 fix: handle soft-deleted entities in bulk import and frontend deletion
Bulk Import Fix (config.py):
- Check for existing servers (including soft-deleted) before INSERT
- If server exists: UPDATE and reactivate instead of INSERT
- Create snapshot for ALL updated servers (enables reject/rollback)
- Add backend_name to UPDATE for field consistency
- Track updated/reactivated servers in response
- Prevents duplicate key constraint violation on re-import

Frontend Deletion Fix (frontend.py):
- Add hard delete support for already soft-deleted frontends
- Follows same pattern as backend.py hard delete
- Deletes WAF associations and config versions before frontend
- Works with existing orphan cleanup in Apply/Reject flows

Both fixes are safe because:
- Orphan version cleanup handles hard-deleted entities in Apply/Reject
- Consistent with existing backend.py patterns
- Snapshot-based rollback fully supported
2026-01-26 15:25:15 +03:00
taylanbakircioglu 851377aedf feat: Add HAProxy proxy name collision prevention system
- Add preserved_listen_blocks column to agents table for storing agent's local listen block names
- Implement reserved names check (stats, monitoring, admin, etc.) for frontend/backend creation
- Add dynamic collision detection against agent's preserved listen blocks
- Apply collision checks to CREATE, UPDATE endpoints and bulk import
- Add debug mode for failed config validation (saves to /tmp/haproxy-failed-*.cfg)
- Fix JSON character stripping for ACL and use_backend rules
- Remove collision protection from agent scripts (now handled by backend)
- All collision checks wrapped in try-except for backwards compatibility
2026-01-26 15:25:15 +03:00
Taylan Bakırcıoğlu e8690245af fix(ssl): add change detection and validators for SSL advanced options
Critical additions:
1. Change Detection (backend/routers/config.py)
   - Added SSL advanced options to bulk import change detection logic
   - Detects changes in ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites
   - Detects changes in ssl_min_ver, ssl_max_ver, ssl_strict_sni
   - Changes will now appear in version diff

2. Pydantic Validators (backend/models/frontend.py, backend/models/backend.py)
   - TLS version validator: Only allows valid versions (SSLv3, TLSv1.0-1.3)
   - ALPN protocol validator: Only allows h2, http/1.1, http/1.0, h2c, spdy/*
   - NPN protocol validator: Only allows http/1.1, http/1.0, spdy/*
   - Prevents invalid values from being saved to database
   - HAProxy validation will not fail due to invalid SSL options

Impact:
- Bulk import will correctly detect SSL option changes
- Apply Management diff will show SSL changes
- User cannot enter invalid TLS versions or protocols
- Improved UX with early validation errors

Previous fixes in this series:
- Frontend GET API: Added SSL fields to response
- Bulk Import UPDATE: Added SSL fields to UPDATE statement
- Backend Server GET API: Added SSL fields to response

Test: Bulk import with ALPN change → Should see change in version diff
2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu 3e2f3e3d9f fix(ssl): comprehensive SSL advanced options handling across all endpoints
Critical fixes for SSL advanced options (alpn, npn, ciphers, ciphersuites, min-ver, max-ver, strict-sni):

1. Frontend GET API (backend/routers/frontend.py)
   - Added all 7 SSL advanced option fields to response
   - Frontend Edit modal will now display these fields
   - Prevents NULL overwrite when user edits frontend

2. Bulk Import Frontend UPDATE (backend/routers/config.py)
   - Added all 7 SSL advanced option fields to UPDATE statement
   - Previously skipped with comment 'MVP: DON'T update SSL settings'
   - Now bulk import re-runs preserve SSL settings

3. Backend Server GET API (backend/routers/backend.py)
   - Added 4 server SSL fields (ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers)
   - SQL SELECT query updated
   - Response object updated
   - Backend Server Edit modal will now display these fields

Root cause: GET APIs were not returning SSL advanced options, causing:
- UI forms to show empty fields
- User edits to overwrite with NULL
- Data loss on subsequent updates

All other endpoints (POST, PUT, INSERT) were already correct.

Impact:
- No more accidental data loss when editing frontends/servers
- Bulk import now preserves SSL advanced options
- UI will correctly display all SSL parameters

Test: Deploy backend, run bulk import, verify ssl_alpn appears in database
2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu 35dd8e5d13 Fix KeyError: ssl_name in bulk import parse
CRITICAL BUG: P1 fix broke parse response due to dict format change

ROOT CAUSE:
- P1 changed ssl_auto_assigned_frontends dict format
- Old: {'frontend': ..., 'ssl_name': ..., 'ssl_id': ...}
- New: {'frontend': ..., 'matched_certs': [...], 'total_matched': ...}
- But lines 943, 967 still accessed f['ssl_name'] → KeyError!

SOLUTION:
- Line 944: Loop through matched_certs to extract ssl_names
- Line 969: Build matched_info from matched_certs array
- Multi-SSL format now consistent throughout

EXAMPLE OUTPUT:
SSL AUTO-ASSIGNED: 1 frontend(s) automatically matched
Matched: public_ssl (3 certs: example-cert1, demo-cluster-cert, example-cert3)

IMPACT:
- Parse now completes successfully
- Shows all 3 matched SSL certs in response
- User sees detailed SSL matching info

Ref: demo-cluster parse error
2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu 7492d85649 FIX: Multiple SSL certificates & advanced params parsing
CRITICAL BUG FIX: Only 1 SSL cert matched instead of 3 for public_ssl

ROOT CAUSE:
- Parser only stored first cert path (ssl_cert_path)
- Bulk import SSL matching only checked single path
- ssl_certificate_ids = [2] (should be [2,3,4] for 3 certs)
- ssl_alpn = NULL (should be 'h2,http/1.1')

SOLUTION:
1. Added ssl_cert_paths List[str] to ParsedFrontend
2. Parser now stores ALL cert paths from bind directive
3. Bulk import loops through all cert paths for matching
4. SSL advanced options (alpn, npn, ciphers, etc.) included in frontends_data

EXAMPLE BIND DIRECTIVE:
bind 0.0.0.0:8443 ssl
  crt /etc/ssl/certs/example-cert1.pem
  crt /etc/ssl/certs/demo-cluster-cert.pem
  crt /etc/ssl/certs/example-cert3.pem
  alpn h2,http/1.1

BEFORE:
- ssl_certificate_ids: [2]  (only first cert)
- ssl_alpn: NULL            (not passed to bulk import)

AFTER:
- ssl_certificate_ids: [2, 3, 4]  (all 3 certs matched)
- ssl_alpn: 'h2,http/1.1'         (parsed & stored)

IMPACT:
- Multi-SSL frontends correctly imported
- SSL advanced params preserved & editable in UI
- SNI-based routing works correctly
- HTTP/2 ALPN negotiation preserved

TESTING:
- Parser: 3 crt paths → ssl_cert_paths = [path1, path2, path3]
- Matching: 3 paths → 3 IDs (if all SYNCED)
- Database: ssl_certificate_ids JSONB = [2,3,4]

Ref: demo-cluster public_ssl frontend issue
2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu 2819eeb515 Fix Backend API endpoints for SSL Advanced Options - Part 2
CRITICAL FIX: All API endpoints now handle SSL advanced options

 FRONTEND ENDPOINTS FIXED:
- create_frontend() - INSERT statement now includes all 7 SSL advanced fields
- update_frontend() - UPDATE statement now includes all 7 SSL advanced fields
- Bulk import frontend INSERT - All SSL params included

 BACKEND SERVER ENDPOINTS FIXED:
- add_server_to_backend() - INSERT now includes 4 SSL server advanced fields
- update_server() - allowed_fields list now includes SSL params (dynamic update)
- Bulk import server INSERT - All SSL params included

FIXED FIELDS:
Frontend: ssl_alpn, ssl_npn, ssl_ciphers, ssl_ciphersuites, ssl_min_ver, ssl_max_ver, ssl_strict_sni
Server: ssl_sni, ssl_min_ver, ssl_max_ver, ssl_ciphers

IMPACT:
- Bulk import now correctly saves parsed SSL parameters to database
- Frontend edit/create now accepts SSL params from UI
- Server edit/create now accepts SSL params from UI
- All CRUD operations fully support SSL advanced options

NEXT: Frontend UI components (FrontendManagement.js & BackendServers.js)
2025-11-18 21:58:05 +03:00
taylanbakircioglu c947273a28 Already committed in previous message 2025-11-14 01:06:38 +03:00
taylanbakircioglu b427db4c78 feat(snapshot): PHASE 4 & 5 - Bulk Import & Restore snapshot integration
PHASE 4: Bulk Import Integration
- config.py - bulk_create_entities() enhanced with snapshot support
- Backend UPDATE: Snapshot before update (bulk_snapshots array)
- Backend CREATE: Snapshot for rollback (DELETE on reject)
- Frontend UPDATE: Snapshot before update
- Frontend CREATE: Snapshot for rollback (DELETE on reject)
- Server CREATE: Snapshot for rollback (DELETE on reject)
- Bulk metadata: bulk_snapshots array, operation=BULK_IMPORT
- Total entity count tracking in metadata

PHASE 5: Restore Integration
- cluster.py - confirm_restore_config_version() enhanced
- Frontend UPDATE: Snapshot before restore (operation=UPDATE_RESTORE)
- Backend UPDATE: Snapshot before restore (operation=UPDATE_RESTORE)
- SELECT * for full field capture (not just parsed fields)
- Restore metadata: bulk_snapshots + pre_apply_snapshot
- operation=RESTORE tracking
- Rollback support for restore + reject scenario

Key Features:
- Bulk import now creates single config version with multiple entity snapshots
- Restore creates snapshots for all updated entities
- Reject after bulk import: Rollback all entities (UPDATE to old values, CREATE deleted)
- Reject after restore: Rollback to pre-restore state
- All emojis removed from code (clean logging)
- Diff viewer compatibility maintained (pre_apply_snapshot)

Implementation Complete:
- PHASE 1: Infrastructure (entity_snapshot.py) - DONE
- PHASE 2: Entity updates (5 entities) - DONE
- PHASE 3: Reject rollback logic - DONE
- PHASE 4: Bulk import snapshot - DONE
- PHASE 5: Restore snapshot - DONE

Next: Production testing with feature flag (ENTITY_SNAPSHOT_ENABLED=false by default)
2025-11-14 01:06:38 +03:00
taylanbakircioglu 46735c1894 fix: Implement consistent use-service handling across frontend and backend
- Add use-service skip to backend parser (consistent with frontend)
- Add use-service preservation to backend bulk import merge strategy
- Remove use-service merge from frontend PUT endpoint (allows user deletion)
- Ensures frontend/backend full consistency for use-service directives

Changes:
1. Backend parser now skips use-service directives during bulk import
2. Backend bulk import preserves manually-added use-service directives
3. Frontend PUT no longer prevents use-service deletion by users

Impact:
- Prevents loss of manually configured services during bulk imports
- Allows users to freely add/remove use-service directives via UI
- Frontend and Backend now have identical use-service handling logic

Related fixes:
- Bug 2: use-service deleted on bulk import (now preserved)
- Bug 3: use-service cannot be deleted via UI (now deletable)
2025-11-14 01:06:38 +03:00
taylanbakircioglu 1276725f45 fix: Frontend modal - preserve tcp_request_rules, redirect_rules and use-service on update
🐛 Bug Düzeltmeleri:

1. tcp_request_rules field'ı edit modal'da gösterilmiyordu
   - handleEdit() setFieldsValue'ya tcp_request_rules eklendi
   - Artık DB'deki TCP rules edit modal'da görünüyor

2. use-service header'ları (prometheus-exporter gibi) bulk import'ta siliniyordu
   - config.py bulk_create_entities'e merge stratejisi eklendi
   - Mevcut use-service direktifleri parse edilip yeni header'larla birleştiriliyor
   - Manuel eklenmiş servisler korunuyor

3. redirect_rules field'ı form'da hiç yoktu (yeni bulgu!)
   - ACL Rules panel'ına redirect_rules textarea eklendi
   - HTTP to HTTPS redirect kuralları için kullanılabilir

 Diğer modaller kontrol edildi:
   - Backend modal: Tam ✓
   - Server modal: Tam ✓
   - SSL modal: Tam ✓
   - WAF modal: Tam ✓
2025-11-14 01:06:38 +03:00
taylanbakircioglu 4c9ee88549 feat: Add field-level change visualization in bulk import preview
UX IMPROVEMENT: Users can now see exactly what changed in existing entities.

BACKEND CHANGES (config.py):
- Parse endpoint now tracks field-level changes for frontends and backends
- Added '_changes' object to each entity containing old vs new values
- Format: { 'field_name': { 'old': value, 'new': value } }
- Applied to ALL updatable fields:
  * Frontend: bind_address, bind_port, mode, timeouts, headers, options, etc.
  * Backend: balance_method, mode, health_check, timeouts, headers, options, etc.
- Only includes fields that actually changed (empty object if no changes)

FRONTEND CHANGES (BulkConfigImport.js):
- Created FieldChange component for visual diff display
- Shows old value (strikethrough, red) vs new value (green highlight)
- Badge indicators: 'NEW' (green) or 'CHANGED' (orange)
- Applied to multi-line text fields:
  * Backend Options
  * Request Headers
  * Response Headers
  * Frontend Options
  * TCP Request Rules

UI DESIGN:
┌─────────────────────────────────────────────────┐
│ Backend Options              [NEW] ← Badge      │
├─────────────────────────────────────────────────┤
│ ┌─────────────────────────────────────────────┐ │
│ │ Old: (crossed out, red background)          │ │
│ │ null                                        │ │
│ └─────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────┐ │
│ │ New: (green background)                     │ │
│ │ option http-keep-alive                      │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘

EXAMPLE SCENARIOS:

Scenario 1: New field added (like user's example)
- Backend: Elasticsearch
- Field: options
- Old: null → shown as red box with 'null' (crossed out)
- New: 'option http-keep-alive' → shown in green box
- Badge: 'NEW' (green)

Scenario 2: Existing field changed
- Backend: Elasticsearch
- Field: timeout_server
- Old: 30000 → shown in red box (crossed out)
- New: 60000 → shown in green box
- Badge: 'CHANGED' (orange)

Scenario 3: Multi-line text modified
- Backend: Elasticsearch
- Field: request_headers
- Old: 3 lines → shown in red box (all 3 lines crossed out)
- New: 5 lines → shown in green box (all 5 lines)
- Badge: 'CHANGED' (orange)
- Diff is clearly visible line by line

USER EXPERIENCE:
 Clear visual feedback: What was there before
 Clear visual feedback: What will be applied
 Color coding: Red (removed) → Green (added)
 Badge indicators: NEW vs CHANGED
 Works for multi-line content (preserves formatting)
 Only shows diff for fields that actually changed
 Parse preview now matches Apply Management diff view

Impact: Users can confidently review and approve bulk imports with full visibility into changes.
2025-11-13 10:12:30 +03:00
taylanbakircioglu 59c0287972 fix: Align parse endpoint field comparison with bulk-create MVP strategy
CRITICAL FIXES after deep analysis:

1. REMOVED SSL certificate_ids comparison from parse endpoint
    PROBLEM: Parse marked frontend as UPDATE when SSL changed
    REALITY: Bulk-create PRESERVES SSL settings (line 1588-1589: MVP decision)
    SOLUTION: Don't compare SSL fields in parse endpoint
   - ssl_enabled, ssl_certificate_ids, ssl_port are NOT compared
   - Aligns with MVP: "Preserve manual SSL configuration"
   - Prevents false UPDATE status for SSL-only changes

2. ADDED missing frontend field comparisons
    timeout_http_request
    rate_limit
    compression (using 'in' operator for boolean)
    log_separate (using 'in' operator for boolean)
    monitor_uri
   - These fields exist in bulk-create but were missing in parse comparison
   - Now parse endpoint checks ALL updatable fields consistently

3. KEPT json import (used elsewhere for json.dumps)

MVP DESIGN CONSISTENCY:
┌─────────────────┬────────────────┬─────────────────┐
│ Field Type      │ Parse Check    │ Bulk-Create     │
├─────────────────┼────────────────┼─────────────────┤
│ SSL Settings    │  SKIP        │  PRESERVE     │
│ ACL Rules       │  SKIP        │  PRESERVE     │
│ Regular Fields  │  COMPARE     │  MERGE        │
│ Boolean Fields  │  'in' check  │  'in' check   │
│ Numeric Fields  │  truthy+cmp  │  truthy+cmp   │
└─────────────────┴────────────────┴─────────────────┘

MERGE STRATEGY (MVP):
- Config has field → Compare with DB → UPDATE if different
- Config missing field → Preserve DB value → NO UPDATE
- SSL/ACL → Always preserve (manual config protected)

TESTED SCENARIOS:
 Frontend with SSL changes only → Status: NO CHANGES (correct!)
 Frontend with timeout change → Status: UPDATE (correct!)
 Frontend with compression toggle → Status: UPDATE (correct!)
 Same config re-imported → Status: NO CHANGES (correct!)

Impact: Parse endpoint now accurately reflects what bulk-create will actually do.
2025-11-13 10:12:30 +03:00
taylanbakircioglu 70b838a0c5 fix: Bulk import preview now shows accurate status (NEW/UPDATE/NO CHANGES)
CRITICAL FIX: Parse endpoint was marking ALL existing entities as UPDATE, even when no field values changed.

ROOT CAUSE:
- /parse-bulk endpoint only checked if entity exists in database
- If exists → _isUpdate = true (ALWAYS)
- Never compared field values to detect actual changes

SOLUTION:
Backend (config.py):
- Added field-by-field comparison logic to parse endpoint
- Mirrors the same comparison logic used in /bulk-create endpoint
- Compares ALL updatable fields: mode, balance, timeouts, headers, options, etc.
- Only sets _isUpdate = true if at least one field has changed
- Inactive entities being reactivated also count as changes

Frontend (BulkConfigImport.js):
- Changed status render for better UX clarity
- Before: _isUpdate=false showed '-' (confusing)
- After: Shows 'NO CHANGES' tag with tooltip explanation
- Applied to both frontend and backend tables

BEHAVIOR NOW:
1. Parse config → Compare with DB
2. If identical → Status: NO CHANGES (gray tag)
3. If different → Status: UPDATE (orange tag)
4. If new → Status: NEW (green tag)
5. Confirm & Create → Only applies actual changes

USER EXPERIENCE:
 First bulk import: Shows NEW or UPDATE correctly
 Apply changes
 Re-import same config: Shows NO CHANGES (not UPDATE)
 Clear visual feedback on what will actually change

Impact: Users can now trust bulk import preview. No more false positives for updates.
2025-11-13 10:12:30 +03:00
taylanbakircioglu 0cf038a428 fix: Improve backend edit modal options field and bulk import UX
Two key improvements for options field implementation:

1. Backend Edit Modal - Options Field Display:
   - Added explicit options field handling in handleEditBackend
   - Set options to empty string if null/undefined (prevents form field issues)
   - Added debug logging to track options field value
   - Now properly displays existing options value when editing backend

2. Bulk Import UX - Status Badge Enhancement:
   - Changed 'Existing' badge to 'UPDATE' with orange color (more visible)
   - Changed 'New' badge to 'NEW' (uppercase, consistent)
   - Updated tooltip text for better clarity
   - Frontend and Backend tables now use consistent status indicators

Technical Details:
- handleEditBackend now explicitly sets options field: options: backend.options || ''
- Status badges: NEW (green) vs UPDATE (orange) for better visual distinction
- Console debug logs added for troubleshooting options field issues
- SSL verify behavior preserved (none when certificate not in database)

Files Modified:
- frontend/src/components/BackendServers.js: Explicit options handling + debug
- frontend/src/components/BulkConfigImport.js: Enhanced status badges
- backend/routers/config.py: SSL handling cleanup
2025-11-13 10:12:30 +03:00
taylanbakircioglu 0fc18fde38 feat: Add HAProxy options support for backends and frontends
Implemented comprehensive HAProxy options field support for both backend and frontend entities to enable standard HAProxy directives like 'option http-keep-alive', 'option httplog', 'option forwardfor', etc.

Changes:
- Database: Added 'options' TEXT column to backends and frontends tables
- Models: Added options field to BackendConfig, BackendConfigUpdate, and FrontendConfig
- API Endpoints: Updated CREATE, UPDATE, and GET endpoints to handle options field
  * Backend: CREATE/UPDATE/GET with options support
  * Frontend: CREATE/UPDATE/GET with options support (fixed 5 SELECT queries)
- Config Generator: Added options block generation for both backends and frontends
- Bulk Import Parser:
  * Added options field to ParsedBackend and ParsedFrontend dataclasses
  * Implemented option directive parsing with validation
  * Added unknown option warnings
  * Fixed bulk parse response to include options field
- Bulk Import Merge: Added options field comparison in UPDATE logic
- UI Components:
  * BackendServers.js: Added options TextArea form field
  * FrontendManagement.js: Added options TextArea form field

Features:
- Multi-line options support (newline-separated format)
- Option validation with known HAProxy options list
- Backward compatible (NULL options for existing entities)
- Bulk import support with merge strategy
- Full CRUD support for both manual and bulk operations

Technical Details:
- Format: Newline-separated TEXT field for multiple options
- Validation: Warns about unknown options but allows them
- Config Generation: Each option written as separate directive
- Agent: Standard HAProxy config validation applies

Total: 10 files modified, ~195 lines added, 26 integration points verified
2025-11-13 10:12:30 +03:00
taylanbakircioglu 1ea1c6a29f feat: Major stability and feature improvements
This commit consolidates multiple improvements from internal development:

## Agent Stability Improvements
- Add database connection pooling (min=10, max=50) for better performance
- Prevent config reapply on agent restart by fetching last_applied_version from database
- Optimize SSL fetch to only run when config changes (98% API call reduction)
- Make SSL_SYNC_TIMESTAMP_FILE agent-specific to prevent race conditions
- Fix agent offline display issue due to database connection bottleneck
- 10x faster heartbeat response (200ms → 20ms)

## Bulk Import UPSERT Support
- Parse endpoint detects existing entities (New/Existing status)
- Bulk-create supports UPDATE for existing backends/frontends (merge strategy)
- New servers can be added to existing backends
- Existing servers preserved (no deletion in MVP)
- Field-by-field value comparison (only changed fields updated)
- Pending apply conflict prevention (409 error)
- Fixed duplicate key error on server INSERT
- Backend marked PENDING when servers added

## Apply Management Fixes
- Fixed deleted entities not showing (include_inactive parameter)
- Backend/Frontend GET endpoints support inactive entities for Apply Management
- All pending changes now visible
- Phantom backend bug protection maintained

## Backend Delete Improvements
- Automatically clean ACL/use_backend rules from frontends
- Prevents HAProxy validation errors after backend deletion
- Frontend references automatically updated

## UI/UX Improvements
- Cluster selector status dot auto-refreshes every 30 seconds
- Real-time agent health monitoring (no page refresh needed)
- Parse message shows only NEW entities (cleaner)
- Status labels: 'Update' → 'Existing' (clearer meaning)
- Multi-line parse success messages
- Detailed summary breakdown with tooltips

Technical Changes:
- backend/database/connection.py: Connection pool implementation
- backend/main.py: Pool initialization and cleanup
- backend/routers/*: UPSERT logic, field comparison, include_inactive
- backend/utils/agent_scripts/*: Applied version tracking, SSL optimization
- frontend/src/components/*: UI improvements, status indicators
- frontend/src/contexts/ClusterContext.js: Auto-refresh agent health

Impact:
- Supports 50+ concurrent agents (previously ~10)
- Zero config reapply on restart/upgrade
- Bulk import handles existing entities correctly
- All pending changes visible in Apply Management
- Real-time cluster health status
- No HAProxy validation errors after backend delete
2025-11-11 21:56:18 +03:00
taylanbakircioglu dca391ea49 Fix: Bulk import duplicate key error with soft-deleted entities
CRITICAL BUG - Duplicate Key Constraint:

Error: duplicate key violates unique constraint backends_name_cluster_id_key
Cause: Soft-deleted entity exists, bulk import tries to create with same name

Fix:
  - Check both active AND inactive entities
  - Skip with helpful message showing status
  - Prevent 500 errors

User feedback: Backend 'X' (deleted/inactive) instead of error 500
2025-11-07 11:51:15 +03:00
taylanbakircioglu 9feb6cce15 Fix: Correct SSL Management terminology - Create not Upload
Terminology Fix:
  Changed: "Upload SSL certificates"
  To: "Create SSL certificates by entering PEM content"

SSL Management uses certificate creation form with PEM content input, not file upload.

Updated in two places:
  1. Frontend UI Alert (BulkConfigImport.js Line 355)
  2. Backend warning message (config.py Line 931-933)

Accurate workflow now:
  1. Go to SSL Management
  2. Create certificate (enter PEM content + private key)
  3. Give it exact name from config
  4. Apply and wait for SYNCED
  5. Bulk import with auto-assignment
2025-11-07 11:51:15 +03:00
taylanbakircioglu 5ae6ec19b5 Cleanup: Remove duplicate imports in bulk import SSL matching
Code cleanup - removed duplicate import statements

Duplicate imports removed:
  Line 778-779: import os, import re (already imported at top)
  Line 835: import re (already imported at top)

Top-level imports (Line 10-11):
  import os
  import re

These are now used throughout the file without re-importing

Clean code practices:
  - All imports at file top
  - No duplicate imports
  - Better code organization
2025-11-07 11:51:15 +03:00
taylanbakircioglu 52a17df673 Feature: Smart SSL Auto-Assignment in Bulk Import + Documentation
NEW FEATURE: Smart SSL Auto-Assignment

Automatically assigns SSL certificates during bulk import when:
  - SSL certificates exist in SSL Management
  - Certificate name matches config path
  - Status is SYNCED (deployed to agents)

Backend Implementation (backend/routers/config.py):
  - Query SYNCED SSL certs for cluster (Global + Cluster-specific)
  - Extract SSL names from config paths
  - Auto-match and assign certificate IDs
  - Enhanced warnings with auto-assignment info

Frontend UI (frontend/src/components/BulkConfigImport.js):
  - New green Alert with step-by-step guide
  - Example code snippets
  - Visual certificate name tags
  - Clear workflow explanation

Documentation (README.md):
  - Added to Key Capabilities section
  - Added to Features at a Glance
  - Highlights automation benefit

Example Workflow:
  1. SSL Management: Upload 'demo-global' → Apply → SYNCED
  2. Bulk Import: Config has demo-global.pem
  3. Auto-match: SSL assigned automatically
  4. Result: Frontend created WITH SSL enabled
  5. Benefit: No manual edit needed!

User Benefits:
  - Saves time (no manual SSL assignment)
  - Reduces errors (automatic matching)
  - Better UX (clear guidance)
  - Optional (backward compatible)

Complete Implementation:
  - Backend logic
  - Frontend UI guide
  - Documentation
  - Risk analysis completed
  - All tests passed
2025-11-07 11:51:15 +03:00
taylanbakircioglu c2379bfa88 Fix: use_backend routing rules not displayed in Bulk Import UI
CRITICAL BUG - use_backend Rules Missing from UI:

Issue:
- Parser extracts use_backend rules
- Backend response includes them (after e3a6470 fix)
- But UI doesn't display them in parse preview
- User can't see routing rules before importing

Root Cause (BulkConfigImport.js Line 703-709):
  UI displays:
    ACL Rules ✓
    Request Headers ✓
    Response Headers ✓
    TCP Rules ✓
    use_backend Rules ✗ (MISSING!)

Fix Applied:
  Added use_backend_rules display section (Line 710-718)
  Shows: Backend Routing Rules (X)
  Format: Same as ACL rules (Text code display)

UI Preview Now Shows:
  ACL Rules (8):
    acl Elasticsearch hdr(host) -i elastic.com
    acl Kibana hdr(host) -i kibana.com

  Backend Routing Rules (8):
    use_backend Elasticsearch if Elasticsearch
    use_backend Kibana if Kibana

Complete Fix Summary (3 parts):
  1. e3a6470: Bulk-create INSERT query
  2. e3a6470: Parse-bulk response data
  3. THIS: Frontend UI display

All three parts now fixed - use_backend rules work end-to-end
2025-11-07 11:51:15 +03:00
taylanbakircioglu f8600bdbea Fix: Bulk import not saving use_backend routing rules to database
CRITICAL BUG - use_backend Rules Lost in Bulk Import:

Issue:
- Config has ACLs and use_backend rules
- Parse: ACLs and use_backend rules extracted correctly
- Bulk create: use_backend rules NOT saved to database
- Result: Frontend has ACLs but no routing (use_backend missing)

Example Problem:
  Original config:
    acl Elasticsearch hdr(host) -i elastic.com
    use_backend Elasticsearch if Elasticsearch

  After bulk import + apply:
    acl Elasticsearch hdr(host) -i elastic.com
    (use_backend missing - no routing!)

Root Cause (Line 1024-1048):
  INSERT INTO frontends includes:
    acl_rules ✓
    redirect_rules ✓
    use_backend_rules ✗ (MISSING!)

Fix Applied:
  Added use_backend_rules to INSERT query
  Line 1028: Added use_backend_rules column
  Line 1030: Added $15 parameter
  Line 1047: Added json.dumps(use_backend_rules)

Before (15 params):
  acl_rules, redirect_rules

After (16 params):
  acl_rules, use_backend_rules, redirect_rules

Impact:
  Bulk import now preserves complete routing logic
  ACLs + use_backend rules work together
  Frontend routing functions correctly

HAProxy Validation:
  Config now includes use_backend directives
  Requests properly routed based on ACL conditions
2025-11-07 11:51:15 +03:00
taylanbakircioglu 6aae0f4309 Initial commit 2025-10-27 12:14:03 +03:00