gsadmin/haproxy-openmanager
1
0
Fork 0
mirror of https://github.com/taylanbakircioglu/haproxy-openmanager.git synced 2026-09-19 17:15:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
200 Commits 10 Branches 25 Tags
v1.5.2
Commit Graph

6 Commits

Author SHA1 Message Date
taylanbakircioglu 02b1cb2bca feat: v1.5.0 — Site Wizard (Issue #14) + ACME Diagnostic Panel (Issue #13) 
Closes #13, Closes #14.

This release squashes the v1.4.0 → v1.5.0 development line. v1.4.0
shipped the ACME stability & enterprise audit (Issues #10/#11/#12).
v1.5.0 builds on that foundation with two co-equal headline features
plus a 22-round audit campaign hardening the prior configuration
surface. License remains MIT for v1.5.0 (relicense to AGPL-3.0
lands in v1.5.2).

------------------------------------------------------------------
HEADLINE FEATURE A — ACME Diagnostic Panel (Issue #13)
------------------------------------------------------------------
A live pre-flight + post-failure diagnostic surface for every ACME
order, reachable from the ACME Automation page. The panel exists
to make ACME failures legible to operators who do NOT have shell
access to the API host.

Endpoints (`backend/routers/acme_diagnostics.py`):
  POST /api/letsencrypt/orders/{order_id}/diagnostics
       Run the full 5-check suite (DNS / port-80 / routing /
       account / agents) and humanize the order's `error_detail`
       (>=11 RFC-8555 problem types, backwards compatible with
       legacy plain-string failures).
  POST /api/letsencrypt/orders/{order_id}/diagnostics/
                                {check_id}/rerun
       Re-run a single check in place — used by the "Re-run"
       button on every row of the modal's pre-flight table.
  GET  /api/letsencrypt/orders/{order_id}/events
       Merged event timeline combining the typed
       `acme_order_events` rows with correlated
       `user_activity_logs` entries (resource_type =
       'letsencrypt_order' AND resource_id = order_id). The
       diagnostic modal auto-tails this timeline every 5 seconds
       while open.

Service-level checks (`backend/services/acme_diagnostics.py`):
  * DNS resolution via stdlib socket.gethostbyname_ex through
    run_in_executor (intentionally avoiding an aiodns runtime
    dep for v1.5.0).
  * Port-80 HEAD probe, target locked to the order's domains,
    success on HTTP 200 OR 404, warns on egress timeout
    (corp egress policies routinely blackhole outbound 80 —
    fail-hard would be too noisy).
  * SSRF guard: probe refuses non-public IPs and surfaces the
    skip in the diagnostic result; IPv4-mapped IPv6 normalisation
    closes the `::ffff:169.254.169.254` cloud-metadata vector.
  * HAProxy routing presence check: matches the order's
    cluster_ids to a port-80 HTTP frontend.
  * ACME account validity check against `letsencrypt_accounts`.
  * Agent presence check (>=1 active agent in target cluster).
  * Every sub-check wrapped in a wall-clock timeout to bound
    impact on the API event loop.

RBAC: ssl.read for run, ssl.read for events. Per-user 5/min rate
limit on both run and rerun, backed by the (user_id, action,
created_at DESC) composite index.

Frontend (`frontend/src/components/ACMEAutomation.js`):
  * "Diagnose" button on every order row + the existing
    "stuck order" warning row.
  * Modal with two tabs:
    - Pre-flight Checks (Antd Table with status pills + Re-run
      buttons + humanized error banner)
    - Event Log (Antd Timeline with auto-tail polling, scroll-
      to-bottom, pause-on-hover)
  * Correlation IDs surfaced in error banners and individual
    check fail details for backend-log lookup.

------------------------------------------------------------------
HEADLINE FEATURE B — Site Setup Wizard (Issue #14)
------------------------------------------------------------------
A single guided flow that creates a Backend + Servers + HTTP
Frontend (and optional HTTPS Frontend) in one atomic transaction.

Endpoints (`backend/routers/site_wizard.py`):
  POST /api/site-wizard/preview     — diff-preview the changeset
  POST /api/site-wizard/create      — atomic execute
  POST /api/site-wizard/reject      — clean rollback (including
                                       any wizard_staged ACME
                                       orders)
  GET  /api/site-wizard/drafts      — draft persistence
  PUT  /api/site-wizard/drafts/{id} — save/update
  DELETE /api/site-wizard/drafts/{id}

Feature surface:
  * One screen captures both backend (mode + servers) AND
    frontend (http + optional https + SSL mode) inputs.
  * SSL modes: ACME (new order, HTTP-01 only for v1.5.0),
    Upload (existing PEM), Existing (link to a stored cert),
    or None.
  * ACME-staged path: wizard_staged_until watermark on the
    `letsencrypt_orders` row defers finalisation until agent
    confirmation; per-mode reject cleanly cancels and rolls
    back the staged order.
  * Live diff preview against the cluster's current generated
    config (renderer-evolution noise stripped — track-sc<N>
    dedup, per-server cookie strip, defaults-cookie
    inheritance, listen-block flattening).
  * Draft persistence with PEM stripped at save time (private
    keys never round-trip through the drafts table).
  * Per-cluster multi-tenancy: drafts and wizard_staged orders
    are isolated to the creating user's cluster scope.

Frontend (`frontend/src/components/SiteWizard.js`):
  * 4-step Antd Steps flow: Backend → Frontend → SSL → Review.
  * Render the live diff preview inline before commit.
  * Antd Form-level validation mirrors backend Pydantic
    validators (numeric bounds, HAProxy reserved keywords, ALPN
    consistency, IPv6 scope-id, domain regex, server name
    dedup).

------------------------------------------------------------------
AUDIT CAMPAIGN — Rounds 1 → 22 (Bulgu #1 → #82)
------------------------------------------------------------------
v1.5.0 includes 22 adversarial review passes. Each round produced
its own commit set in the corporate development line; this squash
collapses those into the v1.5.0 release artefact. Highlights:

  Round 1-4   Site Wizard core: dry-run parity, single-line
              value injection guard, ACL -f pattern-file block,
              SSL parity, timeout regex, form-state pin.
  Round 5-7   defaults-cookie inheritance, server-named-cookie
              guard, fe/be mode mismatch, duplicate server
              names, health_check_uri + server_address
              validators.
  Round 8-10  cookie_name / cookie_options newline-injection
              guard, dry-run parity (round 9), TCP-mode HTTP-only
              feature blockers.
  Round 11    SSL name path traversal + health-check >= 1.
  Round 12-13 SSL & ACME deep dive (Bulgu #23-#32).
  Round 14    single-line value injection (Bulgu #33).
  Round 15-17 ACME multi-tenant UX, numeric bounds, HAProxy
              reserved keywords, ALPN/TLS consistency,
              all-backup, multi-domain & multi-user enterprise
              edges, drain/HSTS/post-completion (Bulgu
              #34-#53).
  Round 18-21 concurrency, agent state, TCP-mode HTTP-only,
              list size caps, IPv6 scope-id, preview account
              validation, TCP backend + balance uri reject
              (Bulgu #54-#61).
  Round 22    FE error visibility + 3x stale-data lockouts,
              referential integrity + cascade safety,
              authentication & authorization, multi-cluster
              isolation, apply_pending_changes concurrency,
              script injection + bulk import multi-tenancy,
              prefix-stripped signature comparison
              (Bulgu #62-#82).

------------------------------------------------------------------
NO CORPORATE-SPECIFIC ARTIFACTS
------------------------------------------------------------------
This squash deliberately sanitises corporate hostnames, container
registry references, and TLS secret names into generic
placeholders (`your-registry.example.com/your-org`,
`haproxy-openmanager*.example.com`, `wildcard-tls`,
`taylanbakircioglu/haproxy-openmanager-*`) so the public artefact
contains no internal infrastructure detail. Pilot / development
history that retained those values stays in the corporate fork
and is NOT part of this commit.
 2026-05-14 00:04:19 +03:00
taylanbakircioglu 07942a82e8 feat: ACME stability & enterprise audit (v1.4.0) — fixes #10 #11 #12 
Issue #10 — Silent timezone failure on ACME certificate save
- Normalize tz-aware expiry_date to UTC tz-naive before INSERT/UPDATE in
  ssl_certificates (TIMESTAMP WITHOUT TIME ZONE) — restores ACME download path

Issue #11 — Duplicate _acme_challenge_backend in generated config
- Generator guard: skip auto-append when backend already rendered
- Agent-sync filter: _should_sync_backend() drops system-managed backends and
  detaches their server rows to prevent orphans
- Restore filter: IGNORED_BACKENDS skips reserved names during cluster restore
- Parser warning: reserved_backend_names blocks accidental manual import
- Cleanup migration: removes orphan rows + cascading server entries (idempotent)

Issue #12 — Validated orders required manual completion
- New 60s background task complete_pending_acme_orders, flag-independent,
  multi-replica safe via FOR UPDATE SKIP LOCKED + 30s updated_at watermark
- Per-order pg_advisory_lock(0x41434D45, order_id) serializes UI-Complete and
  auto-task races; idempotency guard returns existing certificate cleanly
- retry_order endpoint reports in_progress: true within 30s window so the UI
  surfaces an info toast instead of duplicating CA requests
- ACMEAutomation surfaces stuck orders (status=valid && !ssl_certificate_id)
  with a one-click Complete action and Cancel fallback; conditional 30s polling

Other hardening
- ACME state machine error_detail persisted as structured JSON across challenge,
  finalize, download stages for actionable post-mortems
- CertificateRequest Pydantic model: domain regex + min_length/max_length and
  cluster_ids defaulting to all ACME-enabled clusters when "global" is selected
- Renewal cluster fallback now requires acme_enabled=TRUE in addition to active
- _complete_certificate preserves manual cluster assignments on renewal,
  surfaces cluster_errors, raises explicit error on missing private key
- Audit logging covers acme_certificate_requested/revoked, ca_chain_imported,
  account created/deactivated/purged, order retried/cancelled
- Settings UI exposes acme.staging_url_override for private test CAs (Pebble)
- Schema additions: acme_challenges.attempts (default 0) and last_attempt_at,
  index idx_letsencrypt_orders_status_updated; all migrations idempotent

Tests (41/41 passing)
- test_acme_expiry_normalize, test_acme_duplicate_backend,
  test_acme_state_machine, test_acme_pydantic_validation,
  test_acme_audit_logging, test_acme_concurrency

CI / packaging
- docker-build.yml reads version.json and pushes additional product-version
  tag (e.g. 1.4.0) alongside latest and timestamp build id

Closes #10
Closes #11
Closes #12
 2026-05-06 23:53:43 +03:00
Taylan Bakırcıoğlu a739dd0f95 PRODUCTION FIX v2: Direct JSON Sanitization in Heartbeat Endpoint 
CRITICAL ISSUE:
Previous middleware approach failed - Starlette middleware cannot
reliably modify request body after it's consumed by FastAPI.

NEW APPROACH - ENDPOINT-LEVEL SANITIZATION:
Moved JSON sanitization directly into agent heartbeat endpoint for
guaranteed execution before Pydantic validation.

ROOT CAUSE CONFIRMED:
Agent daemon mode sends: "server_statuses": ,
This is INVALID JSON (empty value before comma)
Result: JSON decode error at position 322 -> agents stuck offline

SOLUTION IMPLEMENTATION:
Modified: backend/routers/agent.py
- Read raw request body BEFORE Pydantic processing
- Apply 3 regex fixes:
  1. "field": , -> "field": null,
  2. "field": } -> "field": null}
  3. {field,} -> {field}
- Parse sanitized JSON manually
- Create AgentHeartbeat from clean dict
- Continue with normal heartbeat flow

BENEFITS:
✓ NO AGENT SCRIPT CHANGES (production safe)
✓ Guaranteed execution (not middleware dependent)
✓ Detailed logging of sanitization
✓ Graceful error handling
✓ Backward compatible with all agents
✓ Zero impact on valid JSON

LOGGING:
INFO: "Sanitized malformed JSON from agent 'demo-agent1'"
DEBUG: Shows before/after JSON (first 300 chars)

PRODUCTION IMPACT:
- demo-agent1 & agent3 will go online immediately
- No agent restart required
- No agent script update required
- Self-healing for future similar issues

TESTING:
Deploy backend -> Watch logs for:
"Sanitized malformed JSON from agent"

Removed:
- backend/middleware/json_sanitizer.py (approach failed)

This direct approach guarantees the fix executes BEFORE
FastAPI/Pydantic validation, solving the agent offline issue.
 2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu dbf5b8d688 PRODUCTION FIX: JSON Sanitizer Middleware for Malformed Agent Heartbeats 
CRITICAL FIX - No Agent Script Changes Required

PROBLEM IDENTIFIED:
- demo-agent1/agent3 sending malformed JSON
- Error: server_statuses: , (empty value before comma)
- Invalid JSON syntax causing HTTP 422 validation errors
- Agents stuck offline

ROOT CAUSE:
Agent daemon mode get_server_statuses() returns empty string when
HAProxy stats socket unavailable, resulting in: "server_statuses": ,

SOLUTION - BACKEND MIDDLEWARE (Production Safe):
Created JSONSanitizerMiddleware that automatically fixes common JSON
errors BEFORE FastAPI parses request body:

1. Empty values before comma/brace: field: , -> field: null,
2. Trailing commas: {field: value,} -> {field: value}
3. Only processes /api/agents/heartbeat endpoint
4. Logs what was fixed for auditing

BENEFITS:
- NO AGENT SCRIPT CHANGES (production safe)
- NO AGENT UPGRADE REQUIRED
- Backward compatible with all agent versions
- Zero impact on valid JSON
- Self-healing for future similar issues
- Detailed logging for monitoring

MIDDLEWARE ORDER:
PerformanceMonitoring -> RequestLogging -> JSONSanitizer -> ActivityLog -> CORS

HOW IT WORKS:
1. Intercepts POST /api/agents/heartbeat
2. Reads raw body before FastAPI
3. Applies regex fixes for known patterns
4. Replaces request body with sanitized version
5. FastAPI receives valid JSON

TESTING:
Before: {"server_statuses": ,"system_info": {...}}
After:  {"server_statuses": null,"system_info": {...}}

Result: Pydantic validation passes, agent goes online

This middleware approach is MUCH safer than deploying agent script
changes to production servers.
 2025-11-18 21:58:05 +03:00
Taylan Bakırcıoğlu a6b223c0e7 CRITICAL FIX: Enhanced Error Handling for Agent Heartbeat JSON Parse Errors 
PRODUCTION STABILITY FIX - Detailed Logging for Malformed Agent Payloads

PROBLEM:
- demo-agent1 and demo-agent2 sending malformed JSON
- Error: JSON decode error at body position 322
- No visibility into WHAT is malformed or WHY
- Impossible to debug without raw payload inspection

ROOT CAUSE:
- FastAPI consumes request body before error handler
- Pydantic validation fails but does not show raw input
- Agent script might be generating invalid JSON
- No logging of actual problematic payload

SOLUTION - ENHANCED ERROR HANDLING:

1. MIDDLEWARE ENHANCEMENT (error_handler.py):
   - Extract RAW body in validation error handler
   - Parse agent name from JSON (even if malformed)
   - Log first 500 chars of problematic payload
   - Add body size to error details
   - Special handling for /heartbeat endpoint
   - Detailed logging for json_invalid errors

2. HEARTBEAT ENDPOINT (agent.py):
   - Added Request parameter for raw body access
   - Enhanced docstring with troubleshooting info

BENEFITS:
- Instant visibility into malformed JSON
- Agent name logged even on parse failure
- Exact payload position + preview
- No performance impact (only on errors)
- Backward compatible (does not change API)

NEXT STEPS (After Deploy):
1. Check logs for CRITICAL JSON PARSE ERROR
2. Identify exact field causing parse failure
3. Fix agent script if needed
4. Or fix backend to be more tolerant

This enables root cause analysis without SSH access to agent servers
 2025-11-18 21:58:05 +03:00
taylanbakircioglu 6aae0f4309 Initial commit 2025-10-27 12:14:03 +03:00