mirror of
https://github.com/taylanbakircioglu/haproxy-openmanager.git
synced 2026-09-16 15:45:11 +00:00
main
90 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 taylanbakircioglu
|
0ee227363e |
fix(agent): adoption cannot hide a node; Config Import on fresh installs (v1.11.1)
Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.
DISCOVERY REPORTS THAT NEVER REACHED THE SERVER
The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.
- the report is cached only on a 2xx;
- GET /agents/{name}/keepalived-config now reports whether the server actually
holds a discovery for that agent, and the cache may only suppress while it
says yes - which is what lets nodes stuck from earlier releases recover on
their own, with nobody touching them;
- the flag is parsed with has() + tostring, not `// empty`: jq's alternative
operator returns the alternative for **false** as well as null, so the naive
form could not tell "no record" from "older backend" and the recovery would
have been completely inert;
- a 400/413/422 records the refusal so identical bytes are not re-posted
forever - 4xx and 5xx agent calls are never sampled out of the request log,
so an unattended loop would write a row carrying the whole config every
cycle - while 401 and 404 keep retrying, because here they mean a token
rotation or an agent row briefly absent, not a bad payload;
- the CLEAR path had the same exit-code defect, where it left a stale row
offering a managed node for adoption with nothing to ever retry it.
CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS
check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.
DETERMINISTIC CONFIG PATH
A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.
Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.
Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.
|
||
|
taylanbakircioglu
|
fbe223250b |
perf(requestlog): gate the embedded-secret scan behind a substring pre-check
The auth_pass / stats-auth / userlist / URI patterns added by the redaction
fixes on this branch run on the writer task, on every string value of every
captured body, so their cost is paid per row forever. Measured, they were:
writer-side redaction, before the redaction fixes 12.2 us/row
writer-side redaction, with them 359.1 us/row
A 29x regression, and none of it was spent matching anything - almost every
body contains none of these keywords. Profiling the four patterns on an 8 KB
config body, 300 iterations:
one combined alternation, IGNORECASE, \b-anchored 308.2 us
the same four run separately (sum) 219.7 us
text.lower() once + four substring pre-checks 5.3 us
`\b` and IGNORECASE each defeat the regex engine's literal-prefix scan, so
every alphanumeric position in 8 KB became a candidate start and the engine
walked the whole body four times to find nothing. Isolated, `auth_pass` costs
27.5 us with IGNORECASE and 2.4 us without.
Split the alternation and gate each pattern behind a substring test on one
lowercased copy. `str.lower()` and `in` are C-level scans; a pattern now runs
only when its keyword is actually present, and then on text that genuinely
contains it. Cost becomes O(total string bytes) instead of O(bytes x patterns).
The URI pattern also drops IGNORECASE and `\b` - its character class already
covers both cases, and `://` gives the engine a literal to scan for.
writer-side redaction, after 18.8 us/row
= 3.4s of CPU/day
at 180 000 rows/day
6.6 us/row over the pre-fix baseline, for four secret classes that were
previously written to the table in cleartext.
The pre-checks are on the lowercased copy, so the patterns stay IGNORECASE: the
marker may well have been `AUTH_PASS` in the original. Request-path cost is
unchanged at 27.7 us p50 - none of this ever ran there.
All 123 redaction and payload tests still pass, so the behaviour is identical;
only the path to it is cheaper.
|
||
|
taylanbakircioglu
|
bd4a50943f |
fix(requestlog): bound queue memory, and stop the UI reporting things it cannot know
Three hardening fixes with the same shape: a number that was true under the
defaults and untrue at the edges.
1. QUEUE MEMORY WAS AN OPERATOR SETTING, NOT A LIMIT.
The queue was bounded by ROW COUNT only, and how much a row weighs is
`requestlog.max_body_bytes` - editable from Settings, documented ceiling 256 KB,
and a row can hold that twice (request + response). Measured on the real
dataclass with distinct buffers per row:
defaults, 2 000 rows x 8 KB 33.9 MiB 3.3% of the 1 GiB pod limit
max_body_bytes at its 256 KB ceiling 1003 MiB at the pod limit
REQUEST_LOG_QUEUE_MAX at its ceiling 1695 MiB over the pod limit
Both are reachable from in-range, documented values, and the drop warning
advised "raise REQUEST_LOG_QUEUE_MAX" - so following the tool's own advice on a
busy install could OOM the worker. REQUEST_LOG_QUEUE_MAX_BYTES (default 64 MiB)
now caps the queue in bytes as well as in rows, whichever binds first, released
as rows drain. Verified: with max_body_bytes at 256 KB the queue holds 7.5 MiB
against an 8 MiB budget where it would otherwise have held 1003 MiB, and it
accepts rows again as soon as the writer drains it. The warning text now names
the setting that actually helps.
2. SINK COUNTERS ARE PER WORKER AND DID NOT SAY SO.
The sink is a module global, so with UVICORN_WORKERS > 1 each process has its
own queue and its own counters, and `GET /api/request-logs/stats` reports
whichever worker happened to serve the request. The feature is sold on "a
saturated logger drops rows visibly"; at 4 workers the visible number was a
quarter of the truth. Labelled `"scope": "this worker only"` rather than
aggregated - there is no cross-process channel here, and a number that looks
fleet-wide but is not is worse than one that admits its scope.
3. AN EMPTY EXCLUDE LIST IS NOT APPLIED AS "LOG EVERYTHING".
normalize_exclude_paths() falls back to the shipped defaults when the list comes
out empty, which is the right call - it keeps the log viewer and the raw-body
heartbeat endpoint excluded - but the UI kept displaying the empty list the
operator typed, so the form showed a policy that was not in effect. The save
handler now re-applies whatever the server actually stored (which also surfaces
server-side clamping of every numeric field) and says plainly that the defaults
were restored.
|
||
|
taylanbakircioglu
|
bec0613ae5 |
fix(requestlog): give each background pass its own correlation id
Outbound rows from background work fell back to `bg:<asyncio task name>`.
Nothing in main.py passes `name=` to `create_task`, so every loop keeps one
auto-assigned name - `Task-5` - for its entire life, and every call it ever
makes is written with that same `request_id`. Measured: fifteen ACME calls
across five renewal ticks came out as one id.
That is not a cosmetic grouping problem. `GET /api/request-logs/{id}` returns
every other row sharing the id as `related`, up to 100, and the UI presents
that list as "the calls this request triggered" - it is the feature's headline.
An operator opening a failed renewal was therefore shown up to a hundred
unrelated calls, possibly spanning days, labelled as the trace of the one they
were reading. In a forensics tool a confidently wrong trace is worse than no
trace. Task numbers are reused across restarts too, so `bg:Task-5` could mean a
different loop after a redeploy.
begin_background_trace(label) opens `bg:<label>:<uuid12>` for one iteration and
is called at the top of the three loops that make outbound calls:
complete_pending_acme_orders, check_letsencrypt_renewals, monitor_agent_status.
The loop task is dedicated, so the next iteration overwrites it and there is
nothing to reset.
The fallback for background code that has not been wrapped now mints a unique
id per call instead of reusing the task name. That errs toward too little
grouping rather than too much: a row that stands alone is honest, a row falsely
grouped with a hundred others is not.
Verified: five ticks of three calls produce five distinct ids with the three
calls of each tick sharing one, and four calls from an unwrapped task produce
four distinct ids.
|
||
|
taylanbakircioglu
|
4e2d936c27 |
fix(requestlog): stop the table size from scaling with fleet size
The row rate of `request_logs` was a function of how many nodes are installed,
not of what anyone did. Counted from the agent loop in linux_install.sh, each
agent's 30s cycle issues three logged calls - config, pending-requests,
upgrade-status (the heartbeat is already on the default exclude list) - plus
keepalived-config and keepalived-status every fifth cycle. That is ~9 800
rows/day per agent, essentially all of them 200s meaning "nothing changed".
Measured on PostgreSQL 15 against the real DDL and all nine indexes, at 2 424
bytes/row:
20 agents ~196k rows/day 453 MB/day row cap reached in 2.5 days
200 agents ~2.0M rows/day 4.4 GB/day row cap reached in 6 hours
500 agents ~4.9M rows/day 11 GB/day row cap reached in 2 hours
The cap holds, so nothing runs away - but it holds by deleting, and what it
deletes is everything else. The shipped policy says 7 days of successes and 30
days of failures; on a 200-node fleet it delivers about six HOURS of both. The
forensic record the feature exists for is evicted by polling noise, and the
larger the installation the less history it keeps.
`requestlog.capture_agent_success`, default FALSE: a SUCCESSFUL inbound call
from an agent is not recorded. Failures always are, whatever the flag says -
they are what an operator needs and they are rare, so they cost nothing. With
this the table's size follows operator activity, and adding nodes does not
shorten anyone's retention.
Agent traffic is identified by header only, no database round-trip on the hot
path: the installed agent sends `X-API-Key` and never `Authorization`, the UI
sends a JWT and never an agent key. `generate-install-script`, the one endpoint
that accepts either, classifies correctly under the same rule - an operator
generating a script sends Authorization, a self-upgrading agent sends only the
key. The result is stored in the existing `target` column, which already means
"who was on the other end" for outbound rows and now means the same for inbound
ones, so no schema change and the existing target index applies.
Second half, and the reason this is one commit: `operator` holds
`requestlog.read` because, per the migration that grants it, "operators debug
failing applies and ACME orders". They could not. An apply fails on the NODE,
and the node reports that over its own API key, so the row carrying the
diagnosis has `user_id IS NULL` - and own-rows-only scoping hid it from exactly
the role the grant was written for. Scoping now admits agent rows alongside the
caller's own. Deliberately keyed on `target = 'agent'` rather than `user_id IS
NULL`: anonymous traffic is not agent traffic, so failed logins and their
usernames, and unauthenticated probes, stay admin-only.
Verified end to end through the real middleware: a successful agent poll is
dropped, a 422 from config-validation-failed is kept, operator and anonymous
calls are unaffected, and flipping the setting on restores the old behaviour.
|
||
|
taylanbakircioglu
|
82e6fe3f9c |
fix(requestlog): mask HAProxy credentials in uploaded config bodies
This application never RENDERS a credential into a haproxy.cfg - grepping the
generator and every sample config for `stats auth`, `userlist` and
`insecure-password` returns nothing - so none of our own output is at risk. The
exposure comes from the other direction: the agent uploads the node's REAL
on-disk file.
linux_install.sh: config_content=$(cat "$config_path")
-> POST /api/configuration/agents/{n}/config-response
and `POST /api/config/validate` plus the bulk import take whatever the operator
pastes. A production haproxy.cfg routinely carries `stats auth admin:<password>`
and a `userlist` block, so what is captured is credential-bearing even though
what we generate is not.
Two patterns, folded into the existing single-pass alternation:
stats auth admin:S3cr3t -> stats auth admin:********
user ops password $6$... -> user ops password ********
user dev insecure-password Hunter2 -> user dev insecure-password ********
The username on `stats auth` is deliberately kept: an operator debugging a 401
still needs to know WHICH account it was about.
The load-bearing detail is where a value ENDS. A config upload is routinely
larger than the 8 KB capture cap, so for exactly these payloads the common case
is not the parsed body - it is the truncated `{"_raw": ...}` fallback, where the
line breaks are still the two-character escape `\n` rather than real newlines.
A "rest of the line" match that does not know that runs past every apparent
line break: measured on a 17 KB upload, `auth_pass ...` masked the entire
remainder of the captured string. No leak, but the row is then worthless. Every
value pattern here stops at a real newline OR at a literal backslash-n, so the
same three credentials are masked and the surrounding config stays readable in
both forms. Both paths are verified.
Anchoring is to HAProxy keyword syntax, not to the bare word "password", so
ordinary prose survives: "invalid password format" and "the password must be 8
chars" are untouched. One accepted false positive: "user admin password reset
requested" masks the word "reset", because it is indistinguishable from a
userlist line without parsing the file. That is the same trade the existing
REDACT_CONTAINS entry for `token` already makes - a blanked word costs a little
readability, an unmasked credential costs a credential.
|
||
|
taylanbakircioglu
|
190d45fe09 |
fix(requestlog): scrub credentials carried inside URI values
`POST /api/mfa/enroll` returns the new TOTP secret twice: once as `secret`,
which redaction already caught, and once inside `otpauth_uri` as a query
parameter, which it did not. Blanking one field while the same value sits three
keys away in cleartext is not redaction.
before: {"secret": "***REDACTED***",
"otpauth_uri": "otpauth://totp/OpenManager:admin?secret=JBSWY3DP..."}
after: {"secret": "***REDACTED***",
"otpauth_uri": "otpauth://totp/OpenManager:admin?secret=***REDACTED***"}
routers/mfa.py states the rule this restores: its own activity-log call records
`{"secret_len": len(secret_plain)}` with the comment "NEVER log the secret
itself".
The fix is not otpauth-specific. Any URI found in any captured string value now
goes through scrub_url, which was already written for exactly this and was only
ever pointed at the request line. That also covers:
* userinfo credentials - `https://user:pass@host/path` keeps only the host;
* any query-string credential in a body value or an error message, using the
same is_secret_key rules as the request line, so `?token=`, `?api_key=`,
`?password=` are all handled without naming them again here;
* fragments, which are dropped - they never reach a server and can carry
tokens.
Folded into the existing alternation rather than added as a second pass, so a
captured body is still scanned once. Two guards keep it from doing harm: a URI
with neither `?` nor `@` is returned untouched rather than rebuilt, and if
scrub_url reports a parse failure the original text is kept - inside a larger
string its placeholder would corrupt the surrounding sentence.
Verified: the TOTP secret and a userinfo password are removed, an in-text URL
inside an error message is scrubbed in place, and two innocent ACME URLs
(directory_url, account_url) come through byte-identical.
Also withdrawn here: the review flagged `POST /api/agents/generate-install-script`
as leaking the agent API key through its `script` field. That was wrong. The
finding was built from the endpoint's docstring EXAMPLE (routers/agent.py:497),
which shows an `api_key` field and an `API_KEY="agt_..."` line; the handler's
actual return is {script, platform, cluster_id, filename} and the template
substitution map has no token placeholder. The agent token reaches a node by a
different path entirely, and is issued as `api_key` (routers/security.py:144),
which REDACT_CONTAINS already covers. No change was needed and none is made.
|
||
|
taylanbakircioglu
|
c50408026b |
fix(requestlog): keep the VRRP password out of the request log
Measured against the redaction module from the previous commit, using the real
field names this codebase uses. Three paths wrote the keepalived `auth_pass`
secret to `request_logs` in cleartext:
POST/PUT /api/vip request body, `auth_pass` field
GET /api/agents/{n}/keepalived-config response, keepalived.config_content
POST /api/agents/{n}/keepalived-discovery request body, config_content
The middle one is the worst: the agent polls it on the SSL cadence, so the
secret was re-written to the audit table roughly 576 times a day per member
node.
Two independent holes, closed independently:
1. KEY NAME. `auth_pass` normalizes to "authpass". "password" is not a
substring of it, and the bare "auth" entry in REDACT_EXACT is an exact
match, not a prefix - so nothing in either set matched and the field was
kept verbatim. "authpass" is now in REDACT_EXACT.
2. EMBEDDED IN A CONFIG BLOB. The two agent endpoints do not carry the secret
as its own field at all; they carry a whole keepalived.conf as one string
under `config_content`, with `auth_pass <secret>` on a line inside it. No
key-name rule can see that, and the value-shape guards do not either: it is
neither a PEM block nor a JWT. Added scrub_embedded_secrets(), one compiled
alternation applied in a single pass per string value, masking the whole
remainder of the line so a secret containing whitespace cannot partially
leak - the same pattern and the same reasoning as routers/vip.py's
_redact_secret and routers/agent.py's _AUTH_PASS_MASK_RE.
It runs BEFORE the _MAX_STRING truncation, not after: `auth_pass` sits in the
first few hundred bytes of a rendered keepalived.conf, so letting the cut
"handle" it would be relying on where the secret happens to fall in the file.
This restores an invariant the codebase already states and already enforces
elsewhere. routers/vip.py: "the secret never leaves the server in cleartext ...
only the at-rest Fernet token and the agent-delivery endpoint ever see the real
value". The discovery handler in routers/agent.py goes to three separate
lengths to uphold it - it pops auth_pass out of the parsed analysis, replaces it
with a has_auth_pass boolean, Fernet-encrypts the secret into its own column,
and stores only a masked copy as vip_discoveries.raw_config_masked. Capturing
the request that produced all that, unmasked, put the plaintext right back next
to it.
Verified: the three cases above now redact, and the ten cases that already
worked (login password, JWTs, PEM private keys under any key name, DNS provider
credentials, ACME JWS) are unchanged.
|
||
|
taylanbakircioglu
|
4c84596215 |
feat(logging): unified request/response log with configurable retention
Applies PR #59 by Mustafa Ulukaya (github.com/taylanbakircioglu/haproxy-openmanager/pull/59,
head
|
||
|
taylanbakircioglu
|
1d4e4286af |
fix(agent): keep acknowledging once converged, so a lost report self-heals (v1.10.14)
The deploy report is the server's only evidence that a member node applied its keepalived.conf, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return every cycle and never reported again, so a single lost report - a backend restart, a 5xx, a network blip - left the VIP reading SYNCING with an empty "Last ack" forever while the node was demonstrably running the right config. Nothing would ever reconcile the two; the only escape was to change the rendered config so the agent wrote it again, which means touching a live VIP to fix a display problem. The agent now re-asserts its state on that path too: one request per node per poll cycle (~2.5 min), nothing written, keepalived not reloaded. This gap dates from the original HA/VIP work rather than this release series; it only became visible when acknowledgements were dropped for an unrelated reason. A test pins that both daemon copies report BEFORE the early return, since placing it after would silently restore the old behaviour. Verified end to end on a real HA pair: discovery, instance-based adoption of both nodes, PENDING, Apply, agent pull, the validation gate, the hash-pinned takeover, the acknowledgement, and retirement of the one-shot authorisation. |
||
|
taylanbakircioglu
|
0eb587dfa8 |
fix: keepalived validation gate and deploy acknowledgements (v1.10.13)
Two agent-side fixes found while taking the VIP adoption flow through a real
HA pair, released together.
1. A VALID CONFIG WAS REJECTED BY ITS OWN WARNING (v1.10.12)
Before writing a rendered keepalived.conf the agent validates it with
keepalived -t and, on failure, keeps the running config and does not restart
keepalived. That fail-safe is right, but it treated ANY non-zero exit as
invalid - and keepalived's config-test exit code does not separate fatal from
benign. Measured on 2.2.8:
clean config ..................... 0
auth_pass longer than 8 chars .... 5 "Truncating auth_pass to 8 characters"
missing '}' ...................... 5 "There are 1 missing '}'s"
unknown keyword .................. 5 "Unknown keyword '...'"
script without script_security ... 6 "SECURITY VIOLATION ..."
Exit 5 covers both a harmless truncation and a broken file, so a VRRP password
over eight characters was enough to block every apply - including on a node
whose own running config emits the same warning and had been serving the VIP
for weeks. Accepting exit 5 would have accepted broken configs, so the gate now
judges the OUTPUT: known-benign messages are dropped and anything remaining
still fails. It fails CLOSED - an unrecognised message, or a non-zero exit with
no readable output at all, is fatal - and the filter is an allowlist, never a
denylist. The agent also reports what keepalived said, in its log and in the
status the HA/VIP page shows; discarding it left a correct refusal that nobody
could act on.
2. EVERY DEPLOY ACKNOWLEDGEMENT WAS DROPPED
The takeover-retirement clause on POST /agents/{name}/keepalived-status reused
one placeholder for both the assignment `last_deploy_hash=$n` and the
comparison inside its CASE. PostgreSQL types a placeholder per USE, so it came
out as text in one and character varying in the other and asyncpg rejected the
statement with AmbiguousParameterError. The whole UPDATE never ran, so no
member recorded an acknowledgement: VIPs sat at SYNCING with an empty Last ack
while the nodes were verifiably running the config, and teardown acks were lost
the same way. The hash now has its own placeholder, compared only against the
column.
Verified against real keepalived and a real PostgreSQL rather than by
inspection, including on busybox and bash 3.2, and a test asserts every $n in
those statements is bound exactly once.
|
||
|
taylanbakircioglu
|
a4c74f2a52 |
Merge pull request #60 from mustafaulukaya/fix/acme-http01-split-deployment
HTTP-01 challenge backend on split deployments |
||
|
taylanbakircioglu
|
a36dd87a74 |
fix(agent): port VIP adoption into the in-script daemon so self-upgraded agents get it
Found during an impact analysis of the agent-script change in PR #58. linux_install.sh contains TWO daemon implementations and which one runs depends on how the agent reached its current state: - Fresh install: the heredoc at lines 923-2646 is written to /usr/local/bin/haproxy-agent and systemd runs that file. - Self-upgrade: perform_agent_upgrade copies the downloaded INSTALLER script over that same path (`cp "$temp_script" "$current_script"` with current_script=/usr/local/bin/haproxy-agent). systemd then runs the installer with `daemon` + SKIP_TO_DAEMON=true, which takes the separate in-script daemon that lives after the heredoc. PR #58 added _kp_discover and the one-shot takeover only to the heredoc copy, so both were absent from the path that self-upgraded agents actually run — and self-upgrade is exactly the path the release notes tell operators to rely on ("nodes will pull the new script through the normal agent-upgrade path"). The feature would have worked on a freshly installed node and been silently inert on every upgraded one. The file's own banner warns about this ("check_agent_upgrade() - Multiple locations ... TIP: Search for function name to find all occurrences!"), and send_heartbeat / fetch_and_deploy_keepalived_config / get_haproxy_stats_csv are already maintained as parallel copies for the same reason. Verified empirically rather than by reading: the script was instrumented and run in a container exactly as systemd invokes it after an upgrade (SKIP_TO_DAEMON=true, `bash linux_install.sh daemon`), inspecting the live `declare -f fetch_and_deploy_keepalived_config`. before: DISCOVERY_YOK TAKEOVER_YOK after: DISCOVERY_VAR TAKEOVER_VAR ENDPOINT_VAR Both blocks are ported verbatim from the heredoc copy with indentation adjusted; no logic changed, so the guard semantics are identical in both paths — takeover still requires allow_takeover AND a non-empty expected hash AND a matching on-disk md5, and anything else falls through to the existing "externally managed — refusing to overwrite" branch. `bash -n` passes. Backend suite unchanged at 1263 passed. |
||
|
Mustafa ULUKAYA
|
bb774141d4 |
fix(acme): make the challenge backend fixable from the panel
Correcting a wrong ACME challenge backend was impossible without a shell, and
even with one the correction did not reach the nodes.
The mint gate only fired when `acme_enabled` flipped. `acme_backend_url` was
written to the DB and minted nothing, so Apply answered "No pending changes to
apply" and the nodes kept the old address forever. It is now decided by
comparing the rendered `server _acme_mgmt` line against the active version —
the one line that answers "would the nodes talk to a different address?".
Comparing whole configs would flag every unrelated pending edit.
The field had no UI at all. Added to the cluster form with validation that
mirrors the backend rules, and keyed on `model_fields_set` so clearing it
reverts to the global setting — with a plain `is not None` test an empty box is
indistinguishable from "not submitted", so a value could never be removed.
Validation is asymmetric on purpose (utils/acme_backend_url):
- at the write boundary, reject what cannot express a reachable target —
including the two silent traps: a scheme-less value became `localhost`, and
an out-of-range port raised inside the generator and destroyed the config
- at render time, never reject. The shipped defaults are themselves loopback,
so refusing to render would make every acme_enabled cluster unappliable,
including for changes unrelated to ACME. Problems are logged and surfaced.
The port-less default stays 8080 rather than moving to HTTP's 80: the bundled
compose publishes nginx on 8080, so installs relying on it work today and the
first sign of breaking them would be the unattended renewal loop months later.
The omission is warned about instead.
RFC1918 is allowed and is usually the right answer here, and no DNS resolution
is performed — both deliberate departures from utils/ssrf_guard, whose policy
is the opposite of what this address needs. What the management host can
resolve says nothing about what the HAProxy node can reach.
Diagnostics stop reporting success on a dead path:
- check_port80 uses GET instead of HEAD and classifies the body. A proxy that
has lost its /.well-known/acme-challenge/ location serves its SPA with HTTP
200, which `status in (200, 404)` accepted as healthy. Warnings also surface
when other domains pass, which previously hid the most diagnostic outcome.
- check_routing filters `mode`, joins `acme_enabled` and reads the APPLIED
config instead of counting database rows, and reports a loopback target.
- every new condition is `warn`, never `fail`: the site wizard blocks submit on
any fail, so a new failing condition would lock every install on upgrade day.
Also: normalise `frontends.mode` once per frontend. It is nullable, and the
raw value was interpolated into `mode {}`, emitting a literal `mode None` that
HAProxy rejects — taking down the whole cluster config. The ACME gate and the
backend-mode check now read the same normalised value.
And stop hardcoding PUBLIC_URL / MANAGEMENT_BASE_URL in docker-compose, which
silently ignored the operator's .env and made the wrong default load-bearing.
|
||
|
mustafa.ulukaya
|
ef26860df9 |
feat(logging): unified request/response log with configurable retention (v1.11.0)
Until now the only record of what happened was `user_activity_logs`, which stores non-GET 2xx operations with no bodies. When something failed you could see that a counter went up, never what was sent or what came back. This adds one queryable timeline covering both directions: - inbound: every API call, including GETs and including 4xx/5xx, with the user, client IP, status, duration and — redacted, size-capped — the request and response bodies. - outbound: every HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows inherit the inbound request's id, so one operator action and the CA/DNS calls it triggered read as a single trace: opening a failed "Request Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath. Implementation notes: - Capture is a pure-ASGI middleware that TEES the request and response streams rather than draining them. `await request.body()` inside a BaseHTTPMiddleware would consume the receive channel and break the raw-body agent heartbeat handler. Registered last so it is outermost: it then sees the final client-visible response and seeds correlation_id_context before the error handler reads it. - Rows are written by a batching background writer with a bounded queue, so the request path never awaits the database and a saturated logger drops rows visibly (surfaced on the page) instead of blocking. Redaction runs on the writer, off the request coroutine. - Secrets never land: headers are an allowlist with Authorization/Cookie kept only as a presence marker; body keys and value shapes are redacted (passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME JWS request body is never stored, because a stored protected+signature pair is a replayable credential — a summary is logged instead; DNS-provider errors record only the exception type; the ACME HTTP-01 challenge endpoint is excluded so key_authorization is never captured. - Retention is operator-configurable in Settings -> Request Log: separate day counts for successful and failed rows (7 / 30) plus a hard row cap (500k), whichever is reached first. Pruned in batches under a Postgres advisory lock, with the day counts bound as parameters, never interpolated. - New permissions requestlog.read / requestlog.manage. super_admin and security_admin get both, operator gets read, viewer gets neither. Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION 10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered) or the `enabled` toggle in Settings. Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend + 17 frontend passing. |
||
|
mustafa.ulukaya
|
164841219a |
feat(agent): report an unmanaged keepalived.conf and honour a one-shot takeover
Two additions to the Linux agent, both inside the existing keepalived converge function so no new poll or timer is introduced. Discovery is strictly read-only: when the node has a keepalived.conf without OpenManager's ownership marker, the agent posts it so an existing VIP can be adopted from the UI. Nothing is written to the node. It is rate-limited by content - the md5 of the last report is cached next to the config, so a file that may carry the VRRP password is posted only when it actually changes rather than every cycle. Once we own the file there is nothing left to adopt, so the record is cleared exactly once. The content is JSON-encoded with `jq -Rs` so newlines survive verbatim and the hash the server pins the takeover to is the hash of what is really on disk. The ownership guard now has exactly one exception, and it does not weaken it. Previously any file without the marker was refused, which is what protects a hand-maintained setup - and also what would block adoption forever. The server authorises a single takeover of a specific file by sending the md5 the operator adopted from, and the agent overwrites only when the on-disk hash still matches. If the file changed in between, the agent refuses again and reports why, so an edit made after adoption wins over the stale adoption instead of being destroyed. The fallback latest Linux agent version moves 2.0.0 -> 2.1.0 so nodes pull the new script through the normal upgrade path. Discovery simply does not happen on a node that has not upgraded yet. |
||
|
taylanbakircioglu
|
eee0a4716a |
feat(ssl): encrypt the pending CSR private key at rest (v1.10.1, closes #53)
Closes the follow-up filed during the v1.9.0 CSR review. The private key of a
PENDING CSR is now Fernet-encrypted in the database instead of being stored as
a raw PEM.
Why this key specifically: it is the one key in the system that sits idle. It
is generated at CSR creation, waits for an external CA to sign the request
(days to weeks), and is destroyed the moment the signed certificate is
imported. It is never transmitted to an agent and never leaves the server.
ssl_certificates.private_key_content and the ACME order keys are deliberately
NOT covered, because agents must receive those in plaintext on every poll, so
encrypting them at rest buys nothing without an end-to-end redesign.
Implementation follows the pattern already used for the VRRP secret, TOTP
secrets and DNS provider credentials: a new utils/csr_key_crypto.py with its
own CSR_ENCRYPTION_KEY env var and its own HKDF info string
("csr-private-key-v1"), so rotating one secret class never affects another.
No schema change and deliberately NO SCHEMA_VERSION bump: the Fernet token
replaces the PEM inside the existing ssl_csrs.private_key_pem TEXT column. A
bump would re-run the migration sequence and re-seed the four built-in roles to
their defaults, which is a needless side effect for a storage-format change.
Backward compatible with no data migration. Rows written before this release
hold a raw PEM and are still read unchanged; the discriminator is exact rather
than a heuristic, since a Fernet token is base64url and can never contain the
"-----BEGIN" marker. Legacy rows drain naturally because a CSR's key copy is
NULLed on import.
A key that cannot be decrypted (SECRET_KEY rotated while CSR_ENCRYPTION_KEY was
unset) now fails with an explicit "delete this CSR and create a new one" error.
Previously that situation would have surfaced as the far more confusing
"certificate does not match this CSR's private key".
Also documents all four per-purpose encryption keys in .env.template. Only
VIP_ENCRYPTION_KEY was listed; MFA_ENCRYPTION_KEY and
DNS_PROVIDER_ENCRYPTION_KEY had been missing since v1.6.0 and v1.8.0.
Verified before release, on a corporate pre-production environment and locally:
- Full backend suite 1234 -> 1243 passed (+9 new tests), 0 failed.
- Against a real Postgres: a CSR created through the API stores a Fernet token
with no PEM header in the column, and imports successfully.
- Full 1.10.0 -> 1.10.1 -> 1.10.0 drill on one database volume. The upgrade
logs "Schema already at version 10 (>= 10); skipping migration run", so no
migration executes and the built-in roles are not re-seeded. A CSR created on
1.10.0 with a plaintext key imports successfully after the upgrade, which is
the backward-compatibility guarantee proven against a real row rather than a
mock.
- rsa-2048, rsa-4096 and ecdsa-p384 all round-trip through create, encrypt,
decrypt and import.
- Key derivation is stable across processes: two independent containers sharing
SECRET_KEY decrypt each other's tokens (required for UVICORN_WORKERS > 1 and
multi-replica deployments), while a different SECRET_KEY yields None rather
than a wrong key or an exception.
- Downgrade behaviour was measured, not assumed: 1.10.0 cannot parse the token
and fails with HTTP 500 "key parse failed (encrypted?)" rather than pairing a
wrong key. The rollback note states the measured behaviour.
- No CSR endpoint returns the key in any form: list and detail responses
contain neither a PEM nor a Fernet token.
Not changed here, from the issue's "worth folding in" list: the create rate
limit is not a concurrency guard, create_csr holds a pooled connection across
RSA key generation, detail=str(e) echoes internal error text (a repo-wide
convention), and is_global skips cluster validation in both routers/ssl.py and
routers/csr.py. None are storage concerns and each is a separate change.
|
||
|
taylanbakircioglu
|
520b69a1c6 |
fix(security): remediate RCE, missing-auth and SSRF advisories (backend-only, no agent changes)
Addresses three reported advisories, all verified against the code. Fixes are
entirely server-side — deployed agents already send a valid X-API-Key on every
call, so enforcing it does not require any agent-script change or upgrade.
GHSA-7rhv-c5pc-69r8 (CRITICAL RCE — agent script-template poisoning):
- POST/GET /api/agents/script-templates/{platform} now require the agents.version
permission (was authentication-only), matching POST /versions. Blocks a viewer
JWT from overwriting the root install/upgrade script.
GHSA-3p5c-m5m4-mjpx (missing authentication):
- Agent data-plane endpoints now REQUIRE a valid X-API-Key (was optional/skipped
when the header was absent), checked before any DB access: config,
ssl-certificates (private keys!), upgrade-status, heartbeat (by-name and the
previously auth-less by-id), configuration pending-requests. Removes keyless
heartbeat spoofing and keyless rogue-agent auto-registration.
- Operator/UI endpoints now require a JWT: GET /api/agents, the entire
/api/dashboard-stats router, /api/health/{deep,agents,clusters}, and
/api/ssl/certificates/{id}/config-versions. The simple /api/health liveness
probe stays public. Adds shared auth_middleware.require_authenticated_user.
GHSA-3vh4-gvxx-wm2p (SSRF via ACME directory_url):
- New utils/ssrf_guard.py (https-only + public-IP-only, IPv4-pinned, no redirects),
applied to settings test-connection, acme_service.get_directory and
_signed_request, and validated at Let's Encrypt account creation. The
test-connection response no longer reflects arbitrary upstream JSON keys
(information-disclosure oracle) — only fixed ACME field names.
Verified: full pytest tests/ (1128 passed, 0 failed) + live localtest stack smoke
(valid JWT/key paths return 200/404 as expected; anonymous requests 401; SSRF to
metadata/private/loopback refused). No changes to backend/utils/agent_scripts/*.
|
||
|
taylanbakircioglu
|
9e2ea04777 |
feat(haproxy): preserve SPOE filter + frontend log-format on import/edit (v1.8.8, Issue #38)
Bulk import / manual edit silently dropped `filter spoe engine ...` (Coraza WAF) and frontend `log-format` because the parser recognised only a fixed directive set. The regenerated config then missed the SPOE engine, so HAProxy failed with "unable to find SPOE engine 'coraza' used by the send-spoe-group". - parser: capture `filter` + `log-format`/`log-format-sd` into new ParsedFrontend fields - db: additive nullable `log_format` + `filters` TEXT columns on frontends (SCHEMA_VERSION 8->9) - generator: new `filter` bucket flushed before http-request rules so `filter` precedes `send-spoe-group`; `log-format` kept in prelude - bulk import: preview dict, change-detection, persist (create + merge-update); cluster-aware SPOE pre-flight advisories (missing-filter + host-prerequisite) surfaced in the UI - manual CRUD: full round-trip (get/create/update) incl. React form fields (no null-wipe) - reject/rollback: restore the new columns; restore path + wizard helper kept in parity - backend `option spop-check` recognised (suppresses spurious warning for coraza-spoa) - tests: test_spoe_filter_import.py; full suite green (1079 passed) |
||
|
taylanbakircioglu
|
64d42663cd |
fix(agent): stop installer self-kill in pre-installation cleanup (v1.8.4)
The Linux/macOS agent installer could abort during "pre-installation cleanup"
(terminal showed `Killing processes matching: haproxy-agent` then `Killed`,
returning to the prompt) when the install script's own command line contained
"haproxy-agent". The cleanup killed processes via `pgrep -f "$pattern"` starting
with the bare string "haproxy-agent", which also matched the running installer's
own command line and a sudo/PAM ancestor that the $$/$PPID self-exclusion did not
cover, so the installer terminated itself before installing.
- linux_install.sh / macos_install.sh: the cleanup kill loop now targets ONLY
the installed agent - "$INSTALL_DIR/haproxy-agent" (the daemon binary path) and
the agent service/label ("haproxy-agent.service" / "com.haproxy.agent") - never
the bare "haproxy-agent" substring. Neither pattern can match the installer's
own command line. The redundant bare pattern is dropped (the service is stopped
separately, and the binary-path pattern still catches a running daemon).
- frontend (AgentManagement.js): name the downloaded scripts
install-agent-<platform>.sh / uninstall-agent-<platform>.sh (matching the
backend's suggested filename) - defense in depth so this cannot resurface.
Installer-only change. The running agent and its privilege model are unchanged
(it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade).
The cleanup runs only on a full interactive install (gated by SKIP_TO_DAEMON), so
daemon mode, self-upgrade, and config/version apply are unaffected. Both agent
scripts kept in sync. Scripts parse on bash 4.2-5.2; full backend suite green.
Addresses #31.
|
||
|
taylanbakircioglu
|
27fbe48c4b |
fix(agent): tolerate empty system_info in heartbeat JSON (v1.8.3)
A self-hosted agent could fail every heartbeat with HTTP 400 `Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host. The agent builds the heartbeat JSON as text, so an empty `$system_info` collapsed the `$system_info,` line to a bare comma and broke the payload. - Agent script (linux + macos, kept in sync): guard the fragment-form register_agent and send_heartbeat builders so an empty system_info falls back to a valid key and can never emit a bare comma. Uses the most portable bash glob test (no POSIX class / pattern substitution; verified on bash 3.2-5.2 and on Ubuntu/Debian/Rocky/Alpine/Amazon Linux). True no-op for healthy agents. - Backend heartbeat endpoint: parse the body as-is first and only run the malformed-JSON repair when parsing fails, so a valid heartbeat from any agent version is byte-for-byte untouched. The repair (now a testable helper) recovers a leading or doubled comma (the empty-system_info artifact) in addition to the existing empty-value / trailing-comma fixes. No agent version bump; self-upgrade and daemon mode are unaffected. Healthy agents of every version behave identically. Full backend suite green. Addresses #31. |
||
|
taylanbakircioglu
|
c492b26bb1 |
feat(acme): DNS-01 challenge support with pluggable DNS providers (v1.8.0)
Add ACME DNS-01 (TXT-record) validation alongside the existing HTTP-01, for internal/isolated clusters with no public port 80 and for wildcard certificates. Opt-in via a global kill-switch (default off); HTTP-01 is byte-for-byte unchanged, with zero agent or rendered-config changes. - Pluggable DNS provider interface (Manual + Cloudflare). Per-account credentials are Fernet-encrypted at rest, verified on save, and never returned by the API or written to logs/events/error_detail. - Non-blocking per-cycle orchestrator: publish (CAS) -> propagation grace (across cycles, no in-loop sleep) -> respond -> finalize/download, with a bounded fresh-order retry chain (1 original + 3 retries) on propagation lag. - Manual flow: user publishes the TXT record and confirms; manual DNS-01 cannot auto-renew unattended (auto-renew forced off and surfaced in the UI). - Migration v8: additive, idempotent columns on letsencrypt_accounts/orders and acme_challenges, plus a new letsencrypt_account_dns_credentials table. - Challenge-type-aware diagnostics (port80/routing/DNS checks skipped for DNS-01) and a DNS-01 event timeline in the order detail. - Frontend: DNS-01 account + credentials management, cert wizard adaptation, order-detail TXT records + verify, orders/renewal Method columns, and a Settings kill-switch. README, release notes, and API docs updated. Implements #35. |
||
|
taylanbakircioglu
|
a1192e602d |
feat: HA / VIP (Keepalived) management from the UI (#27)
Manage highly-available virtual IPs backed by Keepalived (VRRP) directly from the OpenManager
UI — no more SSHing into nodes to install/configure Keepalived by hand. Builds on the agent
pull-architecture: define the VIP centrally, click Apply, and the agents converge.
Highlights:
- New "HA / VIP" tab: create a virtual IP, pick a per-node interface, select which pool nodes
participate (MASTER/BACKUP roles + priorities); live MASTER/BACKUP/FAULT per node.
- On Apply, agents install & configure Keepalived (unicast VRRP, cloud-safe default) across the
major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine) with a
HAProxy health-check, so the VIP fails over automatically when HAProxy drops.
- Single-node (a managed floating IP without failover) and multi-node VRRP failover both work.
- VIP changes ride the standard Apply Management flow with the standard "View Change" diff.
- Approval-gated deletion (safety): deleting a running VIP is staged for approval and the VIP
keeps running, untouched, until you approve it — an agent never tears a VIP down without an
explicit human approval. Per-VIP Diagnostics view; opt-in package uninstall (only on nodes
where OpenManager installed it). A node already running a hand-managed Keepalived is detected
and never overwritten ("externally managed").
- Fully opt-in and backward compatible: nodes/clusters without a VIP are unaffected. Adds
vip_instances + vip_members tables (idempotent SCHEMA_VERSION bump; existing data and
passwords unaffected) and a `vip` RBAC permission group.
- Also includes a HAProxy config-generator robustness fix: auto-inject a stick-table when a
frontend uses a stick counter (track-sc / sc_*_rate) but declares none.
On-prem / L2 (VRRP) scope; the UI notes the cloud caveat.
|
||
|
taylanbakircioglu
|
f3d4fb11bb |
fix(agent): accept agent token (X-API-Key) on cluster read endpoints (Issue #22)
Assigning a HAProxy agent failed with '401: Authorization header missing'
on GET /api/clusters. A cluster-read hardening had made GET /api/clusters and
GET /api/clusters/{id} accept only a user JWT in the Authorization header;
agents authenticate with their agent token in the X-API-Key header, so the
token was never read.
Both endpoints now accept either a user JWT (Authorization) or an agent token
(X-API-Key via validate_agent_api_key), mirroring the existing dual-auth on
POST /api/agents/generate-install-script. Anonymous access is still rejected,
so the original hardening is preserved. The auth guard is placed before the
try block so the failure surfaces as a clean 401 (not the 500-wrapped-401 in
the report). Agent install scripts now consistently send the token via
X-API-Key (pre-flight cluster check on linux/macos, and macOS get_cluster_paths
which previously used the wrong Authorization: Bearer header).
Also normalizes the platform in the uninstall-script generator so macOS agents
(which report platform 'darwin') no longer get a 400 from
GET /api/agents/generate-uninstall-script/darwin.
version 1.6.0 -> 1.6.2.
|
||
|
taylanbakircioglu
|
02b1cb2bca |
feat: v1.5.0 — Site Wizard (Issue #14) + ACME Diagnostic Panel (Issue #13)
Closes #13, Closes #14. This release squashes the v1.4.0 → v1.5.0 development line. v1.4.0 shipped the ACME stability & enterprise audit (Issues #10/#11/#12). v1.5.0 builds on that foundation with two co-equal headline features plus a 22-round audit campaign hardening the prior configuration surface. License remains MIT for v1.5.0 (relicense to AGPL-3.0 lands in v1.5.2). ------------------------------------------------------------------ HEADLINE FEATURE A — ACME Diagnostic Panel (Issue #13) ------------------------------------------------------------------ A live pre-flight + post-failure diagnostic surface for every ACME order, reachable from the ACME Automation page. The panel exists to make ACME failures legible to operators who do NOT have shell access to the API host. Endpoints (`backend/routers/acme_diagnostics.py`): POST /api/letsencrypt/orders/{order_id}/diagnostics Run the full 5-check suite (DNS / port-80 / routing / account / agents) and humanize the order's `error_detail` (>=11 RFC-8555 problem types, backwards compatible with legacy plain-string failures). POST /api/letsencrypt/orders/{order_id}/diagnostics/ {check_id}/rerun Re-run a single check in place — used by the "Re-run" button on every row of the modal's pre-flight table. GET /api/letsencrypt/orders/{order_id}/events Merged event timeline combining the typed `acme_order_events` rows with correlated `user_activity_logs` entries (resource_type = 'letsencrypt_order' AND resource_id = order_id). The diagnostic modal auto-tails this timeline every 5 seconds while open. Service-level checks (`backend/services/acme_diagnostics.py`): * DNS resolution via stdlib socket.gethostbyname_ex through run_in_executor (intentionally avoiding an aiodns runtime dep for v1.5.0). * Port-80 HEAD probe, target locked to the order's domains, success on HTTP 200 OR 404, warns on egress timeout (corp egress policies routinely blackhole outbound 80 — fail-hard would be too noisy). * SSRF guard: probe refuses non-public IPs and surfaces the skip in the diagnostic result; IPv4-mapped IPv6 normalisation closes the `::ffff:169.254.169.254` cloud-metadata vector. * HAProxy routing presence check: matches the order's cluster_ids to a port-80 HTTP frontend. * ACME account validity check against `letsencrypt_accounts`. * Agent presence check (>=1 active agent in target cluster). * Every sub-check wrapped in a wall-clock timeout to bound impact on the API event loop. RBAC: ssl.read for run, ssl.read for events. Per-user 5/min rate limit on both run and rerun, backed by the (user_id, action, created_at DESC) composite index. Frontend (`frontend/src/components/ACMEAutomation.js`): * "Diagnose" button on every order row + the existing "stuck order" warning row. * Modal with two tabs: - Pre-flight Checks (Antd Table with status pills + Re-run buttons + humanized error banner) - Event Log (Antd Timeline with auto-tail polling, scroll- to-bottom, pause-on-hover) * Correlation IDs surfaced in error banners and individual check fail details for backend-log lookup. ------------------------------------------------------------------ HEADLINE FEATURE B — Site Setup Wizard (Issue #14) ------------------------------------------------------------------ A single guided flow that creates a Backend + Servers + HTTP Frontend (and optional HTTPS Frontend) in one atomic transaction. Endpoints (`backend/routers/site_wizard.py`): POST /api/site-wizard/preview — diff-preview the changeset POST /api/site-wizard/create — atomic execute POST /api/site-wizard/reject — clean rollback (including any wizard_staged ACME orders) GET /api/site-wizard/drafts — draft persistence PUT /api/site-wizard/drafts/{id} — save/update DELETE /api/site-wizard/drafts/{id} Feature surface: * One screen captures both backend (mode + servers) AND frontend (http + optional https + SSL mode) inputs. * SSL modes: ACME (new order, HTTP-01 only for v1.5.0), Upload (existing PEM), Existing (link to a stored cert), or None. * ACME-staged path: wizard_staged_until watermark on the `letsencrypt_orders` row defers finalisation until agent confirmation; per-mode reject cleanly cancels and rolls back the staged order. * Live diff preview against the cluster's current generated config (renderer-evolution noise stripped — track-sc<N> dedup, per-server cookie strip, defaults-cookie inheritance, listen-block flattening). * Draft persistence with PEM stripped at save time (private keys never round-trip through the drafts table). * Per-cluster multi-tenancy: drafts and wizard_staged orders are isolated to the creating user's cluster scope. Frontend (`frontend/src/components/SiteWizard.js`): * 4-step Antd Steps flow: Backend → Frontend → SSL → Review. * Render the live diff preview inline before commit. * Antd Form-level validation mirrors backend Pydantic validators (numeric bounds, HAProxy reserved keywords, ALPN consistency, IPv6 scope-id, domain regex, server name dedup). ------------------------------------------------------------------ AUDIT CAMPAIGN — Rounds 1 → 22 (Bulgu #1 → #82) ------------------------------------------------------------------ v1.5.0 includes 22 adversarial review passes. Each round produced its own commit set in the corporate development line; this squash collapses those into the v1.5.0 release artefact. Highlights: Round 1-4 Site Wizard core: dry-run parity, single-line value injection guard, ACL -f pattern-file block, SSL parity, timeout regex, form-state pin. Round 5-7 defaults-cookie inheritance, server-named-cookie guard, fe/be mode mismatch, duplicate server names, health_check_uri + server_address validators. Round 8-10 cookie_name / cookie_options newline-injection guard, dry-run parity (round 9), TCP-mode HTTP-only feature blockers. Round 11 SSL name path traversal + health-check >= 1. Round 12-13 SSL & ACME deep dive (Bulgu #23-#32). Round 14 single-line value injection (Bulgu #33). Round 15-17 ACME multi-tenant UX, numeric bounds, HAProxy reserved keywords, ALPN/TLS consistency, all-backup, multi-domain & multi-user enterprise edges, drain/HSTS/post-completion (Bulgu #34-#53). Round 18-21 concurrency, agent state, TCP-mode HTTP-only, list size caps, IPv6 scope-id, preview account validation, TCP backend + balance uri reject (Bulgu #54-#61). Round 22 FE error visibility + 3x stale-data lockouts, referential integrity + cascade safety, authentication & authorization, multi-cluster isolation, apply_pending_changes concurrency, script injection + bulk import multi-tenancy, prefix-stripped signature comparison (Bulgu #62-#82). ------------------------------------------------------------------ NO CORPORATE-SPECIFIC ARTIFACTS ------------------------------------------------------------------ This squash deliberately sanitises corporate hostnames, container registry references, and TLS secret names into generic placeholders (`your-registry.example.com/your-org`, `haproxy-openmanager*.example.com`, `wildcard-tls`, `taylanbakircioglu/haproxy-openmanager-*`) so the public artefact contains no internal infrastructure detail. Pilot / development history that retained those values stays in the corporate fork and is NOT part of this commit. |
||
|
taylanbakircioglu
|
07942a82e8 |
feat: ACME stability & enterprise audit (v1.4.0) — fixes #10 #11 #12
Issue #10 — Silent timezone failure on ACME certificate save - Normalize tz-aware expiry_date to UTC tz-naive before INSERT/UPDATE in ssl_certificates (TIMESTAMP WITHOUT TIME ZONE) — restores ACME download path Issue #11 — Duplicate _acme_challenge_backend in generated config - Generator guard: skip auto-append when backend already rendered - Agent-sync filter: _should_sync_backend() drops system-managed backends and detaches their server rows to prevent orphans - Restore filter: IGNORED_BACKENDS skips reserved names during cluster restore - Parser warning: reserved_backend_names blocks accidental manual import - Cleanup migration: removes orphan rows + cascading server entries (idempotent) Issue #12 — Validated orders required manual completion - New 60s background task complete_pending_acme_orders, flag-independent, multi-replica safe via FOR UPDATE SKIP LOCKED + 30s updated_at watermark - Per-order pg_advisory_lock(0x41434D45, order_id) serializes UI-Complete and auto-task races; idempotency guard returns existing certificate cleanly - retry_order endpoint reports in_progress: true within 30s window so the UI surfaces an info toast instead of duplicating CA requests - ACMEAutomation surfaces stuck orders (status=valid && !ssl_certificate_id) with a one-click Complete action and Cancel fallback; conditional 30s polling Other hardening - ACME state machine error_detail persisted as structured JSON across challenge, finalize, download stages for actionable post-mortems - CertificateRequest Pydantic model: domain regex + min_length/max_length and cluster_ids defaulting to all ACME-enabled clusters when "global" is selected - Renewal cluster fallback now requires acme_enabled=TRUE in addition to active - _complete_certificate preserves manual cluster assignments on renewal, surfaces cluster_errors, raises explicit error on missing private key - Audit logging covers acme_certificate_requested/revoked, ca_chain_imported, account created/deactivated/purged, order retried/cancelled - Settings UI exposes acme.staging_url_override for private test CAs (Pebble) - Schema additions: acme_challenges.attempts (default 0) and last_attempt_at, index idx_letsencrypt_orders_status_updated; all migrations idempotent Tests (41/41 passing) - test_acme_expiry_normalize, test_acme_duplicate_backend, test_acme_state_machine, test_acme_pydantic_validation, test_acme_audit_logging, test_acme_concurrency CI / packaging - docker-build.yml reads version.json and pushes additional product-version tag (e.g. 1.4.0) alongside latest and timestamp build id Closes #10 Closes #11 Closes #12 |
||
|
taylanbakircioglu
|
cd4a94beb1 |
feat: agent IP/VIP live update + script update detection banner
- Agent scripts now detect and send ip_address in DAEMON heartbeat (Linux: ip route, macOS: ifconfig) - Backend validates agent-reported IPs via ipaddress stdlib, COALESCE preserves existing on NULL - IP/VIP change logging (non-critical, try/except wrapped) for operational visibility - New source_file_hash column on agent_script_templates for reliable update detection - Migration changed to ON CONFLICT DO NOTHING to prevent overwriting UI-customized scripts on restart - GET /versions returns script_update_available flag (disk hash vs DB hash comparison with fallback) - Frontend Alert banner warns users of new agent script versions and directs to Reset to Defaults - Reset to Defaults and Popconfirm modals explicitly warn about custom script edit loss - Full backward compatibility: old agents without ip_address field continue working unchanged Made-with: Cursor |
||
|
taylanbakircioglu
|
71c717364c |
fix: allow dot character in entity names for UI and backend validation
Bulk import accepted dots in frontend/backend/server names but UI and backend validators rejected them with ^[a-zA-Z0-9_-]+$. After import, entities with dots could not be edited. HAProxy itself allows dots in section names, so the regex is expanded to ^[a-zA-Z0-9_.-]+$ across all 12 validation points (5 React form rules, 1 ACL char-strip, 3 Pydantic validators, 1 WAF validator, 2 config-validator warnings). |
||
|
taylanbakircioglu
|
7fffaddaf8 |
fix: bulk import should not inject hardcoded timeout defaults
When a backend block in haproxy.cfg does not specify explicit timeout values, the bulk import was injecting hardcoded defaults (connect 10s, server 60s, queue 60s) into the database. These then appeared in the generated config and overrode the agent's defaults section. Now, only explicitly declared timeouts are stored; omitted ones remain NULL so the agent's existing defaults section stays in effect. |
||
|
taylanbakircioglu
|
36fdba52bc |
fix: ACME setup guide accuracy, reject rollback, and UX improvements
- Step 3 (Enable ACME on Cluster) now shows a process icon instead of a misleading green checkmark when ACME is enabled but not yet applied. Per-cluster "(pending apply)" annotation for multi-cluster setups. - Step 4 button and all /apply-management navigation buttons now say "Apply Changes" instead of "Configure" for clearer guidance. - Setup Guide auto-selects the correct cluster before navigating to Apply Management, showing pending cluster names in alerts. - Pending ACME disable changes are now correctly detected in Step 4 even when acme_enabled is already FALSE in the database. - Entity snapshot rollback for cluster ACME settings: reject correctly restores acme_enabled/acme_backend_url to pre-change values. - Deduplication logic prevents "last wins" bug when multiple ACME toggles are rejected in sequence. - Connection leak prevention with try/finally around conn2 in ACME config version creation. - Step 4 branching uses boolean has_enabled instead of fragile string truthiness check. Made-with: Cursor |
||
|
taylanbakircioglu
|
fabe7aba4c |
fix: prevent empty HAPROXY_CONFIG_PATH when get_cluster_paths() fails
Agent startup left HAPROXY_CONFIG_PATH empty when the /api/clusters jq
select() returned no output (e.g. transient API failure or cluster_id
mismatch). This caused "No existing HAProxy config found at: " errors
and partial config merge failures for newly added agents.
Three-layer fix:
1. Load HAProxy paths from config.json as baseline before get_cluster_paths()
2. Use local variables in get_cluster_paths() - only override globals when
API returns non-empty values (defensive against jq select() empty output)
3. Dynamically update paths from /api/agents/{name}/config response in
check_config_updates() - allows cluster path changes without reinstall
Applied to both linux_install.sh and macos_install.sh.
Made-with: Cursor
|
||
|
taylanbakircioglu
|
4082152312 |
fix: resolve stale keepalive state when keepalived stops
- Fix DB: COALESCE(NULLIF) prevented clearing stale MASTER/BACKUP state.
Use CASE WHEN IS NOT NULL to distinguish absent fields (old agents)
from empty fields (keepalived stopped) and properly clear to NULL.
- Fix Redis: explicitly delete cache key when keepalived stops instead
of relying on 90s TTL expiry, preventing stale dashboard data.
- Fix agent scripts: SKIP_TO_DAEMON send_heartbeat now always sends
keepalive_state/keepalive_ip fields (even empty) so backend can
detect stopped keepalived and clear stale data.
- Fix legacy heartbeat endpoint: POST /{agent_id}/heartbeat now also
updates keepalive_state and keepalive_ip fields.
- UI: add horizontal scroll to ClusterManagement table to prevent
column overflow with new Keepalive column.
- UI: add VIP tooltip to Dashboard AgentStatusCard keepalive tag.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
taylanbakircioglu
|
b0feca3c2e |
feat: add keepalived VRRP state (MASTER/BACKUP) detection to agent heartbeat
- Add keepalive_state and keepalive_ip columns to agents table (migration + schema) - Add keepalive fields to AgentHeartbeat Pydantic model (backward compatible) - Update heartbeat endpoint to persist keepalive data to DB and cache in Redis - Add multi-method keepalived detection in agent scripts (journalctl, log files, VIP check) - Update dashboard-stats agents/status API with Redis-first keepalive lookup - Update GET /api/agents to include keepalive_state and keepalive_ip - Show MASTER/BACKUP tag in Dashboard AgentStatusCard - Show keepalive info in Agent Management registered agents table - Add Keepalive column to Cluster Management table with VIP search support Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
0be3c299dd |
fix: serialize new_values in entity snapshot to prevent silent JSON failure
Root cause: SSL update passes expiry_date as Python datetime object in
new_values. json.dumps() fails on datetime, causing save_entity_snapshot
to return {} (empty). Entity snapshot is never saved in config_version
metadata, so reject/rollback can never find it to restore old values.
Fix: Apply same JSON serialization to new_values as old_values. Only
affects SSL certificates (only entity with datetime in new_values).
All other entity types (frontend, backend, server, waf) are unaffected
as their new_values contain only JSON-serializable types.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
taylanbakircioglu
|
1f100d01e7 |
fix: SSL reject rollback, expiry calculation, and UI improvements
- SSL Reject: Remove conditional has_same_update_applied check, always rollback since Auto-Reject handles cross-cluster consistency - SSL Rollback: Restore expiry_date by parsing ISO string back to datetime - SSL UI: Add In Use filter toggle, fix Expiring Soon count using actual date calculation, smart Private Key content validation messages - Remove emojis from log messages Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
7dedd0f573 |
feat: SSL sync status tracking and smart reload optimization
- Add per-cluster SSL sync status tracking with APPLYING/SYNCED/PARTIAL states - Add timeout detection (10min) for stalled SSL deployments showing PARTIAL status - Add SSL-specific info in Apply Management dialog (cluster count, auto-apply notice) - Add Deployment Status tab in SSL details showing per-cluster agent sync progress - Implement SSL auto-reject (reject propagates to all clusters) and auto-undo - Add conditional entity rollback for SSL reject (time-window based safety check) - Filter REJECTED status from SSL last_config_status display - Add pending_cluster_names to SSL list API response - Optimize agent SSL reload: only trigger HAProxy reload when changed cert is actually referenced in the cluster's HAProxy config (grep -qF check), preventing unnecessary reloads on clusters that don't use the updated certificate - Applied to all 4 SSL deploy paths: Linux/macOS standalone and daemon embedded modes Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
c19f0b5275 |
fix: SSL certificate updates now trigger HAProxy reload after self-upgrade
Root cause: After agent self-upgrade, the embedded daemon (SKIP_TO_DAEMON block) runs instead of run_daemon(). HAPROXY_BIN and HAPROXY_CONFIG variables were uninitialized before the daemon loop, causing SSL-triggered HAProxy reloads to silently fail with empty path validation. Changes: - Initialize HAPROXY_BIN/HAPROXY_CONFIG before embedded daemon loop - Add md5 checksum comparison in deploy_ssl_certificates() to detect actual cert file changes (avoid unnecessary writes and reloads) - Add check_ssl_updates() for independent SSL sync every ~2.5 min in run_daemon(), independent of config version changes - Add SSL-aware reload in check_config_updates(): if config validation fails but SSL certs changed, reload HAProxy with existing config - Add "full" fetch mode to fetch_and_deploy_ssl_certificates() to bypass incremental timestamp filter for standalone SSL checks Applied to both linux_install.sh and macos_install.sh. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
dc978f847d |
fix: prevent daemon crash-loop on exit 1 with retry-backoff mechanism
- Replace immediate exit 1 with retry-with-backoff in all daemon mode paths (dependency checks: 5 retries, 30/60/90/120/150s; config file: 5 retries, 15/30/45/60/75s) - Make socat missing non-fatal in daemon mode (agent continues without stats) - Add SystemD StartLimitBurst=5/StartLimitIntervalSec=120 for new installs - Add macOS launchd ThrottleInterval=30 for new installs - Fix Linux daemon fallback to installer mode (would hang on interactive read) Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
93602fa65d |
fix: check and install jq before cluster validation in agent scripts
Move jq dependency check to run before cluster validation which requires it. Auto-install jq via the detected package manager (apt, dnf, yum, zypper, apk for Linux; brew for macOS). Exit with clear manual install instructions if automatic installation fails. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
cc10015952 |
fix: prevent install script from killing itself during cleanup
Same pgrep self-kill bug as uninstall scripts. When user downloads install script as install-haproxy-agent.sh, pgrep -f "haproxy-agent" matches the installer's own bash process and kills it before installation starts. Now filters out INSTALLER_PID and PPID. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
2d5c863735 |
fix: replace static protected paths with dynamic safety in uninstall
Instead of a hardcoded PROTECTED_PATHS list (varies per environment): - safe_rm() blocks any path not containing "haproxy-agent" - Process killer verifies ps output contains "haproxy-agent" - Pre/post HAProxy integrity check via md5 hash comparison of config and service status diff (was running -> still running?) Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
8351a6439b |
fix: add HAProxy safety guards to uninstall scripts
- PROTECTED_PATHS array prevents accidental removal of haproxy.cfg, haproxy binary, haproxy.service, haproxy logs - safe_rm() blocks any path not containing "haproxy-agent" - Process killer verifies command line contains "haproxy-agent" and skips HAProxy master PID - Binary removal validates filename is exactly "haproxy-agent" - Pre-flight safety check reports HAProxy status before starting - Post-uninstall HAProxy integrity check confirms nothing was touched Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
08c500855e |
fix: rewrite uninstall scripts - simple and reliable
Replaced overcomplicated 380-line scripts with clean ~125-line versions. Steps: stop service, kill processes (excluding self), remove binary, remove config/logs/temps, verify. No emojis, no fragile helpers. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
b7dde62972 |
fix: prevent uninstall script from killing itself via pgrep
pgrep -f "haproxy-agent" was matching the uninstall script's own process (uninstall-haproxy-agent-linux.sh contains "haproxy-agent"), causing the script to terminate itself at step 1/7 before reaching file cleanup. Now filters out $$ (own PID) and $PPID from kill lists. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
5cec6cf85a |
fix: remove emojis from all agent scripts and fix install/uninstall bugs
- Remove all emoji characters from install scripts (linux, macos) - Rewrite uninstall scripts: remove set -e, fix safe_remove to always return 0, replace ((var++)) with $((var + 1)), fix local keyword usage, add comprehensive cleanup including config.json - Fix install script QUIET_MODE/DAEMON_MODE detection to correctly handle interactive runs when old config.json exists Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
taylanbakircioglu
|
eac00b3e39 |
fix: consistent section header detection in listen extraction
Additional fix for listen block extraction: - Section boundary check now uses ([[:space:]]|$) pattern - Previously only checked [[:space:]], missing 'defaults' at end of line - Ensures listen extraction stops correctly at any section header Pattern consistency verified across all 4 locations: - Linux install mode: global, defaults, listen extraction ✓ - Linux daemon mode: global, defaults, listen extraction ✓ - macOS install mode: global, defaults, listen extraction ✓ - macOS daemon mode: global, defaults, listen extraction ✓ Edge cases tested: - Named defaults (defaults http): works correctly - Multiple listen blocks: all preserved - Tab characters: handled by [[:space:]] - User's actual config structure: produces clean merge No impact on other agent functions (self-upgrade, metrics, etc.) |
||
|
taylanbakircioglu
|
25408a618e |
fix: awk patterns now match keywords at end of line
CRITICAL BUG FIX: - Previous pattern `/^defaults[[:space:]]/` required whitespace after keyword - HAProxy allows `defaults` without a name (no trailing whitespace) - Pattern failed to match, causing global extraction to include defaults section New pattern uses `([[:space:]]|$)`: - Matches keyword followed by whitespace OR end of line - `defaults` (no name) now correctly triggers exit - `defaults http` (named) also correctly triggers exit Test results: - Old pattern: extracted 8 lines (included defaults) ❌ - New pattern: extracted 4 lines (stopped at defaults) ✅ |
||
|
taylanbakircioglu
|
995d87df43 |
fix: defaults extraction captures only first defaults section
CRITICAL BUG FIX: - Previous awk pattern allowed multiple defaults sections to be captured - Pattern `!/^defaults/` meant "don't exit if line IS defaults" - This caused duplicate defaults when config had listen before defaults New pattern uses `started` flag: - First `defaults` line: set started=1, begin capturing - Second `defaults` line: started is set, EXIT immediately - Any `frontend/backend/listen`: started is set, EXIT Also includes: debug logging for validation error storage with fallback Tested scenarios: - Normal config (defaults → listen → frontend): ✓ - Listen before defaults: ✓ - Two defaults sections: ✓ Only first captured - No defaults section: ✓ Empty output - Defaults at end of file: ✓ - Empty defaults section: ✓ |
||
|
taylanbakircioglu
|
4a6bc1b139 |
fix: global section extraction includes listen blocks when listen comes before defaults
Bug: When haproxy.cfg has 'listen stats' BEFORE 'defaults' section, the global extraction incorrectly included listen stats because it only stopped at 'defaults', not at any section header. Fix: Changed extraction to stop at ANY section (defaults/listen/frontend/backend) This prevents duplicate listen blocks in merged config. Root cause of 'proxy stats has same name as proxy stats' HAProxy validation error. |
||
|
taylanbakircioglu
|
e75bfdf706 |
feat: dynamic HAProxy binary path from cluster configuration
- /api/agents/{name}/config now returns haproxy_bin_path, haproxy_config_path, stats_socket_path from cluster
- Agent daemon uses these dynamic paths from API response for validation
- Fallback to local config file if API values not present
- Allows cluster admin to change paths without reinstalling agents
This fixes validation when HAProxy binary is in non-standard location
|