Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.
DISCOVERY REPORTS THAT NEVER REACHED THE SERVER
The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.
- the report is cached only on a 2xx;
- GET /agents/{name}/keepalived-config now reports whether the server actually
holds a discovery for that agent, and the cache may only suppress while it
says yes - which is what lets nodes stuck from earlier releases recover on
their own, with nobody touching them;
- the flag is parsed with has() + tostring, not `// empty`: jq's alternative
operator returns the alternative for **false** as well as null, so the naive
form could not tell "no record" from "older backend" and the recovery would
have been completely inert;
- a 400/413/422 records the refusal so identical bytes are not re-posted
forever - 4xx and 5xx agent calls are never sampled out of the request log,
so an unattended loop would write a row carrying the whole config every
cycle - while 401 and 404 keep retrying, because here they mean a token
rotation or an agent row briefly absent, not a bad payload;
- the CLEAR path had the same exit-code defect, where it left a stale row
offering a managed node for adoption with nothing to ever retry it.
CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS
check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.
DETERMINISTIC CONFIG PATH
A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.
Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.
Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.
The page and its policy are two halves of one feature: what gets captured and
how long it is kept lives in Settings -> Request Log, and the log itself is the
page. Sitting between Config Versions and Clusters put the two at opposite ends
of the menu. Now directly above Settings.
Menu placement only. The route, the page, the permission gate and every
backend behaviour are unchanged.
Three hardening fixes with the same shape: a number that was true under the
defaults and untrue at the edges.
1. QUEUE MEMORY WAS AN OPERATOR SETTING, NOT A LIMIT.
The queue was bounded by ROW COUNT only, and how much a row weighs is
`requestlog.max_body_bytes` - editable from Settings, documented ceiling 256 KB,
and a row can hold that twice (request + response). Measured on the real
dataclass with distinct buffers per row:
defaults, 2 000 rows x 8 KB 33.9 MiB 3.3% of the 1 GiB pod limit
max_body_bytes at its 256 KB ceiling 1003 MiB at the pod limit
REQUEST_LOG_QUEUE_MAX at its ceiling 1695 MiB over the pod limit
Both are reachable from in-range, documented values, and the drop warning
advised "raise REQUEST_LOG_QUEUE_MAX" - so following the tool's own advice on a
busy install could OOM the worker. REQUEST_LOG_QUEUE_MAX_BYTES (default 64 MiB)
now caps the queue in bytes as well as in rows, whichever binds first, released
as rows drain. Verified: with max_body_bytes at 256 KB the queue holds 7.5 MiB
against an 8 MiB budget where it would otherwise have held 1003 MiB, and it
accepts rows again as soon as the writer drains it. The warning text now names
the setting that actually helps.
2. SINK COUNTERS ARE PER WORKER AND DID NOT SAY SO.
The sink is a module global, so with UVICORN_WORKERS > 1 each process has its
own queue and its own counters, and `GET /api/request-logs/stats` reports
whichever worker happened to serve the request. The feature is sold on "a
saturated logger drops rows visibly"; at 4 workers the visible number was a
quarter of the truth. Labelled `"scope": "this worker only"` rather than
aggregated - there is no cross-process channel here, and a number that looks
fleet-wide but is not is worse than one that admits its scope.
3. AN EMPTY EXCLUDE LIST IS NOT APPLIED AS "LOG EVERYTHING".
normalize_exclude_paths() falls back to the shipped defaults when the list comes
out empty, which is the right call - it keeps the log viewer and the raw-body
heartbeat endpoint excluded - but the UI kept displaying the empty list the
operator typed, so the form showed a policy that was not in effect. The save
handler now re-applies whatever the server actually stored (which also surfaces
server-side clamping of every numeric field) and says plainly that the defaults
were restored.
The row rate of `request_logs` was a function of how many nodes are installed,
not of what anyone did. Counted from the agent loop in linux_install.sh, each
agent's 30s cycle issues three logged calls - config, pending-requests,
upgrade-status (the heartbeat is already on the default exclude list) - plus
keepalived-config and keepalived-status every fifth cycle. That is ~9 800
rows/day per agent, essentially all of them 200s meaning "nothing changed".
Measured on PostgreSQL 15 against the real DDL and all nine indexes, at 2 424
bytes/row:
20 agents ~196k rows/day 453 MB/day row cap reached in 2.5 days
200 agents ~2.0M rows/day 4.4 GB/day row cap reached in 6 hours
500 agents ~4.9M rows/day 11 GB/day row cap reached in 2 hours
The cap holds, so nothing runs away - but it holds by deleting, and what it
deletes is everything else. The shipped policy says 7 days of successes and 30
days of failures; on a 200-node fleet it delivers about six HOURS of both. The
forensic record the feature exists for is evicted by polling noise, and the
larger the installation the less history it keeps.
`requestlog.capture_agent_success`, default FALSE: a SUCCESSFUL inbound call
from an agent is not recorded. Failures always are, whatever the flag says -
they are what an operator needs and they are rare, so they cost nothing. With
this the table's size follows operator activity, and adding nodes does not
shorten anyone's retention.
Agent traffic is identified by header only, no database round-trip on the hot
path: the installed agent sends `X-API-Key` and never `Authorization`, the UI
sends a JWT and never an agent key. `generate-install-script`, the one endpoint
that accepts either, classifies correctly under the same rule - an operator
generating a script sends Authorization, a self-upgrading agent sends only the
key. The result is stored in the existing `target` column, which already means
"who was on the other end" for outbound rows and now means the same for inbound
ones, so no schema change and the existing target index applies.
Second half, and the reason this is one commit: `operator` holds
`requestlog.read` because, per the migration that grants it, "operators debug
failing applies and ACME orders". They could not. An apply fails on the NODE,
and the node reports that over its own API key, so the row carrying the
diagnosis has `user_id IS NULL` - and own-rows-only scoping hid it from exactly
the role the grant was written for. Scoping now admits agent rows alongside the
caller's own. Deliberately keyed on `target = 'agent'` rather than `user_id IS
NULL`: anonymous traffic is not agent traffic, so failed logins and their
usernames, and unauthenticated probes, stay admin-only.
Verified end to end through the real middleware: a successful agent poll is
dropped, a 422 from config-validation-failed is kept, operator and anonymous
calls are unaffected, and flipping the setting on restores the old behaviour.
Applies PR #59 by Mustafa Ulukaya (github.com/taylanbakircioglu/haproxy-openmanager/pull/59,
head ef26860) as authored, with only the merge conflicts resolved. Behavioural
gaps found in review are closed by the follow-up commits on this branch rather
than by rewriting the contribution.
One queryable timeline covering both directions: every inbound API call
(including GETs and 4xx/5xx) with user, client IP, status, duration and
redacted, size-capped bodies; and every outbound HTTP call the backend makes,
tagged with who it went to. Outbound rows inherit the inbound request's id, so
one operator action and the CA/DNS calls it triggered read as a single trace.
Conflict resolution (the branch was cut at v1.10.3, this tree is v1.10.14):
* SCHEMA_VERSION: 11 -> 12, NOT the 11 the branch proposed. 11 was taken in the
meantime by v1.10.4 (vip_discoveries). Landing this as 11 would be silently
inert: run_all_migrations() returns early on `applied_version >=
SCHEMA_VERSION`, so every database already at 11 skips the whole sequence and
gets neither request_logs nor the requestlog.* permissions, while a fresh
install gets both. The branch's own test asserts `>= 11`, so it still holds.
* services/acme_diagnostics.py: the branch instrumented a `session.head(...)`
probe, which is what that function did when it was cut. It has since become a
GET that classifies the response body, because a status code alone cannot
tell a working challenge endpoint from a SPA catch-all answering 200 with
index.html. Taking the branch's side would reintroduce that bug, so the GET
probe is kept and the span wraps it. The span records the classification, not
the body: `_PROBE_BODY_LIMIT` is 64 KB of a third party's page and storing it
would put an arbitrary remote document in the audit table per probed domain.
* backend/version.json, frontend/package.json: 1.11.0, release date moved to
the date this actually ships.
* README.md, UPGRADE_GUIDE.md: the v1.11.0 sections are added above the
existing entries; every note from v1.10.4 through v1.10.14 is preserved.
Schema: one new table (request_logs) plus its settings seed. No existing table
altered, no agent or rendered-config change. As with every SCHEMA_VERSION bump,
the four built-in roles are re-seeded to their defaults - export role
customizations before upgrading.
Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered) or the
`enabled` toggle in Settings -> Request Log.
The deploy report is the server's only evidence that a member node applied its
keepalived.conf, and it was sent on the write path alone. Once the rendered
config was on disk the agent took the idempotency early return every cycle and
never reported again, so a single lost report - a backend restart, a 5xx, a
network blip - left the VIP reading SYNCING with an empty "Last ack" forever
while the node was demonstrably running the right config. Nothing would ever
reconcile the two; the only escape was to change the rendered config so the
agent wrote it again, which means touching a live VIP to fix a display problem.
The agent now re-asserts its state on that path too: one request per node per
poll cycle (~2.5 min), nothing written, keepalived not reloaded.
This gap dates from the original HA/VIP work rather than this release series;
it only became visible when acknowledgements were dropped for an unrelated
reason. A test pins that both daemon copies report BEFORE the early return,
since placing it after would silently restore the old behaviour.
Verified end to end on a real HA pair: discovery, instance-based adoption of
both nodes, PENDING, Apply, agent pull, the validation gate, the hash-pinned
takeover, the acknowledgement, and retirement of the one-shot authorisation.
Two agent-side fixes found while taking the VIP adoption flow through a real
HA pair, released together.
1. A VALID CONFIG WAS REJECTED BY ITS OWN WARNING (v1.10.12)
Before writing a rendered keepalived.conf the agent validates it with
keepalived -t and, on failure, keeps the running config and does not restart
keepalived. That fail-safe is right, but it treated ANY non-zero exit as
invalid - and keepalived's config-test exit code does not separate fatal from
benign. Measured on 2.2.8:
clean config ..................... 0
auth_pass longer than 8 chars .... 5 "Truncating auth_pass to 8 characters"
missing '}' ...................... 5 "There are 1 missing '}'s"
unknown keyword .................. 5 "Unknown keyword '...'"
script without script_security ... 6 "SECURITY VIOLATION ..."
Exit 5 covers both a harmless truncation and a broken file, so a VRRP password
over eight characters was enough to block every apply - including on a node
whose own running config emits the same warning and had been serving the VIP
for weeks. Accepting exit 5 would have accepted broken configs, so the gate now
judges the OUTPUT: known-benign messages are dropped and anything remaining
still fails. It fails CLOSED - an unrecognised message, or a non-zero exit with
no readable output at all, is fatal - and the filter is an allowlist, never a
denylist. The agent also reports what keepalived said, in its log and in the
status the HA/VIP page shows; discarding it left a correct refusal that nobody
could act on.
2. EVERY DEPLOY ACKNOWLEDGEMENT WAS DROPPED
The takeover-retirement clause on POST /agents/{name}/keepalived-status reused
one placeholder for both the assignment `last_deploy_hash=$n` and the
comparison inside its CASE. PostgreSQL types a placeholder per USE, so it came
out as text in one and character varying in the other and asyncpg rejected the
statement with AmbiguousParameterError. The whole UPDATE never ran, so no
member recorded an acknowledgement: VIPs sat at SYNCING with an empty Last ack
while the nodes were verifiably running the config, and teardown acks were lost
the same way. The hash now has its own placeholder, compared only against the
column.
Verified against real keepalived and a real PostgreSQL rather than by
inspection, including on busybox and bash 3.2, and a test asserts every $n in
those statements is bound exactly once.
The tag and the disabled Adopt button were computed by two separate ladders and
could disagree. Seen on a live pair: one node's keepalived.conf had an
unbalanced brace, so it was excluded from the instance; its partner was then
tagged "MASTER missing" - technically true, because the unreadable node's
state MASTER had not been counted - while the actual reason (the peer cannot be
taken over, so adopting would strand it) sat only in the button's tooltip. The
label pointed the operator at the wrong node.
groupState now makes ONE ordered decision and returns the label, its colour and
the reason together, so the tag can never describe a different condition than
the one disabling the button. A group held up by a node that references the
same address but cannot be adopted with it reads "blocked by peer"; two MASTERs
is distinguished from none.
Display only: the endpoint's checks and refusals are untouched.
Backend suite: 1366 passed, 152 skipped. Frontend build clean.
Since the panel groups a VRRP instance into one row, the blocker lists of all
its members are merged - and every node reports the SAME problems about the
SAME shared config. A two-node pair therefore showed each issue twice, in the
Adoptable tooltip and in the adopt dialog. Plain de-duplication does not
collapse them because the two files report different line numbers for the same
directive.
mergeBlockers keys on the message with a leading "line N:" stripped and keeps
the first occurrence, so each distinct problem appears once while the text the
operator reads still carries a line reference.
Display only: the endpoint already evaluated the combined set across every
node, and what it accepts or refuses is unchanged.
Backend suite: 1366 passed, 152 skipped. Frontend build clean.
Three findings from a second pass over the adoption flow, all of the same
class: something real leaving the set silently.
1. STRANDING. _collect_instance_participants can only match a node it can
READ, that is ENABLED, and that is in the SAME pool. Each of those is a door
a genuine member of the VRRP group leaves through without a word, and the
nodes that remain are rewritten while it keeps serving the same address from
an unmanaged config. Found on a live pool: one node of a pair had an
unclosed vrrp_instance block, so it parsed to nothing while its partner
parsed cleanly.
Rather than guard each door, ask the question directly: does any reported
keepalived.conf mention THIS virtual address without being one of the nodes
we are about to adopt? Refuses naming the node and the reason. Scoped on the
address so an unrelated file elsewhere cannot block every adoption, and
excluding nodes already under management (a standing VIP, or our ownership
marker) because those are not stranded.
2. SILENT NORMALISATION. prefix_length, unicast/multicast mode, HAProxy
tracking and the VRRP password are stored ONCE on the VIP and re-rendered
onto EVERY member, so whichever node was clicked imposed its settings on the
others. prefix_length is the sharpest: the design refuses to GUESS a netmask
for a live VIP, and copying one node's netmask onto another is that same
change wearing a different hat. All four must now agree, with both values
named in the refusal. The VRRP secret is compared by decrypting each node's
token - Fernet is non-deterministic, so ciphertexts cannot be compared - and
a token that will not decrypt is an error rather than an assumed match.
3. THE TAKEOVER AUTHORISATION WAS NOT ONE-SHOT. takeover_expected_hash is the
permission to overwrite a keepalived.conf that lacks our ownership marker.
It was written at adoption and never cleared, so it stayed valid for that
file content indefinitely: restoring the pre-adoption file would have been
overwritten again with no fresh human approval. It is now retired when a
member acknowledges our rendered config, gated on the acked hash matching
applied_config_hash so a partial or failed deploy never drops it and leaves
the VIP unable to converge.
The panel applies the stranding rule too, so the Adopt button is disabled with
the reason instead of letting the operator click into a 422.
No schema change, no agent change, no API-shape break.
Backend suite: 1366 passed, 152 skipped. Frontend build clean.
Four defects found while tracing the adoption flow end to end after v1.10.4
reached a live HA pair.
B3/B4 (one root, one fix). Adoption took only the node whose row was clicked:
- adopting the BACKUP alone produced a VIP that apply always rejects, because
apply requires exactly one MASTER member;
- adopting the MASTER alone left the peer unmanaged, and adopting it
afterwards hit the VRID-collision guard with 409, so a pair could never be
completed from the panel;
- on a UNICAST instance the single-member render dropped the unicast block
entirely (render_keepalived_conf emits it only when peer_ips is non-empty),
so keepalived fell back to multicast on the adopted node while its peer
stayed unicast. They stop seeing each other and BOTH claim the VIP.
Adoption now resolves the whole instance via _collect_instance_participants,
keyed on (virtual_router_id, virtual address) - the same key keepalived uses to
group nodes. Each participant becomes a member with the role, priority and
interface its own file declares, and its own one-shot takeover hash, so the
per-node overwrite guard is unchanged. It refuses, naming the reason, when the
group has other than one MASTER, when advert_int differs across nodes, when a
declared unicast peer is not among the nodes being adopted, or when a node is
already in a live VIP - the one-active-VIP-per-agent rule that create/update
enforce via _validate_members_against_pool and adoption never called.
B1. The Apply Management "View Change" regex matched vip-(create|update|delete)
only, so an adopt version fell through to the generic HAProxy diff and rendered
the cluster's whole haproxy.cfg as removed. Display-only, but alarming. A test
now asserts every action _stage_vip_version can stage is in that alternation.
B2. Rejecting an adoption hid the node from the panel permanently:
vip_discoveries.adopted_vip_id is write-once, a VIP is only ever soft-deleted so
the column's ON DELETE SET NULL never fires, and the agent does not re-report a
file whose hash has not changed. Rather than clearing the column on each path,
adoptability is derived from whether the linked VIP is still active, which
self-heals reject, undo-reject and approved teardown alike.
The panel now lists one row per instance instead of per node, and the adopt
dialog names every node that will be taken over. Blockers are aggregated across
all of them, matching what the endpoint checks.
No schema change, no agent change, no API-shape break: /api/vip/discoveries
gains a derived field and /api/vip/adopt keeps its request body.
Backend suite: 1359 passed, 152 skipped. Frontend build clean (no new lint
warnings in VIPManagement.js).
The page ignored the cluster picker in the header. Both the VIP table and the
v1.10.4 "Unmanaged keepalived detected" panel queried the whole fleet, so on an
install with more than one cluster the lists showed every cluster's nodes at
once and did not change when the selection did - the panel looked stuck on one
cluster's keepalived.
- GET /api/vip/discoveries takes the same optional cluster_id the VIP list
already took, mapped to the cluster's pool via haproxy_clusters exactly like
list_vips does. Omitting it still returns the whole fleet, so no existing
caller changes behaviour.
- VIPManagement reads selectedCluster from ClusterContext (it only took the
cluster list before) and sends cluster_id on both fetches. The fetch callbacks
depend on the scope, so switching cluster refetches instead of showing stale
rows.
- tests/test_vip_discoveries_scope.py pins both defects this endpoint has had:
that the route reaches its own handler rather than being parsed as a vip_id
(asserting != 422 specifically, since the repo's generic endpoint-auth tests
accept 422 alongside 401/403 and therefore could not catch it), and that the
cluster filter resolves cluster -> pool and short-circuits when absent.
Behaviour change worth calling out: the VIP table is now scoped to the selected
cluster where it was fleet-wide before. The API still serves the fleet-wide view
to any caller that omits cluster_id.
Backend suite: 1345 passed, 152 skipped. Frontend production build clean.
Follow-up to #58. The "Unmanaged keepalived detected" panel it shipped never
appeared on any deployment. GET /discoveries was declared after GET /{vip_id} in
routers/vip.py, and FastAPI matches routes in declaration order, so every request
for the discovery list was routed into get_vip, which takes vip_id: int and
rejected "discoveries" with 422 before list_vip_discoveries ever ran.
The failure was completely silent. The agents reported their configs correctly,
the rows landed in vip_discoveries, and the HA/VIP page treats any non-OK
response as "nothing to show" - so the feature was invisible with no error in
any log. Confirmed against a real fleet: two discovery rows present in the
database, one with a parsed candidate, and an empty panel.
- move list_vip_discoveries above the /{vip_id} routes, with a comment stating
the ordering requirement
- add tests/test_router_path_shadowing.py: a static source scan that fails if
any literal path in any router is declared after a parameterised route that
would swallow it. The whole router tree is clean; the detector is itself
tested against the pre-fix ordering so the guard cannot pass vacuously.
POST /adopt is unaffected: no POST /{vip_id} route exists.
Also carries the release metadata for #58 (v1.10.4) and #60 (v1.10.5), which
are published together with this fix rather than as separate artifacts.
No schema, API-shape, frontend or agent change. Data reported under the earlier
code is not lost - existing rows show up as soon as this backend is deployed,
with no agent action needed.
Correcting a wrong ACME challenge backend was impossible without a shell, and
even with one the correction did not reach the nodes.
The mint gate only fired when `acme_enabled` flipped. `acme_backend_url` was
written to the DB and minted nothing, so Apply answered "No pending changes to
apply" and the nodes kept the old address forever. It is now decided by
comparing the rendered `server _acme_mgmt` line against the active version —
the one line that answers "would the nodes talk to a different address?".
Comparing whole configs would flag every unrelated pending edit.
The field had no UI at all. Added to the cluster form with validation that
mirrors the backend rules, and keyed on `model_fields_set` so clearing it
reverts to the global setting — with a plain `is not None` test an empty box is
indistinguishable from "not submitted", so a value could never be removed.
Validation is asymmetric on purpose (utils/acme_backend_url):
- at the write boundary, reject what cannot express a reachable target —
including the two silent traps: a scheme-less value became `localhost`, and
an out-of-range port raised inside the generator and destroyed the config
- at render time, never reject. The shipped defaults are themselves loopback,
so refusing to render would make every acme_enabled cluster unappliable,
including for changes unrelated to ACME. Problems are logged and surfaced.
The port-less default stays 8080 rather than moving to HTTP's 80: the bundled
compose publishes nginx on 8080, so installs relying on it work today and the
first sign of breaking them would be the unattended renewal loop months later.
The omission is warned about instead.
RFC1918 is allowed and is usually the right answer here, and no DNS resolution
is performed — both deliberate departures from utils/ssrf_guard, whose policy
is the opposite of what this address needs. What the management host can
resolve says nothing about what the HAProxy node can reach.
Diagnostics stop reporting success on a dead path:
- check_port80 uses GET instead of HEAD and classifies the body. A proxy that
has lost its /.well-known/acme-challenge/ location serves its SPA with HTTP
200, which `status in (200, 404)` accepted as healthy. Warnings also surface
when other domains pass, which previously hid the most diagnostic outcome.
- check_routing filters `mode`, joins `acme_enabled` and reads the APPLIED
config instead of counting database rows, and reports a loopback target.
- every new condition is `warn`, never `fail`: the site wizard blocks submit on
any fail, so a new failing condition would lock every install on upgrade day.
Also: normalise `frontends.mode` once per frontend. It is nullable, and the
raw value was interpolated into `mode {}`, emitting a literal `mode None` that
HAProxy rejects — taking down the whole cluster config. The ACME gate and the
backend-mode check now read the same normalised value.
And stop hardcoding PUBLIC_URL / MANAGEMENT_BASE_URL in docker-compose, which
silently ignored the operator's .env and made the wrong default load-bearing.
Until now the only record of what happened was `user_activity_logs`, which
stores non-GET 2xx operations with no bodies. When something failed you could
see that a counter went up, never what was sent or what came back.
This adds one queryable timeline covering both directions:
- inbound: every API call, including GETs and including 4xx/5xx, with the
user, client IP, status, duration and — redacted, size-capped — the request
and response bodies.
- outbound: every HTTP call the backend makes, tagged with who it went to
(ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME
diagnostics probe).
Outbound rows inherit the inbound request's id, so one operator action and the
CA/DNS calls it triggered read as a single trace: opening a failed "Request
Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath.
Implementation notes:
- Capture is a pure-ASGI middleware that TEES the request and response streams
rather than draining them. `await request.body()` inside a BaseHTTPMiddleware
would consume the receive channel and break the raw-body agent heartbeat
handler. Registered last so it is outermost: it then sees the final
client-visible response and seeds correlation_id_context before the error
handler reads it.
- Rows are written by a batching background writer with a bounded queue, so the
request path never awaits the database and a saturated logger drops rows
visibly (surfaced on the page) instead of blocking. Redaction runs on the
writer, off the request coroutine.
- Secrets never land: headers are an allowlist with Authorization/Cookie kept
only as a presence marker; body keys and value shapes are redacted
(passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME
JWS request body is never stored, because a stored protected+signature pair
is a replayable credential — a summary is logged instead; DNS-provider errors
record only the exception type; the ACME HTTP-01 challenge endpoint is
excluded so key_authorization is never captured.
- Retention is operator-configurable in Settings -> Request Log: separate day
counts for successful and failed rows (7 / 30) plus a hard row cap (500k),
whichever is reached first. Pruned in batches under a Postgres advisory lock,
with the day counts bound as parameters, never interpolated.
- New permissions requestlog.read / requestlog.manage. super_admin and
security_admin get both, operator gets read, viewer gets neither.
Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION
10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config
change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered)
or the `enabled` toggle in Settings.
Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend +
17 frontend passing.
The page came up empty on a fleet that already runs keepalived, with nothing to
explain why. It now lists the nodes whose keepalived.conf the agent found and
deliberately left alone, in a section separate from managed VIPs so the
distinction is visible: OpenManager is not managing these.
Each discovered vrrp_instance shows the address, VRID, and this node's own
role, priority and interface, plus whether it can be adopted. Adopt opens a
modal that states what will happen rather than just asking for confirmation:
which directives would be deleted on takeover (with an explicit tick to accept
that, disabled otherwise), which values were assumed from keepalived's
documented defaults rather than read from the file, and the config itself with
the VRRP password masked.
Blockers are split the same way the backend splits them, from the same marker
strings, so the button cannot offer an adoption the API would reject: a loss is
waivable with the tick, a missing prefix is resolvable by supplying it, and an
impossibility disables Adopt outright with the reason shown.
After a successful adopt the peers of the adopted instance are named, because
those nodes hold their own keepalived.conf and the VIP is not a complete VRRP
group until they are members too.
Follow-up to #57. The contributed tests render the whole ACMEAutomation tree
(antd Steps + Form + Select) and drive it through all three wizard steps, which
takes 4-9 seconds per test. Under jest's default 5s per-test limit two of them
failed, so `npm test` did not pass as shipped:
✕ an explicitly picked HTTP-01 account survives the step change and is what
gets submitted -> Exceeded timeout of 5000 ms
✕ the wildcard guard still applies on the Review step, where Submit lives
-> Exceeded timeout of 5000 ms
The PR's reported 5/5 holds only when the runner is invoked with an explicit
--testTimeout. Setting it in the file instead means the suite passes however it
is invoked, which matters because the frontend image build runs `npm run build`
and never the tests, so nothing else would have caught this.
Verified with the default runner (no flags) after the change: 5/5 pass.
Test-only. No production code touched.
Renders the real component and walks it through all three wizard steps, because
the bug these cover was invisible to any unit test: it only appeared once the
wizard advanced PAST the step that owns the account Select, since Form.useWatch
reports only currently-rendered fields.
Assertions describe behaviour rather than markup - the primary evidence is the
POST body (account_id paired with challenge_type), compared against a fixture
whose DNS-01 account is deliberately both the lower id and the older account,
which is the exact shape that made the UI default and the backend default
disagree.
Verified by running the suite against the pre-fix component: the picked-account
test reports challenge_type "dns-01" where "http-01" is expected, the Review
test shows "Active (dns@example.com) / Challenge Method: DNS-01 / DNS provider:
godaddy" for a chosen HTTP-01 account - the reported symptom reproduced - and
the wildcard-guard test finds Submit enabled. Four fail, one passes: the DNS-01
selection path, kept as a positive control because it worked before (the
default the wizard fell back to happened to be the DNS-01 account) and must
keep working after.
Adds src/setupTests.js with the ResizeObserver and matchMedia polyfills jsdom
lacks and Ant Design 5 needs before any Select can open.
With more than one account registered, picking an HTTP-01 account in Request
ACME Certificate still submitted a DNS-01 request, which the API rejected with
"The selected ACME account has no DNS provider configured for DNS-01."
Three faults compounded:
1. Form.useWatch reports only fields that are currently rendered. The account
Select lives on the Configuration step, so the moment the wizard advanced to
Review the watch read undefined and the wizard fell back to the default
account - even though the value was still in the form store. Both wizard
watches now pass preserve: true. The same fault silently disabled the
wildcard guard on Review, the one step where Submit lives.
2. The UI and the backend disagreed on which account is the default. The
backend takes the newest valid account (ORDER BY created_at DESC); the UI
took the first valid entry of a list ordered by id, i.e. the oldest - the
opposite account whenever the two differ. The wizard now resolves the same
account and sends account_id explicitly, so there is no guess left to
disagree about.
3. account_id was read from the form store while challenge_type came from the
reverted account object. Both are now derived from one resolved account, so
the pair can no longer describe two different accounts.
Also: the Review step showed the default account's address instead of the
chosen one, and Submit stayed enabled when the resolved account was
deactivated (the fallback can land on a non-valid account, and the Select
lists deactivated accounts).
Single-account installations are unaffected.
Three dark-mode defects reported on the Apply Management page, all the same
class of bug: light-mode colour literals hardcoded where theme tokens belong.
1. The "Pending Changes" box was painted background #fffbe6 with border
#ffe58f. In dark mode the text on top is light, so the version name,
timestamp and "View Change" link sat on a cream panel and were unreadable.
Measured contrast was 1.03:1; it is now 11.50:1 (secondary text 1.03:1 ->
7.02:1).
2. The added/removed rows in the View Change diff used #f6ffed/#52c41a and
#fff2f0/#ff4d4f, which stayed near-white inside the otherwise dark diff
panel. Now 5.49:1 (added) and 4.01:1 (removed), from 2.21:1 and 2.99:1.
3. The "Apply All Configuration Changes" confirm dialog came up white. This one
is not a colour literal: in Ant Design 5 the STATIC Modal.confirm / message /
notification APIs render into their own detached root and never see the app's
ConfigProvider, so they always fall back to the light algorithm. Registering
ConfigProvider.config({ holderRender }) once at the app root wraps that
detached root in the same ConfigProvider. Verified against the installed antd
5.29.3 source rather than assumed: config-provider/index.js sets
globalHolderRender, and modal/confirm.js wraps the dialog with it. This fixes
EVERY static dialog in the application — 12 components call Modal.confirm —
not just this page.
While in the file, six more instances of the same bug were fixed: the error
Alert border, the VIP pending-delete row, two ACME/pending version panels, the
applied-version panel and the agent-error recommendation box.
Light mode is byte-identical. Each token resolves under the default algorithm
to exactly the literal it replaced (colorWarningBg -> #fffbe6, colorSuccessBg ->
#f6ffed, colorErrorBg -> #fff2f0, colorInfoBg, colorErrorBorder, ...), so this
release can only change dark mode. Contrast was measured by resolving the real
design tokens under both algorithms and computing WCAG ratios, not by eye.
Note the diff rows were already low-contrast in LIGHT mode (2.21:1 and 2.99:1)
and remain so; that is the design system's own success/error pair and changing
it would alter the established light-mode appearance, so it is left alone.
holderRender is registered in an effect rather than during render, since
ConfigProvider.config() mutates antd module state; effects still run long
before a user can click anything that opens a static dialog.
Frontend only: no schema, no SCHEMA_VERSION bump, no API change, no environment
variable, zero agent impact. Backend suite unchanged at 1243 passed.
Closes the follow-up filed during the v1.9.0 CSR review. The private key of a
PENDING CSR is now Fernet-encrypted in the database instead of being stored as
a raw PEM.
Why this key specifically: it is the one key in the system that sits idle. It
is generated at CSR creation, waits for an external CA to sign the request
(days to weeks), and is destroyed the moment the signed certificate is
imported. It is never transmitted to an agent and never leaves the server.
ssl_certificates.private_key_content and the ACME order keys are deliberately
NOT covered, because agents must receive those in plaintext on every poll, so
encrypting them at rest buys nothing without an end-to-end redesign.
Implementation follows the pattern already used for the VRRP secret, TOTP
secrets and DNS provider credentials: a new utils/csr_key_crypto.py with its
own CSR_ENCRYPTION_KEY env var and its own HKDF info string
("csr-private-key-v1"), so rotating one secret class never affects another.
No schema change and deliberately NO SCHEMA_VERSION bump: the Fernet token
replaces the PEM inside the existing ssl_csrs.private_key_pem TEXT column. A
bump would re-run the migration sequence and re-seed the four built-in roles to
their defaults, which is a needless side effect for a storage-format change.
Backward compatible with no data migration. Rows written before this release
hold a raw PEM and are still read unchanged; the discriminator is exact rather
than a heuristic, since a Fernet token is base64url and can never contain the
"-----BEGIN" marker. Legacy rows drain naturally because a CSR's key copy is
NULLed on import.
A key that cannot be decrypted (SECRET_KEY rotated while CSR_ENCRYPTION_KEY was
unset) now fails with an explicit "delete this CSR and create a new one" error.
Previously that situation would have surfaced as the far more confusing
"certificate does not match this CSR's private key".
Also documents all four per-purpose encryption keys in .env.template. Only
VIP_ENCRYPTION_KEY was listed; MFA_ENCRYPTION_KEY and
DNS_PROVIDER_ENCRYPTION_KEY had been missing since v1.6.0 and v1.8.0.
Verified before release, on a corporate pre-production environment and locally:
- Full backend suite 1234 -> 1243 passed (+9 new tests), 0 failed.
- Against a real Postgres: a CSR created through the API stores a Fernet token
with no PEM header in the column, and imports successfully.
- Full 1.10.0 -> 1.10.1 -> 1.10.0 drill on one database volume. The upgrade
logs "Schema already at version 10 (>= 10); skipping migration run", so no
migration executes and the built-in roles are not re-seeded. A CSR created on
1.10.0 with a plaintext key imports successfully after the upgrade, which is
the backward-compatibility guarantee proven against a real row rather than a
mock.
- rsa-2048, rsa-4096 and ecdsa-p384 all round-trip through create, encrypt,
decrypt and import.
- Key derivation is stable across processes: two independent containers sharing
SECRET_KEY decrypt each other's tokens (required for UVICORN_WORKERS > 1 and
multi-replica deployments), while a different SECRET_KEY yields None rather
than a wrong key or an exception.
- Downgrade behaviour was measured, not assumed: 1.10.0 cannot parse the token
and fails with HTTP 500 "key parse failed (encrypted?)" rather than pairing a
wrong key. The rollback note states the measured behaviour.
- No CSR endpoint returns the key in any form: list and detail responses
contain neither a PEM nor a Fernet token.
Not changed here, from the issue's "worth folding in" list: the create rate
limit is not a concurrency guard, create_csr holds a pooled connection across
RSA key generation, detail=str(e) echoes internal error text (a repo-wide
convention), and is_global skips cluster validation in both routers/ssl.py and
routers/csr.py. None are storage concerns and each is a separate change.
New CSRManagement component as a third tab (deep-linkable via ?tab=csr):
- Create modal: name (path-traversal-safe client rules mirroring the
server), CN with wildcard support, SAN tag input, key algorithm select,
optional subject fields in a collapse panel. On success the view modal
opens immediately with the CSR PEM.
- View modal: subject/SAN summary, read-only CSR PEM with copy and a
Download .csr button (Blob download).
- Import modal: paste signed certificate + optional chain, usage type,
global/cluster scope with cluster multi-select, and an optional
certificate-name override for collisions that appeared after CSR
creation; SAN-drift warnings surface in a warning dialog.
- Duplicate action pre-fills the create modal (forceRender so the form
accepts values before first open); delete confirm spells out that a
pending CSR key is destroyed permanently.
- SSL certificate table now renders a distinct CSR source tag next to
the existing Manual / Auto (ACME) tags.
Single-source version bump (backend/version.json) with frontend/package.json and
package-lock kept in sync (test_version_consistency). Marks the release that ships
the GHSA-7rhv / GHSA-3p5c / GHSA-3vh4 fixes.
Follow-up to the consolidated security bump:
- websocket-driver -> 0.7.5 (CRITICAL, message corruption; dev/build tooling)
- resolve-url-loader -> 5.0.0 (pulls postcss ^8), resolving the last postcss<8.5.10
instance (#12) and #2 at the source. Project uses no SASS, so resolve-url-loader
v4->v5 is inert at build time.
Verified: full production docker build succeeds on node 18; postcss now resolves
to a single 8.5.10 across the tree; deferred dev-only deps unchanged.
The manual Frontend editor, wizard and visual ACL builder hard-rejected the ACL
`-f <file>` flag while bulk import accepted it. Worse, a frontend imported with
an `-f` ACL could not be edited at all (422) until the ACL was dropped.
The original guard predated the fail-safe apply flow: the agent runs `haproxy -c`
before every reload, so a missing pattern file is rejected safely and the previous
config keeps running. Pattern files are operator-managed host files — the same
policy adopted for SPOE filter configs in v1.8.8.
- models: remove the 5 `-f` hard rejects (frontend acl/redirect/use_backend
validators + wizard string/dict-redirect guards); `$(`/backtick and X!X
contradiction guards unchanged
- routers/frontend: `_pattern_file_warnings` helper; non-blocking warning on
create + update responses listing referenced pattern files (empty when no
rule uses `-f` — zero noise)
- routers/config: bulk-import preview advisory listing pattern files per
frontend (cluster config-dir aware, next to the SPOE advisories)
- React: remove the FrontendManagement submit gate and SiteWizard step gate;
ACLRuleBuilder renders informational notes instead of errors and re-adds
`-f (pattern file on host)` to the flag dropdown; create path now renders
server warnings like update
- tests: 4 reject-pins inverted to accept-pins; new test_acl_pattern_file_allow.py
(accept/guards-kept/zero-noise/advisory); full suite green (1094 passed)
Bulk import / manual edit silently dropped `filter spoe engine ...` (Coraza WAF)
and frontend `log-format` because the parser recognised only a fixed directive
set. The regenerated config then missed the SPOE engine, so HAProxy failed with
"unable to find SPOE engine 'coraza' used by the send-spoe-group".
- parser: capture `filter` + `log-format`/`log-format-sd` into new ParsedFrontend fields
- db: additive nullable `log_format` + `filters` TEXT columns on frontends (SCHEMA_VERSION 8->9)
- generator: new `filter` bucket flushed before http-request rules so `filter` precedes
`send-spoe-group`; `log-format` kept in prelude
- bulk import: preview dict, change-detection, persist (create + merge-update); cluster-aware
SPOE pre-flight advisories (missing-filter + host-prerequisite) surfaced in the UI
- manual CRUD: full round-trip (get/create/update) incl. React form fields (no null-wipe)
- reject/rollback: restore the new columns; restore path + wizard helper kept in parity
- backend `option spop-check` recognised (suppresses spurious warning for coraza-spoa)
- tests: test_spoe_filter_import.py; full suite green (1079 passed)
The version shown in the UI (backend-sourced via /api/version) could lag the
real release. The canonical version lived at repo-root version.json, but the
backend image is built with context ./backend, so that file did not reach the
container unless a pipeline staged it; the backend then fell back to a hardcoded
constant in main.py that had to be bumped by hand and had drifted (it reported
1.8.4 after 1.8.5 and 1.8.6 shipped).
- version.json moves to backend/version.json (single source of truth), now
committed and co-located with main.py, so COPY . . bakes it into every image
directly - correct version in every deployment, no pipeline staging required.
- backend/main.py reads the co-located file; its in-code fallback is no longer a
real version ("unknown") so it can never silently drift again.
- docker-build.yml reads backend/version.json and drops the staging step;
docker-compose.localtest drops the stale root mount.
- backend/tests/test_version_consistency.py fails the build if main.py hardcodes
a real version, if the canonical file is missing/invalid, or if
frontend/package.json drifts from it.
Bumped to 1.8.7. No functional, schema, API, or agent change. Full suite green.
A user running the API on a 2-core/4GB host reported slow-feeling API
responses (Issue #35 follow-up). Review of the hot paths found no
pathological defect; the dominant factors are the single uvicorn worker
(one core serves all requests) and the constant agent-poll baseline
(4 requests per agent every 30s). Two zero-risk improvements:
- backend/Dockerfile: CMD now honors UVICORN_WORKERS, falling back to
WEB_CONCURRENCY and then 1. Flagless uvicorn natively honors
WEB_CONCURRENCY, so the fallback keeps any deployment that relied on it
byte-for-byte compatible; with the final default of 1 worker uvicorn runs
in-process exactly as before. >1 enables the multiprocess supervisor so
multi-core hosts can use all cores. Background tasks are already
multi-replica safe (FOR UPDATE SKIP LOCKED / advisory locks), as exercised
by the k8s HPA deployment (2-10 replicas). `exec` keeps uvicorn as PID 1
(clean SIGTERM, verified ~1s docker stop with 2 workers).
docker-compose.yml passes UVICORN_WORKERS through as empty-when-unset so a
user-set WEB_CONCURRENCY is never overridden; .env.template documents it.
- routers/agent.py heartbeat (by-name endpoint): the agent's
status/version/upgrade_status were read with three separate single-column
SELECTs against the same row; now one SELECT. Identical values and None
semantics (single consistent snapshot instead of three reads); saves two
round-trips per heartbeat per agent every 30s. The legacy by-id heartbeat
endpoint is untouched; the heartbeat API contract is unchanged for agents
of every version.
- README: new "Performance Tuning" section (worker/replica scaling, and how
to use the X-Response-Time header plus "Slow request detected" logs to
pinpoint slow endpoints).
Verification: full backend suite in docker green (1063 passed, 151 skipped;
also re-run by the runtime image build); worker-count expansion matrix
(unset->1, UVICORN_WORKERS=2->2, WEB_CONCURRENCY=3->3, both->UVICORN_WORKERS,
empty->fallback) all correct; default run confirmed single-process with
uvicorn as PID 1 and healthy API; UVICORN_WORKERS=2 confirmed parent + 2
workers, healthy API, clean shutdown; live heartbeats verified for
register + existing-agent paths AND degraded agents (no stats socket /
haproxy stopped / garbage stats CSV / unknown backend in server_statuses):
all return 200, agent row updates correctly, zero backend errors.
No schema, API, or agent changes.
The background order-completion task (complete_pending_acme_orders, 60s
cycle) failed on EVERY cycle since v1.8.0 with:
[ACME-COMPLETE] Error in completion task: syntax error at or near ")"
Root cause: the bounded DNS-01 retry OR-arm added to the atomic order-claim
query in v1.8.0 (13b65d9) carried one extra closing parenthesis, making the
whole SELECT invalid PostgreSQL. The claim is the task's first statement, so
the generic except swallowed it each minute and NO background ACME work ever
ran on v1.8.0-v1.8.4:
- orders were never claimed for finalize -> download -> save (http-01 too);
- advance_dns01_order never ran, so the DNS-01 TXT record was never
published - DNS-01 with an automated provider (e.g. Cloudflare) could
never validate (exactly the report in Issue #35);
- wizard-staged orders never left wizard_staged (same try block);
- retry_invalid_dns01 / reconcile_dns01_cleanup never executed;
- hourly-created renewal orders could never complete in the background.
Fix: drop the stray ')' (one line). Query semantics are unchanged.
Why the suite missed it: the unit tests mock asyncpg, so raw SQL never
reaches a real parser. Added a regression test that AST-scans the ACME
modules' SQL string literals (comments/quoted literals stripped) and fails
on unbalanced parentheses - it is red on the pre-fix tree and would have
caught the v1.8.0 regression at commit time. Scanned all six ACME modules:
this query was the only unbalanced SQL.
Verification: full backend suite in docker green (1062 passed, 151
skipped); the fixed query EXPLAINs cleanly on postgres:15; live localtest
run shows zero completion-task errors and a seeded pending order is claimed
("[ACME-COMPLETE] Claimed 1 order(s)"). Backend-only, no schema/API/agent
changes; fully backward compatible.
Reported-by: @tkkost (GitHub Issue #35)
The Linux/macOS agent installer could abort during "pre-installation cleanup"
(terminal showed `Killing processes matching: haproxy-agent` then `Killed`,
returning to the prompt) when the install script's own command line contained
"haproxy-agent". The cleanup killed processes via `pgrep -f "$pattern"` starting
with the bare string "haproxy-agent", which also matched the running installer's
own command line and a sudo/PAM ancestor that the $$/$PPID self-exclusion did not
cover, so the installer terminated itself before installing.
- linux_install.sh / macos_install.sh: the cleanup kill loop now targets ONLY
the installed agent - "$INSTALL_DIR/haproxy-agent" (the daemon binary path) and
the agent service/label ("haproxy-agent.service" / "com.haproxy.agent") - never
the bare "haproxy-agent" substring. Neither pattern can match the installer's
own command line. The redundant bare pattern is dropped (the service is stopped
separately, and the binary-path pattern still catches a running daemon).
- frontend (AgentManagement.js): name the downloaded scripts
install-agent-<platform>.sh / uninstall-agent-<platform>.sh (matching the
backend's suggested filename) - defense in depth so this cannot resurface.
Installer-only change. The running agent and its privilege model are unchanged
(it runs as root for HAProxy reload, config writes, keepalived, and self-upgrade).
The cleanup runs only on a full interactive install (gated by SKIP_TO_DAEMON), so
daemon mode, self-upgrade, and config/version apply are unaffected. Both agent
scripts kept in sync. Scripts parse on bash 4.2-5.2; full backend suite green.
Addresses #31.
A self-hosted agent could fail every heartbeat with HTTP 400
`Invalid JSON: Expecting property name enclosed in double quotes` when the
system-info block it collects came back empty on an unusual host. The agent
builds the heartbeat JSON as text, so an empty `$system_info` collapsed the
`$system_info,` line to a bare comma and broke the payload.
- Agent script (linux + macos, kept in sync): guard the fragment-form
register_agent and send_heartbeat builders so an empty system_info falls back
to a valid key and can never emit a bare comma. Uses the most portable bash
glob test (no POSIX class / pattern substitution; verified on bash 3.2-5.2 and
on Ubuntu/Debian/Rocky/Alpine/Amazon Linux). True no-op for healthy agents.
- Backend heartbeat endpoint: parse the body as-is first and only run the
malformed-JSON repair when parsing fails, so a valid heartbeat from any agent
version is byte-for-byte untouched. The repair (now a testable helper) recovers
a leading or doubled comma (the empty-system_info artifact) in addition to the
existing empty-value / trailing-comma fixes.
No agent version bump; self-upgrade and daemon mode are unaffected. Healthy
agents of every version behave identically. Full backend suite green.
Addresses #31.
ZeroSSL/Google account registration failed with
`malformed: The Replay Nonce could not be base64url-decoded`: the ACME client
(a process-wide singleton) kept a single anti-replay nonce shared across
certificate authorities, so a nonce issued by one CA could be sent to another,
and the auto-retry only covered `badNonce`.
- Scope the nonce per CA (self._nonce_by_dir keyed by directory_url): a nonce
from one CA is never sent to another; account registration always uses a fresh
nonce from the target CA.
- Broaden the 400 retry to also recover from the nonce-malformed rejection.
- Fix _b64url_decode padding (used for the EAB HMAC key).
Backend-only; HTTP-01 and Let's Encrypt are unaffected.
Addresses #35.
Follow-up fixes for the DNS-01 feature reported on #35:
- Cloudflare: sanitize the API token (strip surrounding quotes + any non
token68 chars) so a pasted token with quotes/spaces no longer fails with
"Invalid request headers"; verify-on-save shows a precise hint when it
cleaned the input. Covers the automated orchestrator path too.
- ZeroSSL/Google EAB: enter the EAB Key ID and HMAC Key per-account in the
Register Account dialog (falls back to the global Settings value when blank);
base64-validate the HMAC key; humanize the externalAccountRequired failure;
and preserve the deliberate 409/422 instead of downgrading them to 400.
- Apply Management: cluster ACME enable/disable changes now show under a
dedicated "ACME Challenge Routing" section, are counted in the Apply/Reject
dialogs, and Apply/Reject All process them (previously "Rejected 0 HA/VIP
change(s)") - consistent with every other entity. Reject rolls acme_enabled
back to the original via ORDER BY created_at ASC over the snapshot chain.
- getErrorMsg surfaces field-level validation messages.
Backward compatible (additive / strict superset; HTTP-01 unchanged).
Addresses #35.
Add ACME DNS-01 (TXT-record) validation alongside the existing HTTP-01,
for internal/isolated clusters with no public port 80 and for wildcard
certificates. Opt-in via a global kill-switch (default off); HTTP-01 is
byte-for-byte unchanged, with zero agent or rendered-config changes.
- Pluggable DNS provider interface (Manual + Cloudflare). Per-account
credentials are Fernet-encrypted at rest, verified on save, and never
returned by the API or written to logs/events/error_detail.
- Non-blocking per-cycle orchestrator: publish (CAS) -> propagation grace
(across cycles, no in-loop sleep) -> respond -> finalize/download, with a
bounded fresh-order retry chain (1 original + 3 retries) on propagation lag.
- Manual flow: user publishes the TXT record and confirms; manual DNS-01
cannot auto-renew unattended (auto-renew forced off and surfaced in the UI).
- Migration v8: additive, idempotent columns on letsencrypt_accounts/orders
and acme_challenges, plus a new letsencrypt_account_dns_credentials table.
- Challenge-type-aware diagnostics (port80/routing/DNS checks skipped for
DNS-01) and a DNS-01 event timeline in the order detail.
- Frontend: DNS-01 account + credentials management, cert wizard adaptation,
order-detail TXT records + verify, orders/renewal Method columns, and a
Settings kill-switch. README, release notes, and API docs updated.
Implements #35.
Patches the critical shell-quote advisory (quote() does not escape
newlines in object .op values, CVSS 8.1). shell-quote is a dev/build-only
transitive dependency (react-dev-utils / launch-editor) and is not present
in the production image or the browser bundle, so there is no runtime
exposure; this clears the alert and the dev-time risk. Lockfile-only change
verified to install with shell-quote resolving to 1.8.4.
Manage highly-available virtual IPs backed by Keepalived (VRRP) directly from the OpenManager
UI — no more SSHing into nodes to install/configure Keepalived by hand. Builds on the agent
pull-architecture: define the VIP centrally, click Apply, and the agents converge.
Highlights:
- New "HA / VIP" tab: create a virtual IP, pick a per-node interface, select which pool nodes
participate (MASTER/BACKUP roles + priorities); live MASTER/BACKUP/FAULT per node.
- On Apply, agents install & configure Keepalived (unicast VRRP, cloud-safe default) across the
major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine) with a
HAProxy health-check, so the VIP fails over automatically when HAProxy drops.
- Single-node (a managed floating IP without failover) and multi-node VRRP failover both work.
- VIP changes ride the standard Apply Management flow with the standard "View Change" diff.
- Approval-gated deletion (safety): deleting a running VIP is staged for approval and the VIP
keeps running, untouched, until you approve it — an agent never tears a VIP down without an
explicit human approval. Per-VIP Diagnostics view; opt-in package uninstall (only on nodes
where OpenManager installed it). A node already running a hand-managed Keepalived is detected
and never overwritten ("externally managed").
- Fully opt-in and backward compatible: nodes/clusters without a VIP are unaffected. Adds
vip_instances + vip_members tables (idempotent SCHEMA_VERSION bump; existing data and
passwords unaffected) and a `vip` RBAC permission group.
- Also includes a HAProxy config-generator robustness fix: auto-inject a stick-table when a
frontend uses a stick counter (track-sc / sc_*_rate) but declares none.
On-prem / L2 (VRRP) scope; the UI notes the cloud caveat.
Supersedes the v1.6.4 stable pin (1.30.2-alpine) with the mainline
patched release. No config, schema, or behavior changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bundled nginx reverse proxy was flagged for the nginx 'poolslip' advisory
(affected: mainline <=1.31.0; fixed: stable 1.30.2+ / mainline 1.31.1+). The
config-level mitigation (named capture groups instead of $1/$2 in rewrite) does
not apply — the product's nginx config (nginx/nginx.conf and the k8s configmap)
has no rewrite capture-group directives, only prefix locations + proxy_pass. So
the fix is the version: pin nginx:alpine -> nginx:1.30.2-alpine in
docker-compose.yml and k8s/manifests/10-nginx.yaml.
No config/schema/behavior change (nginx only reverse-proxies). Version bumped to
1.6.4 across all layers. Verified in Docker: nginx -v=1.30.2; nginx -t OK on both
the compose and production configmap configs; all proxied routes work through
nginx; no nginx errors.
A backend server toggled OFF (is_active=false) vanished from the UI with no way
to reactivate it: GET /api/backends honored include_inactive for backends but the
server sub-queries hardcoded 'AND is_active = TRUE'.
- get_backends: server sub-queries now honor include_inactive (default callers
unchanged); added last_config_status to the server payload so the UI can tell a
DISABLED server (re-enableable) from a DELETION (pending delete).
- toggle_server: persists an entity snapshot so an Apply-Management Reject rolls
back is_active (previously left the server stuck disabled).
- BackendServers.js: requests include_inactive, shows disabled servers with the
ON/OFF switch + an 'Inactive' tag, hides only DELETION-pending servers, and
keeps soft-deleted BACKENDS hidden (so include_inactive doesn't resurface them).
- Config generation unchanged: disabled servers stay '# DISABLED:' comments and
convert back to live lines when re-enabled.
Startup migration hardening (multi-replica / rolling-deploy safety): create_essential_tables
fails fast on lock contention and retries; run_all_migrations is serialized by a
session advisory lock and gated by a schema_migrations version marker, so an
already-current schema is skipped instead of issuing lock-heavy DDL that a serving
replica's traffic could block at startup. Idempotent and fail-open.
Version reported consistently across all layers (version.json, backend fallback,
frontend package) -> 1.6.3.
Assigning a HAProxy agent failed with '401: Authorization header missing'
on GET /api/clusters. A cluster-read hardening had made GET /api/clusters and
GET /api/clusters/{id} accept only a user JWT in the Authorization header;
agents authenticate with their agent token in the X-API-Key header, so the
token was never read.
Both endpoints now accept either a user JWT (Authorization) or an agent token
(X-API-Key via validate_agent_api_key), mirroring the existing dual-auth on
POST /api/agents/generate-install-script. Anonymous access is still rejected,
so the original hardening is preserved. The auth guard is placed before the
try block so the failure surfaces as a clean 401 (not the 500-wrapped-401 in
the report). Agent install scripts now consistently send the token via
X-API-Key (pre-flight cluster check on linux/macos, and macOS get_cluster_paths
which previously used the wrong Authorization: Bearer header).
Also normalizes the platform in the uninstall-script generator so macOS agents
(which report platform 'darwin') no longer get a 400 from
GET /api/agents/generate-uninstall-script/darwin.
version 1.6.0 -> 1.6.2.
taylanbakircioglu