Three findings from a second pass over the adoption flow, all of the same
class: something real leaving the set silently.
1. STRANDING. _collect_instance_participants can only match a node it can
READ, that is ENABLED, and that is in the SAME pool. Each of those is a door
a genuine member of the VRRP group leaves through without a word, and the
nodes that remain are rewritten while it keeps serving the same address from
an unmanaged config. Found on a live pool: one node of a pair had an
unclosed vrrp_instance block, so it parsed to nothing while its partner
parsed cleanly.
Rather than guard each door, ask the question directly: does any reported
keepalived.conf mention THIS virtual address without being one of the nodes
we are about to adopt? Refuses naming the node and the reason. Scoped on the
address so an unrelated file elsewhere cannot block every adoption, and
excluding nodes already under management (a standing VIP, or our ownership
marker) because those are not stranded.
2. SILENT NORMALISATION. prefix_length, unicast/multicast mode, HAProxy
tracking and the VRRP password are stored ONCE on the VIP and re-rendered
onto EVERY member, so whichever node was clicked imposed its settings on the
others. prefix_length is the sharpest: the design refuses to GUESS a netmask
for a live VIP, and copying one node's netmask onto another is that same
change wearing a different hat. All four must now agree, with both values
named in the refusal. The VRRP secret is compared by decrypting each node's
token - Fernet is non-deterministic, so ciphertexts cannot be compared - and
a token that will not decrypt is an error rather than an assumed match.
3. THE TAKEOVER AUTHORISATION WAS NOT ONE-SHOT. takeover_expected_hash is the
permission to overwrite a keepalived.conf that lacks our ownership marker.
It was written at adoption and never cleared, so it stayed valid for that
file content indefinitely: restoring the pre-adoption file would have been
overwritten again with no fresh human approval. It is now retired when a
member acknowledges our rendered config, gated on the acked hash matching
applied_config_hash so a partial or failed deploy never drops it and leaves
the VIP unable to converge.
The panel applies the stranding rule too, so the Adopt button is disabled with
the reason instead of letting the operator click into a 422.
No schema change, no agent change, no API-shape break.
Backend suite: 1366 passed, 152 skipped. Frontend build clean.
Four defects found while tracing the adoption flow end to end after v1.10.4
reached a live HA pair.
B3/B4 (one root, one fix). Adoption took only the node whose row was clicked:
- adopting the BACKUP alone produced a VIP that apply always rejects, because
apply requires exactly one MASTER member;
- adopting the MASTER alone left the peer unmanaged, and adopting it
afterwards hit the VRID-collision guard with 409, so a pair could never be
completed from the panel;
- on a UNICAST instance the single-member render dropped the unicast block
entirely (render_keepalived_conf emits it only when peer_ips is non-empty),
so keepalived fell back to multicast on the adopted node while its peer
stayed unicast. They stop seeing each other and BOTH claim the VIP.
Adoption now resolves the whole instance via _collect_instance_participants,
keyed on (virtual_router_id, virtual address) - the same key keepalived uses to
group nodes. Each participant becomes a member with the role, priority and
interface its own file declares, and its own one-shot takeover hash, so the
per-node overwrite guard is unchanged. It refuses, naming the reason, when the
group has other than one MASTER, when advert_int differs across nodes, when a
declared unicast peer is not among the nodes being adopted, or when a node is
already in a live VIP - the one-active-VIP-per-agent rule that create/update
enforce via _validate_members_against_pool and adoption never called.
B1. The Apply Management "View Change" regex matched vip-(create|update|delete)
only, so an adopt version fell through to the generic HAProxy diff and rendered
the cluster's whole haproxy.cfg as removed. Display-only, but alarming. A test
now asserts every action _stage_vip_version can stage is in that alternation.
B2. Rejecting an adoption hid the node from the panel permanently:
vip_discoveries.adopted_vip_id is write-once, a VIP is only ever soft-deleted so
the column's ON DELETE SET NULL never fires, and the agent does not re-report a
file whose hash has not changed. Rather than clearing the column on each path,
adoptability is derived from whether the linked VIP is still active, which
self-heals reject, undo-reject and approved teardown alike.
The panel now lists one row per instance instead of per node, and the adopt
dialog names every node that will be taken over. Blockers are aggregated across
all of them, matching what the endpoint checks.
No schema change, no agent change, no API-shape break: /api/vip/discoveries
gains a derived field and /api/vip/adopt keeps its request body.
Backend suite: 1359 passed, 152 skipped. Frontend build clean (no new lint
warnings in VIPManagement.js).
The page ignored the cluster picker in the header. Both the VIP table and the
v1.10.4 "Unmanaged keepalived detected" panel queried the whole fleet, so on an
install with more than one cluster the lists showed every cluster's nodes at
once and did not change when the selection did - the panel looked stuck on one
cluster's keepalived.
- GET /api/vip/discoveries takes the same optional cluster_id the VIP list
already took, mapped to the cluster's pool via haproxy_clusters exactly like
list_vips does. Omitting it still returns the whole fleet, so no existing
caller changes behaviour.
- VIPManagement reads selectedCluster from ClusterContext (it only took the
cluster list before) and sends cluster_id on both fetches. The fetch callbacks
depend on the scope, so switching cluster refetches instead of showing stale
rows.
- tests/test_vip_discoveries_scope.py pins both defects this endpoint has had:
that the route reaches its own handler rather than being parsed as a vip_id
(asserting != 422 specifically, since the repo's generic endpoint-auth tests
accept 422 alongside 401/403 and therefore could not catch it), and that the
cluster filter resolves cluster -> pool and short-circuits when absent.
Behaviour change worth calling out: the VIP table is now scoped to the selected
cluster where it was fleet-wide before. The API still serves the fleet-wide view
to any caller that omits cluster_id.
Backend suite: 1345 passed, 152 skipped. Frontend production build clean.
Follow-up to #58. The "Unmanaged keepalived detected" panel it shipped never
appeared on any deployment. GET /discoveries was declared after GET /{vip_id} in
routers/vip.py, and FastAPI matches routes in declaration order, so every request
for the discovery list was routed into get_vip, which takes vip_id: int and
rejected "discoveries" with 422 before list_vip_discoveries ever ran.
The failure was completely silent. The agents reported their configs correctly,
the rows landed in vip_discoveries, and the HA/VIP page treats any non-OK
response as "nothing to show" - so the feature was invisible with no error in
any log. Confirmed against a real fleet: two discovery rows present in the
database, one with a parsed candidate, and an empty panel.
- move list_vip_discoveries above the /{vip_id} routes, with a comment stating
the ordering requirement
- add tests/test_router_path_shadowing.py: a static source scan that fails if
any literal path in any router is declared after a parameterised route that
would swallow it. The whole router tree is clean; the detector is itself
tested against the pre-fix ordering so the guard cannot pass vacuously.
POST /adopt is unaffected: no POST /{vip_id} route exists.
Also carries the release metadata for #58 (v1.10.4) and #60 (v1.10.5), which
are published together with this fix rather than as separate artifacts.
No schema, API-shape, frontend or agent change. Data reported under the earlier
code is not lost - existing rows show up as soon as this backend is deployed,
with no agent action needed.
GET /api/vip/discoveries lists what the agents found; POST /api/vip/adopt turns
one vrrp_instance into a managed VIP using the values from the node's own file
instead of retyping them. The VIP is created PENDING like any other, so nothing
reaches the node until it is applied from Apply Management.
Adoption replaces the operator's file with our render, so the gate is the
feature. Blockers fall into three kinds and only two are resolvable:
- a LOSS ("our renderer cannot reproduce this, so adopting would delete it")
can be accepted explicitly - that is an informed choice about a notify hook
or an LVS section;
- an UNKNOWN prefix length can be supplied, because picking a netmask for a
live VIP would change its routing;
- anything else is an IMPOSSIBILITY, not a loss: an absent virtual_router_id,
a fractional advert_int, an unsupported auth_type. No flag waves those
through.
That rule now lives in one place, remaining_blockers(), so the endpoint and the
UI cannot drift apart - and it is unit-testable, which matters because getting
it wrong destroys a working config.
Adoption keeps the VRRP identity it found: unlike create_vip, which allocates
the next free VRID, a VRID already used in the pool is a hard 409. Silently
renumbering would put the adopted node in a different VRRP domain from the
peers that still run the original config.
The member row records the reporting node's own role, priority and interface,
and carries the one-shot takeover hash. The response returns the instance's
unicast peers, because those nodes hold their own keepalived.conf and have to
be adopted or added as members before the render describes a complete group.
Manage highly-available virtual IPs backed by Keepalived (VRRP) directly from the OpenManager
UI — no more SSHing into nodes to install/configure Keepalived by hand. Builds on the agent
pull-architecture: define the VIP centrally, click Apply, and the agents converge.
Highlights:
- New "HA / VIP" tab: create a virtual IP, pick a per-node interface, select which pool nodes
participate (MASTER/BACKUP roles + priorities); live MASTER/BACKUP/FAULT per node.
- On Apply, agents install & configure Keepalived (unicast VRRP, cloud-safe default) across the
major distros (Debian/Ubuntu, RHEL/CentOS/Alma/Rocky, Fedora, SUSE/openSUSE, Alpine) with a
HAProxy health-check, so the VIP fails over automatically when HAProxy drops.
- Single-node (a managed floating IP without failover) and multi-node VRRP failover both work.
- VIP changes ride the standard Apply Management flow with the standard "View Change" diff.
- Approval-gated deletion (safety): deleting a running VIP is staged for approval and the VIP
keeps running, untouched, until you approve it — an agent never tears a VIP down without an
explicit human approval. Per-VIP Diagnostics view; opt-in package uninstall (only on nodes
where OpenManager installed it). A node already running a hand-managed Keepalived is detected
and never overwritten ("externally managed").
- Fully opt-in and backward compatible: nodes/clusters without a VIP are unaffected. Adds
vip_instances + vip_members tables (idempotent SCHEMA_VERSION bump; existing data and
passwords unaffected) and a `vip` RBAC permission group.
- Also includes a HAProxy config-generator robustness fix: auto-inject a stick-table when a
frontend uses a stick counter (track-sc / sc_*_rate) but declares none.
On-prem / L2 (VRRP) scope; the UI notes the cloud caveat.
taylanbakircioglu