Compare commits

...

2 Commits

Author SHA1 Message Date
taylanbakircioglu 78641d87ff feat(vip): let a VIP carry additional addresses (issue #73, v1.12.0)
A VIP held exactly one address. It can now carry up to 19 additional IPv4
addresses (with prefix) in the same virtual_ipaddress block, on the same interface,
failing over together with the virtual IP.

- vip_instances.additional_ips (JSONB, default []); an empty list renders exactly
  as before. Create/edit accept the list (omitted keeps, [] clears); it is part of
  the applied snapshot and restored by Reject and undo.
- Every address write takes one advisory lock and checks the addresses against
  node IPs and every other active VIP's primary and additional addresses.
- Adoption accepts an instance with several addresses: the ones after the first
  become additional addresses when they render back exactly (IPv4, explicit
  prefix, the instance's interface, no repeats); the peers must agree on them.
- UI: an "Additional addresses" list in the VIP form and a "+N" tag in the table.

v1.12.0: README, upgrade notes (delta migration, rolling-deploy and downgrade
rules, per-node live state), version files.

Fixes #73
2026-10-17 13:05:29 +03:00
taylanbakircioglu 349434fc4d feat(vip): let a node belong to several keepalived VIPs (issue #73)
A node could be a member of one VIP only: the form refused a second one, and the
delivery endpoint served each agent a single VIP's keepalived.conf.

- Delivery merges the node's applied VIPs into the one file it deploys: global_defs
  once, then per VIP a vrrp_script chk_haproxy_<id> (each keeps its own failover
  weight) and its vrrp_instance, ordered by VIP id. A node with one VIP gets the same
  bytes and hash as before, so existing nodes do not rewrite or reload.
- One ack converges every VIP in the merged file; a deleted VIP's membership is
  marked disabled once the node runs a file without it.
- A snapshot that cannot be merged leaves the node alone (status compose_error,
  members show the reason). A database error during delivery answers "unavailable"
  instead of "not_configured", which on a managed node meant teardown.
- Adoption refuses a keepalived.conf with several vrrp_instance blocks (taking it
  over would remove the others), also for analyses stored by older versions and at
  Apply for an adoption made before the upgrade. Discoveries record the parser
  version; files analysed by an older one are re-reported once by v1.11.1+ agents.
- Schema 13 is a delta from 12 (two additive columns); built-in roles are not
  re-seeded on a database already at 12.

No agent change.
2026-10-17 13:04:38 +03:00
14 changed files with 959 additions and 106 deletions
+2
View File
@@ -1096,6 +1096,7 @@ The HA / VIP page manages **highly-available virtual IPs** backed by Keepalived
- **VIP dashboard**: lists each virtual IP with its pool, VRID, per-node interface, and a **live MASTER / BACKUP / FAULT** column per node (refreshed every 30s from the existing keepalive-state heartbeat pipeline).
- **Creation form**: enter the virtual IP (+ prefix), pick the pool, and select which of the pool's HAProxy nodes participate — each with a network interface (from the node's reported interfaces), a VRRP role (MASTER/BACKUP) and a priority. VRID auto-allocates per pool; an optional VRRP secret can be set.
- **Hands-off automation**: on **Apply**, each member's `keepalived.conf` is rendered and delivered; the agent installs Keepalived if missing, writes the config + a HAProxy health-check (`vrrp_script` + `track_script`), validates with `keepalived -t`, and starts the service. When HAProxy drops on the active node, the health-check lowers VRRP priority and the **VIP fails over automatically** to a backup.
- **Several VIPs per node, several addresses per VIP** (v1.12.0): a node can belong to more than one VIP — each VIP is its own `vrrp_instance` (its own VRID, role and priority) and the node's `keepalived.conf` holds all of them. A VIP can also carry up to 19 **additional addresses** next to its virtual IP; they sit in the same `virtual_ipaddress` block, on the same interface, and fail over together.
**How it works (pull-based, isolated):**
- New tables `vip_instances` + `vip_members`; the agent polls `GET /api/agents/{name}/keepalived-config` (key-authenticated, agent-bound) and converges. It is fully **isolated** from the global HAProxy apply flow — VIP changes never regenerate `haproxy.cfg`.
@@ -2475,6 +2476,7 @@ Developed with ❤️ for the HAProxy community
## Release Notes
- **v1.12.0** (2026-10-16) — **Several keepalived VIPs on one node, and several addresses on one VIP**: a node could belong to only one VIP (the form refused a second one with "node already in VIP …") and a VIP held exactly one address. A node may now be a member of several VIPs: each VIP stays its own `vrrp_instance` with its own VRID, role and priority, and the delivery endpoint merges the node's applied VIPs into the one `keepalived.conf` the node deploys — one `global_defs`, then per VIP a `vrrp_script chk_haproxy_<id>` (each VIP keeps its own failover weight) and its `vrrp_instance`, ordered by VIP id. A node with a single VIP receives exactly the bytes and hash it received before, so existing installs see no change and no reload. A VIP can also carry up to 19 **additional addresses** (IPv4 + prefix) in the same `virtual_ipaddress` block and on the same interface; they are checked against node IPs and every other VIP's addresses under a lock, kept in the applied snapshot, and restored by Reject. The merged file passes `keepalived -t` with no output (which older agents require) on keepalived 2.0.19, 2.1.5, 2.2.4, 2.2.7 and 2.2.8; on 2.0.19, 2.1.5 and 2.2.8 a reload that adds a second VIP did not drop the first one's address, and a VIP with 20 addresses came up. If a node's VIPs cannot be merged, the node is left untouched and its members show the reason. A database error during delivery now answers a status the agent ignores instead of `not_configured`, which on a managed node meant teardown. **Adoption** refuses a `keepalived.conf` with several `vrrp_instance` blocks (taking it over would remove the others; earlier versions allowed it), now accepts an instance with several addresses (the extra ones become additional addresses), and asks agents to re-report files analysed by the older parser once. Schema `13` is a **delta** from `12`: two additive columns, built-in roles are not re-seeded. No agent change. See the upgrade notes for rolling-deploy and downgrade rules.
- **v1.11.3** (2026-10-16) — **Config Import keeps `stick-table` and `stick` rules instead of dropping them**: the import parser had no branch for `stick-table` (frontend or backend) or for backend `stick on|match|store-request|store-response`, so those lines were discarded **without a warning**. A frontend that kept its `track-sc0` / `sc_*` counter rules then received the renderer's fallback table (`type ip size 100k expire 30s store http_req_rate(10s)`), so a re-defined table such as `size 200k expire 3h store http_req_rate(10s),gpc0_rate(10s),gpt0` was silently replaced by the default while the `gpc0` rules imported alongside it referenced counters the table did not store. These lines are now imported as-is into the frontend/backend **Options** field (no schema change) with an import warning, and an import that carries only a stick-table keeps the existing option lines. The table the platform generates itself for Rate Limit / WAF `rate_limit` rules is recognised and not stored, so a round-trip of a generated config shows no change and later rule edits still take effect. The import preview also deleted every `http-request track-sc0 src` / `sc_http_req_rate(0)` line as "auto-generated"; it now does so only where the platform regenerates them (frontend Rate Limit or a WAF `rate_limit` rule), so operator trackers on other frontends survive. Backends now render a single `stick-table` (the first; a second one is a fatal HAProxy error) and the preview warns about a table that does not store `http_req_rate` while Rate Limit is active, a duplicate table left in Request Headers, `stick on` without a table, and `peers` references. Configs without stick lines import and render byte-for-byte as before. No schema change, no agent change.
- **v1.11.2** (2026-10-01) — **Read-only roles read SSL certificates, and private keys need `ssl.download`**: the UI never fills in `user_pool_access`, so every non-admin user created there has no pool access, and the SSL list (`?cluster_id=`) and SSL detail answered **403** on read while clusters, frontends, backends, WAF rules, agents and VIPs were readable. A role that grants `ssl.read` (or `waf.read` for WAF stats) now reads that resource on every cluster; without it the pool check still applies, and write endpoints are unchanged. Separately, `GET /api/ssl/certificates/{id}` and the SSL diff in Apply Management returned the **private key to any authenticated user**, including `viewer`. The key is now included only with `ssl.download` plus pool access; everyone else gets `private_key_content: null` with `private_key_redacted: true`, and the UI shows the key as hidden while editing keeps the stored key. Custom roles with `ssl.update` but no `ssl.download` lose sight of the key. No schema change, no agent change.
- **v1.11.1** (2026-08-15) — **A node can no longer be hidden from adoption for good, and Config Import reaches a freshly installed agent**: the agent posts an unmanaged `keepalived.conf` for adoption and caches the hash of what it sent, so the file (which carries the VRRP password) is re-posted only when it changes. Delivery was judged by `curl`'s exit code, which is **0 for 5xx as well**, so a report the server *rejected* was recorded as delivered — and because a hand-maintained config does not change on its own, that node dropped out of *Unmanaged keepalived detected* permanently, curable only by deleting a cache file on the node by hand. Now: the report is cached only on a 2xx; `GET /agents/{name}/keepalived-config` tells the agent whether the server actually holds a discovery for it, so nodes stuck from earlier releases recover by themselves on the next poll; a **400/413/422** records the refusal so identical bytes are not re-posted forever (4xx and 5xx are never sampled out of the request log, so an unattended loop would write a row carrying the whole config every cycle), while **401 and 404 keep retrying** because in this system they mean a token rotation or an agent row briefly absent, not a bad payload. The same exit-code mistake in the *clear* path is fixed too, where it left a managed node still being offered for adoption. Separately, **Config Import silently did nothing on any agent that had never self-upgraded**: `check_config_requests` was defined in the installer and in the self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by `type`, so the operator asked a node for its `haproxy.cfg` and nothing arrived, with no error anywhere. And the cluster whose `keepalived_config_path` is handed to an agent is now resolved deterministically — a pool may hold several clusters, and the unordered join could return a different one between polls, pointing the agent at a file that does not exist. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema change.
+56
View File
@@ -1,3 +1,59 @@
# Upgrade Notes — v1.12.0 (several keepalived VIPs on one node, several addresses on one VIP)
**Backend + UI change, schema `13` (delta), no agent change.** Agents of every version keep working;
no agent upgrade is needed.
- **Schema 12 → 13 is a delta.** A database at 12 adds two columns —
`vip_instances.additional_ips` (JSONB, default `[]`) and `vip_discoveries.analysis_version`
(SMALLINT, default `0`) — and nothing else runs. **Built-in roles are not re-seeded**, so role
customizations survive this upgrade (unlike v1.10.4 and v1.11.0). A database older than 12 runs the
full sequence as before, including the role re-seed.
- **Nodes with one VIP see no change.** Their `keepalived.conf` keeps the same bytes and hash, so the
agent does not rewrite or reload anything.
- **Check for nodes that already belong to several VIPs before upgrading.** Earlier versions refused
this when a VIP was saved, but without a lock, so two saves at the same moment could still create
it. Such a node switches from one VIP's file to the merged file on its first poll after the upgrade:
```sql
SELECT a.name, count(*) FROM vip_members m
JOIN vip_instances v ON v.id = m.vip_id JOIN agents a ON a.id = m.agent_id
WHERE v.is_active GROUP BY a.name HAVING count(*) > 1;
```
- **Rolling deploy:** do not add a second VIP to a node until **every backend replica** runs v1.12.0.
An older replica serves a node only one of its VIPs, so the node would alternate between two files.
An older replica applying a VIP also drops that VIP's additional addresses.
- **Rollback / downgrade:** first reduce every node to one VIP and remove the additional addresses
(edit, then Apply), then downgrade. The two new columns can stay; older versions ignore them.
- **How a node with several VIPs is served.** Each VIP is still rendered and applied on its own; the
delivery endpoint merges the node's applied VIPs into one file: `global_defs` once, then per VIP a
`vrrp_script chk_haproxy_<id>` and a `vrrp_instance VI_<id>`, ordered by VIP id. VRIDs are unique per
pool, so the instances never collide. One acknowledgement from the node converges all of its VIPs.
Applying two VIPs one after the other reloads keepalived twice; measured on keepalived 2.0.19,
2.1.5 and 2.2.8, a reload that adds or removes a VIP does not drop the other VIPs' addresses.
- **If the merge fails** (an applied snapshot that is not in the platform's own format), the node keeps
running what it has: the agent is told nothing new, and the node's members show
"cannot merge this node's VIPs …" with an ERROR status. Re-apply the VIPs to fix it.
- **Delivery errors no longer tear keepalived down.** A database error while serving
`keepalived-config` answered `not_configured`, which on a node whose file is ours meant teardown. It
now answers `unavailable`, which agents of every version ignore.
- **Additional addresses:** up to 19 per VIP (IPv4, prefix 1–32), on the VIP's interface on every node.
They must not be a node's IP or any address of another active VIP.
- **Live state is per node, not per VIP.** The MASTER/BACKUP column comes from the node's heartbeat,
which reports one state for the node. A node that is MASTER for one VIP and BACKUP for another shows
the same state in both rows; with several VIPs on a node, the heartbeat's config-based fallback can
also report MASTER on a BACKUP node. Check `ip addr` on the node when it matters. The VIPs themselves
fail over correctly; only the display is affected.
- **Adoption:** a `keepalived.conf` with several `vrrp_instance` blocks can no longer be adopted (it
replaces the whole file, so the other instances would be removed — earlier versions allowed it). An
adoption made before the upgrade from such a file is refused at Apply; reject it. An instance with
several addresses is now adoptable; the extra addresses become additional addresses. Files analysed
by an older version are re-reported once by v1.11.1+ agents; until then the list shows them with the
new blocker.
**Rollback:** see the downgrade rule above. With one VIP per node and no additional addresses,
rolling back is safe.
---
# Upgrade Notes — v1.11.3 (Config Import keeps `stick-table` and `stick` rules)
**Backend change, no schema change, no agent change.** No `SCHEMA_VERSION` bump and no agent
+56 -2
View File
@@ -1796,7 +1796,24 @@ async def ensure_agent_activity_logs_table():
# Additive + idempotent; no existing table is altered, agents never read this
# table. Same caveat as v1.10.4: this bump re-seeds the four built-in roles to
# their defaults, so export role customizations before upgrading.
SCHEMA_VERSION = 12
#
# v1.12.0 (issue #73 — several VIPs per node, several addresses per VIP): bumped
# 12 -> 13 as a DELTA. A database already at 12 runs only the steps listed in
# _delta_steps() — two additive columns — and keeps its role customizations; the full
# sequence (and with it the built-in role re-seed) runs only for databases older than
# _DELTA_BASE_VERSION or without a marker, exactly as before. Every delta step is ALSO
# called from _run_all_migrations_inner(), so both paths end at the same schema.
# A future change that alters seeded/role data needs a full bump: raise
# _DELTA_BASE_VERSION to the new SCHEMA_VERSION instead of adding a delta step.
SCHEMA_VERSION = 13
_DELTA_BASE_VERSION = 12
def _delta_steps():
"""(version, step) pairs a database at >= _DELTA_BASE_VERSION runs instead of the full
sequence. Each step takes the migration connection and raises on failure, so the version
marker is not written and the next start retries."""
return [(13, ensure_vip_multi_address_columns)]
async def run_all_migrations():
@@ -1845,7 +1862,13 @@ async def run_all_migrations():
logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
return
await _run_all_migrations_inner()
if applied_version is not None and applied_version >= _DELTA_BASE_VERSION:
for version, step in _delta_steps():
if version > applied_version:
logger.info(f"Applying schema delta {applied_version} -> {version}")
await step(lock_conn)
else:
await _run_all_migrations_inner()
try:
await lock_conn.execute("""
@@ -1935,6 +1958,7 @@ async def _run_all_migrations_inner():
# Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
# MUST run after its FK targets (haproxy_cluster_pools/agents/users), all created above.
await ensure_vip_tables()
await ensure_vip_multi_address_columns()
# v1.9.0 — CSR creation: brand-new ssl_csrs table. FK-references
# ssl_certificates/users, both created above.
@@ -2408,6 +2432,36 @@ async def ensure_vip_tables():
if conn:
await close_database_connection(conn)
async def ensure_vip_multi_address_columns(conn=None):
"""v1.12.0 (issue #73). Additive + idempotent:
* vip_instances.additional_ips — the VIP's addresses beyond virtual_ip, as a JSON list
of {"address", "prefix_length"}. Empty for every existing VIP, which renders exactly
as before.
* vip_discoveries.analysis_version — which parser produced the stored analysis. Rows
written by an older parser read 0, and the agent is asked to re-report them once.
Raises on failure (unlike ensure_vip_tables): code reads these columns unconditionally,
so the schema version must not be recorded without them.
"""
own = conn is None
if own:
conn = await get_database_connection()
try:
# IF EXISTS: ensure_vip_tables() tolerates its own failure, and a missing VIP table
# must keep degrading only the VIP feature, not stop the backend from starting.
await conn.execute(
"ALTER TABLE IF EXISTS vip_instances ADD COLUMN IF NOT EXISTS additional_ips "
"JSONB NOT NULL DEFAULT '[]'::jsonb;")
await conn.execute(
"ALTER TABLE IF EXISTS vip_discoveries ADD COLUMN IF NOT EXISTS analysis_version "
"SMALLINT NOT NULL DEFAULT 0;")
logger.info("✅ VIP multi-address columns ensured (issue #73)")
finally:
if own:
await close_database_connection(conn)
async def add_ssl_certificate_id_to_backend_servers():
"""Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
conn = None
+37
View File
@@ -50,6 +50,38 @@ def _validate_ipv4(v: str) -> str:
return s
# Issue #73: a VIP may carry further addresses in the same VRRP instance. 20 addresses in
# total keeps every advertisement within what VRRPv2 peers are known to accept.
MAX_ADDITIONAL_IPS = 19
class VIPAddressIn(BaseModel):
address: str
prefix_length: int = 24
@field_validator('address')
@classmethod
def _addr(cls, v):
return str(ipaddress.ip_address(_validate_ipv4(v)))
@field_validator('prefix_length')
@classmethod
def _prefix(cls, v):
if not isinstance(v, int) or not (1 <= v <= 32):
raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
return v
def _validate_additional_ips(extras: List['VIPAddressIn'], primary: Optional[str]) -> None:
if len(extras) > MAX_ADDITIONAL_IPS:
raise ValueError(f'a VIP may carry at most {MAX_ADDITIONAL_IPS} additional addresses')
addrs = [e.address for e in extras]
if len(set(addrs)) != len(addrs):
raise ValueError('each additional address may appear only once')
if primary is not None and str(ipaddress.ip_address(primary)) in addrs:
raise ValueError('an additional address must differ from virtual_ip')
class VIPMemberIn(BaseModel):
agent_id: int
network_interface: str
@@ -122,6 +154,7 @@ class VIPCreate(BaseModel):
auth_pass: Optional[str] = None # plaintext on the wire; stored Fernet-encrypted
use_unicast: bool = True
track_haproxy: bool = True
additional_ips: List[VIPAddressIn] = []
members: List[VIPMemberIn]
@field_validator('name')
@@ -172,6 +205,7 @@ class VIPCreate(BaseModel):
@model_validator(mode='after')
def _members_consistent(self):
_validate_members(self.members)
_validate_additional_ips(self.additional_ips, self.virtual_ip)
return self
@@ -187,6 +221,7 @@ class VIPUpdate(BaseModel):
auth_pass: Optional[str] = None # provide only to rotate; omit to keep existing
use_unicast: Optional[bool] = None
track_haproxy: Optional[bool] = None
additional_ips: Optional[List[VIPAddressIn]] = None # None keeps them; [] clears them
members: Optional[List[VIPMemberIn]] = None
@field_validator('name')
@@ -242,4 +277,6 @@ class VIPUpdate(BaseModel):
def _members_consistent(self):
if self.members is not None:
_validate_members(self.members)
if self.additional_ips is not None:
_validate_additional_ips(self.additional_ips, self.virtual_ip)
return self
+153 -24
View File
@@ -931,16 +931,17 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
# with no way to be told to tear down (review MED-3). Make the operator remove it
# from the VIP first — that stages a clean PENDING change they can apply.
try:
vip_member = await conn.fetchrow(
# A node may belong to several VIPs (issue #73); name them all.
vip_names = [r["name"] for r in await conn.fetch(
"SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
"WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
"WHERE vm.agent_id = $1 AND v.is_active = TRUE ORDER BY v.name", agent_id)]
except Exception: # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
vip_member = None
if vip_member:
vip_names = []
if vip_names:
await close_database_connection(conn)
raise HTTPException(
status_code=409,
detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
detail=(f"This node is a member of VIP '{', '.join(vip_names)}'. Remove it from the "
f"VIP on the HA / VIP page (and apply) before deleting the agent."))
await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
@@ -2281,6 +2282,37 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
raise HTTPException(status_code=500, detail=str(e))
async def _node_applied_vips(conn, agent_id: int):
"""Issue #73 — the active VIPs this node runs: every active membership that has an applied
snapshot, ordered by VIP id (the order compose_node_keepalived_conf writes them in)."""
return await conn.fetch("""
SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
v.purge_on_teardown,
m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
WHERE m.agent_id = $1 AND v.is_active = TRUE
AND m.applied_config_content IS NOT NULL AND m.applied_config_content <> ''
ORDER BY v.id
""", agent_id)
async def _flag_compose_error(conn, agent_id: int, vip_ids, exc: Exception) -> None:
"""Show a merge failure on the node's members. The agent never acks a status it does not
know, so without this the VIPs would sit at SYNCING with nothing saying why."""
if not vip_ids:
return
msg = f"cannot merge this node's VIPs into one keepalived.conf ({exc}); re-apply the VIPs"[:500]
try:
await conn.execute("""
UPDATE vip_members SET last_deploy_state='error', last_deploy_message=$3,
last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
AND (last_deploy_state IS DISTINCT FROM 'error' OR last_deploy_message IS DISTINCT FROM $4)
""", agent_id, list(vip_ids), msg, msg)
except Exception as e: # noqa: BLE001
logger.error(f"could not record the keepalived compose failure: {e}")
@router.get("/{agent_name}/keepalived-config")
async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
"""Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
@@ -2293,8 +2325,8 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
the agent API key is a SHARED/global install token, so the config is resolved by the
requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
(a hard 403 would break every VIP member whose name isn't the one row the shared token
resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
the agent stays inert; a node with no membership row always gets not_configured.
resolves to — review HIGH-1). Any unexpected error degrades to a status the agent ignores
(B-7) so the agent stays inert; a node with no membership row always gets not_configured.
"""
conn = None
try:
@@ -2303,6 +2335,7 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
# name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
# unauthenticated caller can't probe which agent names exist.
from auth_middleware import validate_agent_api_key
from services.keepalived_parser import ANALYSIS_VERSION
if not x_api_key:
raise HTTPException(status_code=401, detail="Agent API key required")
agent_auth = await validate_agent_api_key(x_api_key)
@@ -2333,7 +2366,12 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
-- good: the file never changes, so the agent never speaks again. Telling it
-- what we actually hold lets it recover on its own, with no extra request and
-- no one having to touch the node.
EXISTS (SELECT 1 FROM vip_discoveries vd WHERE vd.agent_id = a.id)
--
-- v1.12.0: only a discovery analysed by the CURRENT parser counts. An older
-- analysis reads as unknown, so the agent re-reports once and the stored
-- blockers/candidates catch up without anyone touching the node.
EXISTS (SELECT 1 FROM vip_discoveries vd
WHERE vd.agent_id = a.id AND vd.analysis_version >= $2)
AS discovery_known
FROM agents a
LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
@@ -2354,7 +2392,7 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
OR hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'),
hc.id
LIMIT 1
""", agent_name)
""", agent_name, ANALYSIS_VERSION)
if not agent:
raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
@@ -2367,13 +2405,57 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
WHERE m.agent_id = $1
-- Active VIP first (an agent has at most one). With NO active VIP, pick the most
-- RECENTLY updated inactive membership so a teardown reflects the latest delete
-- (incl. its purge flag) — not a stale older VIP the node was once part of.
-- Active VIP first. With NO active VIP, pick the most RECENTLY updated inactive
-- membership so a teardown reflects the latest delete (incl. its purge flag) —
-- not a stale older VIP the node was once part of.
ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
LIMIT 1
""", agent['id'])
# Issue #73: a node may belong to several active VIPs. Their applied snapshots are
# merged into the one keepalived.conf the node deploys; a single VIP is served exactly
# as before (same content, same hash).
# Any failure in this block must answer with a status the agent ignores: on a node
# whose file is ours the agent treats not_configured as "tear keepalived down". Here
# the members are also flagged, so the operator sees why the node is not converging.
node_vips = []
merged = None
if row and row['is_active']:
try:
node_vips = await _node_applied_vips(conn, agent['id'])
if len(node_vips) > 1:
from services.keepalived_config import (
build_haproxy_check_script, compose_node_keepalived_conf)
composed = compose_node_keepalived_conf(
[(r['vip_id'], r['applied_config_content']) for r in node_vips])
takeover = next((r['takeover_expected_hash'] for r in node_vips
if r['takeover_expected_hash']), None)
merged = {
"desired_state": "enabled",
"install_if_missing": True,
"vip_id": node_vips[0]['vip_id'],
"vip_ids": [r['vip_id'] for r in node_vips],
"vip_name": ", ".join(str(r['vip_name']) for r in node_vips),
"config_content": composed,
"config_hash": hashlib.md5(composed.encode("utf-8")).hexdigest(),
"check_script": build_haproxy_check_script()
if any(r['track_haproxy'] for r in node_vips) else "",
"allow_takeover": bool(takeover),
"takeover_expected_hash": takeover,
}
except Exception as ce: # noqa: BLE001
logger.error(f"keepalived compose failed for '{agent_name}': {ce}")
await _flag_compose_error(conn, agent['id'], [r['vip_id'] for r in node_vips], ce)
return {"agent_name": agent_name, "status": "compose_error", "config_path": config_path,
"discovery_known": bool(agent["discovery_known"]), "keepalived": None}
if merged is not None:
return {"agent_name": agent_name, "status": "available", "config_path": config_path,
"discovery_known": bool(agent["discovery_known"]), "keepalived": merged}
if len(node_vips) == 1:
# The most recently updated active VIP may be a pending one with nothing applied
# yet; serving it would read as not_configured and tear down the running VIP.
row = node_vips[0]
if not row:
return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
if not row['is_active']:
@@ -2413,8 +2495,10 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
raise
except Exception as e:
logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
# Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
return {"agent_name": agent_name, "status": "not_configured",
# Degrade to a no-op rather than 500 (B-7). The status must be one the agent does not
# act on: not_configured is a teardown on a node whose file is ours, so a transient
# database error would have taken a running VIP down.
return {"agent_name": agent_name, "status": "unavailable",
"config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
finally:
if conn:
@@ -2476,12 +2560,54 @@ async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key:
WHERE agent_id=$1
""", agent['id'], state, message, config_hash, config_hash)
else:
await conn.execute(f"""
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
{_retire_takeover.format(p="$6")}
WHERE agent_id=$1 AND vip_id=$2
""", agent['id'], int(vip_id), state, message, config_hash, config_hash)
# Issue #73: a node running several VIPs acks the merged file once, with the first
# VIP's id and the merged file's hash. Spread the ack over every VIP in that file;
# each member is converged against its OWN applied hash.
node_vips = await _node_applied_vips(conn, agent['id'])
node_vip_ids = [r['vip_id'] for r in node_vips]
converged = False
if len(node_vips) > 1 and int(vip_id) in node_vip_ids:
from services.keepalived_config import compose_node_keepalived_conf
try:
composed = compose_node_keepalived_conf(
[(r['vip_id'], r['applied_config_content']) for r in node_vips])
composed_hash = hashlib.md5(composed.encode("utf-8")).hexdigest()
except Exception: # noqa: BLE001 — unmergeable means not converged, never a 500
composed_hash = None
converged = bool(composed_hash) and config_hash == composed_hash
if converged:
await conn.execute("""
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
last_deploy_hash=applied_config_hash, last_deploy_at=CURRENT_TIMESTAMP,
updated_at=CURRENT_TIMESTAMP, takeover_expected_hash=NULL
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
""", agent['id'], node_vip_ids, state, message)
else:
await conn.execute("""
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
""", agent['id'], node_vip_ids, state, message, config_hash)
else:
await conn.execute(f"""
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
{_retire_takeover.format(p="$6")}
WHERE agent_id=$1 AND vip_id=$2
""", agent['id'], int(vip_id), state, message, config_hash, config_hash)
converged = any(r['vip_id'] == int(vip_id) and r['applied_config_hash'] == config_hash
for r in node_vips)
if state == "enabled" and converged:
# The file the node now runs holds only its active VIPs, so a deleted VIP's
# membership on this node is torn down even though no teardown was sent for it.
await conn.execute("""
UPDATE vip_members m SET last_deploy_state='disabled',
last_deploy_message='removed from this node''s keepalived.conf',
last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
FROM vip_instances v
WHERE v.id = m.vip_id AND m.agent_id=$1 AND v.is_active = FALSE
AND m.last_deploy_state IS DISTINCT FROM 'disabled'
""", agent['id'])
return {"status": "ok"}
except HTTPException:
raise
@@ -2539,7 +2665,8 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
is_managed = bool(payload.get("is_managed"))
config_hash = hashlib.md5(content.encode("utf-8", "replace")).hexdigest()
from services.keepalived_parser import analyse_keepalived_conf, KeepalivedParseError
from services.keepalived_parser import (
ANALYSIS_VERSION, analyse_keepalived_conf, KeepalivedParseError)
from services.keepalived_config import encrypt_vrrp_secret
parse_error = None
@@ -2562,8 +2689,8 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
await conn.execute("""
INSERT INTO vip_discoveries
(agent_id, config_path, config_hash, is_managed, raw_config_masked,
auth_pass_encrypted, analysis, parse_error, reported_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,CURRENT_TIMESTAMP)
auth_pass_encrypted, analysis, parse_error, analysis_version, reported_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,CURRENT_TIMESTAMP)
ON CONFLICT (agent_id) DO UPDATE SET
config_path = EXCLUDED.config_path,
config_hash = EXCLUDED.config_hash,
@@ -2572,9 +2699,11 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
auth_pass_encrypted = EXCLUDED.auth_pass_encrypted,
analysis = EXCLUDED.analysis,
parse_error = EXCLUDED.parse_error,
analysis_version = EXCLUDED.analysis_version,
reported_at = CURRENT_TIMESTAMP
""", agent['id'], config_path, config_hash, is_managed, masked, auth_enc,
json.dumps(analysis) if analysis is not None else None, parse_error)
json.dumps(analysis) if analysis is not None else None, parse_error,
ANALYSIS_VERSION)
return {"status": "recorded", "config_hash": config_hash}
except HTTPException:
raise
+157 -60
View File
@@ -16,6 +16,7 @@ Every DB access is wrapped so a (pathological) missing vip_* relation degrades t
empty/None result instead of a 500 (B-7) — preserving the fleet-wide no-op guarantee.
"""
import hashlib
import ipaddress
import json
import logging
import re
@@ -171,7 +172,7 @@ async def render_vip_config_masked(conn, vip_id: int):
vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
"track_haproxy": v["track_haproxy"]}
"track_haproxy": v["track_haproxy"], "additional_ips": _extra_ips(v["additional_ips"])}
member_dicts = [{"role": m["role"], "priority": m["priority"],
"network_interface": m["network_interface"], "agent_id": m["agent_id"],
"ip_address": str(m["ip_address"]) if m["ip_address"] else ""} for m in members]
@@ -241,7 +242,8 @@ async def _capture_vip_pending_state(conn, vip_id: int):
"vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"],
"additional_ips": _extra_ips(v["additional_ips"])},
"members": [{"agent_id": m["agent_id"], "network_interface": m["network_interface"],
"role": m["role"], "priority": m["priority"]} for m in members],
}
@@ -282,37 +284,31 @@ async def restore_vip_from_rejected_version(conn, version_id: int):
if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1", vip_id):
return "the VIP no longer exists — re-create it from the HA/VIP page"
sv, sm = ps["vip"], ps.get("members", [])
# One-VIP-per-agent must STILL hold after reactivation: if a member was meanwhile added
# to another active VIP (while this one was rejected/soft-deleted), undoing here would
# create a double-membership — and since the delivery endpoint serves one VIP per agent,
# reactivating this (never-applied → not_configured) VIP would tear down the other live
# VIP on that node. Block it with a clear message (review round-3 FINDING 2).
member_ids = [m["agent_id"] for m in sm]
if member_ids:
dup = await conn.fetchrow(
"SELECT a.name AS agent_name, v.name AS vip_name FROM vip_members vm "
"JOIN vip_instances v ON v.id = vm.vip_id JOIN agents a ON a.id = vm.agent_id "
"WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE AND v.id <> $2 LIMIT 1",
member_ids, vip_id)
if dup:
return (f"node '{dup['agent_name']}' now belongs to active VIP '{dup['vip_name']}' — "
f"remove it there first, then undo")
# Issue #73: a member that meanwhile joined another active VIP keeps it. The delivery
# endpoint serves a node every active VIP that has an applied snapshot, so reactivating
# this one (never applied, or applied earlier) cannot displace the other.
# Preserve the running (applied) snapshot for members that remain, so undo of an edit
# never momentarily flips a live node to not_configured (T-1).
prev = {r["agent_id"]: r for r in await conn.fetch(
"SELECT agent_id, applied_config_content, applied_config_hash, last_deploy_state, "
"last_deploy_message, last_deploy_hash, last_deploy_at FROM vip_members WHERE vip_id=$1", vip_id)}
# Captured before v1.12.0 → no additional addresses.
extras = _extra_ips(sv.get("additional_ips") or [])
try:
async with conn.transaction():
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
reason = await _address_clash(conn, sv["virtual_ip"], extras, vip_id)
if reason:
return f"cannot restore — {reason}"
await conn.execute("""
UPDATE vip_instances SET name=$2, description=$3, virtual_ip=$4, prefix_length=$5,
virtual_router_id=$6, advert_int=$7, use_unicast=$8, track_haproxy=$9,
auth_pass_encrypted=$10, is_active=TRUE, last_config_status='PENDING',
updated_at=CURRENT_TIMESTAMP
additional_ips=$11::jsonb, updated_at=CURRENT_TIMESTAMP
WHERE id=$1
""", vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
sv.get("auth_pass_encrypted"))
sv.get("auth_pass_encrypted"), json.dumps(extras))
await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
for mm in sm:
o = prev.get(mm["agent_id"])
@@ -395,7 +391,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
if cluster_id is not None:
vips = await conn.fetch(f"""
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
v.is_active, v.last_config_status, v.pending_delete,
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
v.created_at, v.updated_at, p.name AS pool_name
@@ -408,7 +404,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
else:
vips = await conn.fetch(f"""
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
v.is_active, v.last_config_status, v.pending_delete,
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
v.created_at, v.updated_at, p.name AS pool_name
@@ -434,6 +430,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
v["last_config_status"], members, v["pending_delete"], v["is_active"])
result.append({
**dict(v),
"additional_ips": _extra_ips(v["additional_ips"]),
# Convergence-aware status for the table (issue #27 follow-up); the raw
# last_config_status is kept above for the PENDING gate / Apply button.
"deploy_status": deploy_status,
@@ -505,7 +502,7 @@ async def get_vip(vip_id: int, authorization: str = Header(None)):
try:
v = await conn.fetchrow("""
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
v.is_active, v.last_config_status,
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
v.created_at, v.updated_at
@@ -520,7 +517,8 @@ async def get_vip(vip_id: int, authorization: str = Header(None)):
FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
WHERE m.vip_id = $1 ORDER BY m.priority DESC
""", vip_id)
return {**dict(v), "members": [dict(m) for m in members]}
return {**dict(v), "additional_ips": _extra_ips(v["additional_ips"]),
"members": [dict(m) for m in members]}
finally:
await close_database_connection(conn)
@@ -540,24 +538,9 @@ async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_
if row["pool_id"] != pool_id:
raise HTTPException(status_code=400,
detail=f"agent {m.agent_id} is not in pool {pool_id}")
# One active VIP per agent (v1): a node deploys a single keepalived.conf, and the
# delivery endpoint serves one VIP per agent — a second active membership would never
# converge (stuck SYNCING) instead of erroring. Reject it up front (review MED-2).
agent_ids = [m.agent_id for m in members]
if agent_ids:
dq = ("SELECT a.name AS agent_name, v.name AS vip_name "
"FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
"JOIN agents a ON a.id = vm.agent_id "
"WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE")
dparams = [agent_ids]
if exclude_vip_id is not None:
dq += " AND v.id <> $2"
dparams.append(exclude_vip_id)
dup = await conn.fetchrow(dq + " LIMIT 1", *dparams)
if dup:
raise HTTPException(status_code=409,
detail=f"node '{dup['agent_name']}' is already a member of VIP "
f"'{dup['vip_name']}' — a node can belong to only one VIP")
# Issue #73: a node may belong to several VIPs. The delivery endpoint merges their applied
# snapshots into the node's one keepalived.conf; the pool-scoped VRID uniqueness
# (uq_vip_vrid_active) keeps their VRRP instances apart.
# VIP must not be an existing agent's primary IP (INET cast, SQL-1)
clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", vip_virtual_ip)
if clash:
@@ -572,6 +555,57 @@ async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_
raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already used by another VIP")
# Serialises every write that sets a VIP's addresses. The primary address has a partial unique
# index; the additional ones live in a JSON list no index can cover, so two concurrent writes
# could otherwise both pass the clash check below with the same address.
_VIP_ADDRESS_LOCK = (18181873, 0)
def _norm_ip(value) -> str:
try:
return str(ipaddress.ip_address(str(value).strip()))
except ValueError:
return str(value)
def _extra_ips(value) -> List[dict]:
"""vip_instances.additional_ips (JSONB; asyncpg hands it over as a string) as a list."""
items = _jsonb_list(value)
return [{"address": str(i["address"]), "prefix_length": int(i["prefix_length"])}
for i in items if isinstance(i, dict) and i.get("address")]
async def _address_clash(conn, primary: str, extras: List[dict],
exclude_vip_id: Optional[int]) -> Optional[str]:
"""Issue #73 — why these addresses cannot be used, or None. Covers what the primary-address
checks do not: every additional address against node IPs and every active VIP's addresses,
and the primary against the other VIPs' additional addresses. Call it inside the write's
transaction after taking _VIP_ADDRESS_LOCK."""
other = ("SELECT name FROM vip_instances WHERE is_active = TRUE "
"AND ($2::int IS NULL OR id <> $2::int) AND {cond} LIMIT 1")
in_extras = "additional_ips @> jsonb_build_array(jsonb_build_object('address', $1::text))"
hit = await conn.fetchval(other.format(cond=in_extras), primary, exclude_vip_id)
if hit:
return f"{primary} is already an additional address of VIP '{hit}'"
for e in extras:
addr = e["address"]
if await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", addr):
return f"{addr} is already a node's IP"
hit = await conn.fetchval(
other.format(cond=f"(virtual_ip = $1::text OR {in_extras})"), addr, exclude_vip_id)
if hit:
return f"{addr} is already used by VIP '{hit}'"
return None
async def _lock_and_check_addresses(conn, primary: str, extras: List[dict],
exclude_vip_id: Optional[int]) -> None:
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
reason = await _address_clash(conn, primary, extras, exclude_vip_id)
if reason:
raise HTTPException(status_code=409, detail=reason)
@router.post("")
async def create_vip(payload: VIPCreate, request: Request, authorization: str = Header(None)):
current_user = await _require(authorization, "create")
@@ -583,22 +617,25 @@ async def create_vip(payload: VIPCreate, request: Request, authorization: str =
await _validate_members_against_pool(conn, payload.pool_id, payload.members, payload.virtual_ip)
enc = encrypt_vrrp_secret(payload.auth_pass) if payload.auth_pass else None
extras = [e.model_dump() for e in payload.additional_ips]
# Allocate VRID + insert, retrying once on a unique-violation race (B-1).
last_err = None
for _attempt in range(2):
vrid = await _alloc_free_vrid(conn, payload.pool_id, payload.virtual_router_id)
try:
async with conn.transaction():
await _lock_and_check_addresses(conn, payload.virtual_ip, extras, None)
vip_id = await conn.fetchval("""
INSERT INTO vip_instances
(name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
is_active, last_config_status, created_by)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11)
is_active, last_config_status, created_by, additional_ips)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11,$12::jsonb)
RETURNING id
""", payload.name, payload.description, payload.pool_id, payload.virtual_ip,
payload.prefix_length, vrid, payload.advert_int, enc,
payload.use_unicast, payload.track_haproxy, current_user["id"])
payload.use_unicast, payload.track_haproxy, current_user["id"],
json.dumps(extras))
for m in payload.members:
await conn.execute("""
INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority)
@@ -606,6 +643,8 @@ async def create_vip(payload: VIPCreate, request: Request, authorization: str =
""", vip_id, m.agent_id, m.network_interface, m.role, m.priority)
last_err = None
break
except HTTPException:
raise
except Exception as ie: # noqa: BLE001
if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
last_err = ie
@@ -650,10 +689,17 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
"SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE AND id<>$2", new_ip, vip_id)
if dup:
raise HTTPException(status_code=409, detail=f"{new_ip} is already used by another VIP")
new_extras = ([e.model_dump() for e in payload.additional_ips]
if payload.additional_ips is not None else None)
extras = new_extras if new_extras is not None else _extra_ips(v["additional_ips"])
if _norm_ip(new_ip) in {e["address"] for e in extras}:
raise HTTPException(status_code=400, detail="an additional address must differ from virtual_ip")
enc_set = payload.auth_pass is not None and payload.auth_pass != ""
try:
async with conn.transaction():
if payload.virtual_ip or new_extras is not None:
await _lock_and_check_addresses(conn, new_ip, extras, vip_id)
await conn.execute("""
UPDATE vip_instances SET
name = COALESCE($2, name),
@@ -665,6 +711,7 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
use_unicast = COALESCE($8, use_unicast),
track_haproxy = COALESCE($9, track_haproxy),
auth_pass_encrypted = CASE WHEN $10 THEN $11 ELSE auth_pass_encrypted END,
additional_ips = COALESCE($12::jsonb, additional_ips),
last_config_status = 'PENDING',
-- Editing a VIP means you are KEEPING and changing it, so it cancels any
-- staged deletion (otherwise a later Apply would delete instead of applying
@@ -676,7 +723,8 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
""", vip_id, payload.name, payload.description, payload.virtual_ip,
payload.prefix_length, payload.virtual_router_id, payload.advert_int,
payload.use_unicast, payload.track_haproxy,
enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None)
enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None,
json.dumps(new_extras) if new_extras is not None else None)
if members is not None:
# T-1: preserve the APPLIED snapshot + deploy ack for members that REMAIN,
# so a pending member edit never momentarily flips a running node to
@@ -826,6 +874,22 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
master_prio = masters[0]["priority"]
if any(m["role"] == "BACKUP" and m["priority"] >= master_prio for m in members):
raise HTTPException(status_code=400, detail="MASTER priority must exceed every BACKUP")
# Issue #73: an adoption made before v1.12.0 may still be waiting for this Apply. Taking
# over a file with several vrrp_instance blocks would remove the others from the node.
multi = await conn.fetchrow("""
SELECT a.name AS agent_name, (d.analysis->>'instance_count')::int AS n
FROM vip_members m
JOIN vip_discoveries d ON d.agent_id = m.agent_id
JOIN agents a ON a.id = m.agent_id
WHERE m.vip_id = $1 AND m.takeover_expected_hash IS NOT NULL
AND d.config_hash = m.takeover_expected_hash
AND COALESCE((d.analysis->>'instance_count')::int, 0) > 1
LIMIT 1
""", vip_id)
if multi:
raise HTTPException(status_code=409, detail=(
f"{multi['agent_name']}'s keepalived.conf declares {multi['n']} vrrp_instance blocks; "
f"taking it over would remove the others from the node. Reject this adoption."))
auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
# If a secret is set but can't be decrypted (SECRET_KEY/VIP_ENCRYPTION_KEY rotated or
@@ -844,7 +908,7 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
"track_haproxy": v["track_haproxy"]}
"track_haproxy": v["track_haproxy"], "additional_ips": _extra_ips(v["additional_ips"])}
snapshot_members = []
async with conn.transaction():
@@ -870,7 +934,8 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
"vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"],
"additional_ips": vip_dict["additional_ips"]},
"members": snapshot_members}
await conn.execute(
"UPDATE vip_instances SET last_config_status='APPLIED', "
@@ -944,17 +1009,24 @@ async def reject_vip(vip_id: int, request: Request, authorization: str = Header(
sv = snap["vip"]
sm = snap.get("members", [])
# Applied before v1.12.0 → no additional addresses.
extras = _extra_ips(sv.get("additional_ips") or [])
try:
async with conn.transaction():
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
reason = await _address_clash(conn, sv["virtual_ip"], extras, vip_id)
if reason:
raise HTTPException(status_code=409, detail=f"Cannot restore — {reason}")
await conn.execute("""
UPDATE vip_instances SET
name=$2, description=$3, virtual_ip=$4, prefix_length=$5, virtual_router_id=$6,
advert_int=$7, use_unicast=$8, track_haproxy=$9, auth_pass_encrypted=$10,
additional_ips=$11::jsonb,
last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP
WHERE id=$1
""", vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
sv.get("auth_pass_encrypted"))
sv.get("auth_pass_encrypted"), json.dumps(extras))
await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
for m in sm:
await conn.execute("""
@@ -1049,6 +1121,16 @@ def _discovery_row_to_api(row) -> dict:
analysis = json.loads(analysis)
except (json.JSONDecodeError, TypeError):
analysis = None
# Issue #73: an analysis stored by an older parser does not carry the multi-instance
# blocker yet; show it here exactly as adopt_vip enforces it.
count = int((analysis or {}).get("instance_count") or 0) if isinstance(analysis, dict) else 0
if count > 1:
from services.keepalived_parser import MULTI_INSTANCE_MARKER, multi_instance_blocker
for cand in analysis.get("candidates") or []:
blockers = cand.setdefault("blockers", [])
if not any(MULTI_INSTANCE_MARKER in b for b in blockers):
blockers.append(multi_instance_blocker(count))
cand["adoptable"] = False
return {
"agent_id": row["agent_id"],
"agent_name": row["agent_name"],
@@ -1133,6 +1215,7 @@ async def _collect_instance_participants(conn, *, pool_id: int, vrid: int, virtu
"agent_id": r["agent_id"], "agent_name": r["agent_name"],
"ip_address": r["ip_address"], "config_hash": r["config_hash"],
"auth_pass_encrypted": r["auth_pass_encrypted"],
"instance_count": int((analysis or {}).get("instance_count") or 0),
"candidate": cand,
})
break
@@ -1206,7 +1289,8 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
except (TypeError, ValueError):
raise HTTPException(status_code=400, detail="prefix_length must be an integer")
from services.keepalived_parser import remaining_blockers
from services.keepalived_parser import (
MULTI_INSTANCE_MARKER, multi_instance_blocker, remaining_blockers)
vrid = vip_fields.get("virtual_router_id")
virtual_ip = vip_fields.get("virtual_ip")
@@ -1236,6 +1320,9 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
prefix_supplied=vip_fields.get("prefix_length") is not None,
accept_data_loss=bool(payload.get("accept_data_loss")),
)
# Issue #73: also for discoveries analysed before the parser reported this itself.
if p["instance_count"] > 1 and not any(MULTI_INSTANCE_MARKER in b for b in rem):
rem.append(multi_instance_blocker(p["instance_count"]))
if rem:
raise HTTPException(status_code=422, detail={
"message": f"This keepalived config cannot be adopted as-is ({p['agent_name']})",
@@ -1263,18 +1350,21 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
# this VRID in the pool (the clash check above would have refused first);
# * a node whose file carries our ownership marker (is_managed) is ours already.
adopted_ids = [p["agent_id"] for p in participants]
# Issue #73: every address of the instance, not only the first.
extras = _extra_ips(vip_fields.get("additional_ips") or [])
all_addrs = [virtual_ip] + [e["address"] for e in extras]
stranded = await conn.fetch("""
SELECT a.name AS agent_name, a.pool_id, COALESCE(a.enabled, TRUE) AS enabled,
d.parse_error
FROM vip_discoveries d
JOIN agents a ON a.id = d.agent_id
LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
WHERE d.raw_config_masked LIKE '%' || $1 || '%'
WHERE d.raw_config_masked LIKE ANY($1::text[])
AND NOT (a.id = ANY($2::int[]))
AND COALESCE(av.is_active, FALSE) = FALSE
AND d.is_managed = FALSE
ORDER BY a.name
""", virtual_ip, adopted_ids)
""", [f"%{a}%" for a in all_addrs], adopted_ids)
if stranded:
def _why(r):
if r["parse_error"]:
@@ -1286,7 +1376,7 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
return "it does not report this vrrp_instance"
named = "; ".join(f"{r['agent_name']} — {_why(r)}" for r in stranded)
raise HTTPException(status_code=422, detail=(
f"These node(s) also reference {virtual_ip} but cannot be taken over with the rest "
f"These node(s) also reference {', '.join(all_addrs)} but cannot be taken over with the rest "
f"of the instance, so adopting now would rewrite the others and leave them running "
f"an unmanaged config on the same address: {named}. Resolve that first, then adopt."))
@@ -1296,11 +1386,17 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
# the sharpest: the design refuses to GUESS a netmask for a live VIP, and quietly copying
# one node's netmask onto another is the same change by another name.
def _vfield(p, key):
return (p["candidate"].get("vip") or {}).get(key)
value = (p["candidate"].get("vip") or {}).get(key)
if key == "additional_ips":
# Hashable and order-free; an analysis from before v1.12.0 has no key → none.
return tuple(sorted((e["address"], e["prefix_length"])
for e in _extra_ips(value or [])))
return value
for key, label in (("prefix_length", "prefix length"),
("use_unicast", "unicast/multicast mode"),
("track_haproxy", "HAProxy tracking")):
("track_haproxy", "HAProxy tracking"),
("additional_ips", "additional addresses")):
seen = {}
for p in participants:
seen.setdefault(_vfield(p, key), []).append(p["agent_name"])
@@ -1370,9 +1466,9 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
f"holding the VIP. Register those nodes as agents so they report their config, "
f"then adopt again."))
# One active VIP per agent — the delivery endpoint serves a single keepalived.conf per
# node, so a second active membership never converges. create/update enforce this via
# _validate_members_against_pool; adoption did not call it at all.
# Adoption hands a node's whole keepalived.conf over to OpenManager. A node that is
# already a member of a managed VIP is not taken over a second time (issue #73 allows
# several VIPs per node for VIPs created here, not for a second adoption).
busy = await conn.fetchrow("""
SELECT a.name AS agent_name, v.name AS vip_name
FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id
@@ -1396,19 +1492,20 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
f"config must keep its VRID, so resolve the collision first"))
async with conn.transaction():
await _lock_and_check_addresses(conn, virtual_ip, extras, None)
vip_id = await conn.fetchval("""
INSERT INTO vip_instances
(name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
is_active, last_config_status, adopted_at, created_by)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',CURRENT_TIMESTAMP,$11)
is_active, last_config_status, adopted_at, created_by, additional_ips)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',CURRENT_TIMESTAMP,$11,$12::jsonb)
RETURNING id
""", name, payload.get("description") or f"Adopted from {disc['agent_name']}",
disc["pool_id"], virtual_ip, int(vip_fields["prefix_length"]), int(vrid),
int(vip_fields.get("advert_int") or 1),
disc["auth_pass_encrypted"], # already Fernet-encrypted at ingest
bool(vip_fields.get("use_unicast")), bool(vip_fields.get("track_haproxy")),
current_user["id"])
current_user["id"], json.dumps(extras))
# Every node of the instance becomes a member with the role/priority/interface ITS
# OWN file declares, and each carries its own one-shot takeover authorisation pinned
# to the hash of the file we analysed on THAT node. The guard stays per-node: a file
+100
View File
@@ -183,6 +183,8 @@ def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
lines.append(" }")
lines.append(" virtual_ipaddress {")
lines.append(f' {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
for extra in vip.get("additional_ips") or []:
lines.append(f' {extra["address"]}/{int(extra["prefix_length"])} dev {iface}')
lines.append(" }")
if track:
lines.append(" track_script {")
@@ -190,3 +192,101 @@ def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
lines.append(" }")
lines.append("}")
return "\n".join(lines) + "\n"
# ---------------------------------------------------------------------------
# Issue #73 — several VIPs on one node
# ---------------------------------------------------------------------------
# A node deploys ONE keepalived.conf, and each member row stores the full single-VIP file
# render_keepalived_conf produced at apply time. A node that belongs to several VIPs gets
# those applied files merged into one: global_defs once, then every VIP's vrrp_script and
# vrrp_instance. The health-check script is renamed per VIP (chk_haproxy_<id>) because each
# VIP's failover weight is derived from its own priorities, and keepalived needs unique
# script names. A node with a single VIP is served its applied file unchanged.
class KeepalivedComposeError(ValueError):
"""An applied snapshot is not in the shape render_keepalived_conf produces."""
def _top_level_blocks(conf: str) -> List[tuple]:
"""Split our own render into (name, lines) top-level blocks; comment lines are skipped."""
blocks: List[tuple] = []
current: Optional[List[str]] = None
depth = 0
for line in conf.split("\n"):
stripped = line.strip()
if current is None:
if not stripped or stripped.startswith("#"):
continue
if not stripped.endswith("{"):
raise KeepalivedComposeError(f"unexpected top-level line: {stripped[:60]}")
current = [line]
depth = 1
continue
current.append(line)
if stripped.startswith("auth_pass "):
continue # the secret may contain braces; it is a single token, never a block
depth += stripped.count("{") - stripped.count("}")
if depth == 0:
blocks.append((" ".join(current[0].strip().rstrip("{").split()), current))
current = None
elif depth < 0:
raise KeepalivedComposeError("unbalanced braces")
if current is not None:
raise KeepalivedComposeError("unterminated block")
return blocks
def compose_node_keepalived_conf(applied: List[tuple]) -> str:
"""Merge the applied single-VIP files of one node into the file the node deploys.
`applied` is [(vip_id, applied_config_content), ...]. One entry is returned unchanged,
so nodes with a single VIP keep the exact bytes (and hash) they already run. Raises
KeepalivedComposeError when a snapshot does not have the expected shape; the caller
must then leave the node alone rather than deploy a partial file.
"""
if not applied:
raise KeepalivedComposeError("nothing to compose")
if len(applied) == 1:
return applied[0][1]
header: List[str] = [f"{OWNERSHIP_MARKER} — DO NOT EDIT"]
global_defs: Optional[List[str]] = None
scripts: List[str] = []
instances: List[str] = []
for vip_id, conf in sorted(applied, key=lambda item: int(item[0])):
vip_id = int(vip_id)
for line in (conf or "").split("\n"):
if line.startswith("# VIP "):
header.append(line)
blocks = _top_level_blocks(conf or "")
names = [name for name, _ in blocks]
if names.count("global_defs") != 1 or names.count(f"vrrp_instance {_vrrp_instance_name(vip_id)}") != 1:
raise KeepalivedComposeError(f"VIP {vip_id}: applied config has an unexpected layout")
if any(n not in ("global_defs", "vrrp_script chk_haproxy", f"vrrp_instance {_vrrp_instance_name(vip_id)}")
for n in names):
raise KeepalivedComposeError(f"VIP {vip_id}: applied config has an unexpected block")
script_name = f"chk_haproxy_{vip_id}"
for name, body in blocks:
if name == "global_defs":
if global_defs is None:
global_defs = body
elif name == "vrrp_script chk_haproxy":
scripts.append("\n".join([f"vrrp_script {script_name} {{"] + body[1:]))
else:
renamed, in_track = [], False
for line in body:
s = line.strip()
if s == "track_script {":
in_track = True
elif in_track and s == "}":
in_track = False
elif in_track and s == "chk_haproxy":
line = line.replace("chk_haproxy", script_name)
renamed.append(line)
instances.append("\n".join(renamed))
out: List[str] = header + global_defs + [""]
for block in scripts:
out.extend([block, ""])
out.append("\n\n".join(instances))
return "\n".join(out) + "\n"
+61 -5
View File
@@ -31,6 +31,12 @@ import ipaddress
import re
from typing import Any, Dict, List, Optional, Tuple
from models.vip import MAX_ADDITIONAL_IPS
# Stored with every discovery (vip_discoveries.analysis_version). Bump it whenever the
# analysis or its blockers change: rows from an older parser make the agent re-report once.
ANALYSIS_VERSION = 1
# Directives `render_keepalived_conf` emits, and therefore the only ones a takeover can
# reproduce. Anything else found in a vrrp_instance is reported in `unsupported`.
_SUPPORTED_INSTANCE_KEYS = {
@@ -379,6 +385,17 @@ _DEFAULT_STATE = "BACKUP"
OUR_CHECK_SCRIPT_NAME = "chk_haproxy"
MULTI_INSTANCE_MARKER = "vrrp_instance blocks"
def multi_instance_blocker(count: int) -> str:
"""Adoption replaces the node's whole keepalived.conf with ONE instance. Not resolvable:
the text must contain neither _LOSS_MARKER nor _PREFIX_MARKER."""
return (f"the file declares {count} {MULTI_INSTANCE_MARKER}; adoption takes over the whole "
f"file, so the other instances would be removed from the node — adopting from a "
f"keepalived.conf with several instances is not supported")
def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -> Dict[str, Any]:
"""Map one parsed `vrrp_instance` onto vip_instances / vip_members fields.
@@ -399,17 +416,54 @@ def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -
f"line {d['line']}: `{d['directive']}` — OpenManager's renderer cannot reproduce "
f"this, so adopting would delete it")
# Adoption replaces the node's whole keepalived.conf with the render of ONE instance, so
# every other instance in the file would disappear from the node. Not resolvable: none of
# the wording below may contain _LOSS_MARKER or _PREFIX_MARKER.
instance_total = len(parsed.get("instances") or [])
if instance_total > 1:
blockers.append(multi_instance_blocker(instance_total))
vips = instance.get("virtual_ips") or []
if len(vips) == 0:
blockers.append("the instance declares no virtual_ipaddress — nothing to adopt")
elif len(vips) > 1:
addrs = ", ".join(v["address"] for v in vips)
blockers.append(
f"the instance carries {len(vips)} addresses ({addrs}); a managed VIP holds exactly "
f"one, so adopting would drop all but the first")
vip_entry = vips[0] if vips else None
# Issue #73: the addresses after the first become the VIP's additional addresses. Each one
# must render back exactly, so nothing about it may be guessed.
additional_ips: List[Dict[str, Any]] = []
extra_entries = vips[1:]
if len(extra_entries) > MAX_ADDITIONAL_IPS:
blockers.append(
f"the instance carries {len(vips)} addresses; a managed VIP holds at most "
f"{MAX_ADDITIONAL_IPS + 1}")
seen_addrs = {vip_entry["address"]} if vip_entry else set()
for extra in extra_entries:
addr = extra["address"]
if ipaddress.ip_address(addr).version != 4:
blockers.append(f"additional address `{addr}` is IPv6; only IPv4 addresses are managed")
continue
if addr in seen_addrs:
blockers.append(f"additional address `{addr}` is listed more than once")
continue
seen_addrs.add(addr)
prefix = extra.get("prefix_length")
if prefix is None:
blockers.append(
f"additional address `{addr}` has no /prefix in the file; add it there so the "
f"netmask cannot change on takeover, then let the agent report again")
continue
if not (1 <= int(prefix) <= 32):
blockers.append(f"additional address `{addr}/{prefix}` has an invalid prefix length")
continue
if extra.get("dev") and instance.get("interface") and extra["dev"] != instance["interface"]:
blockers.append(
f"additional address `{addr}` is bound to `dev {extra['dev']}` but the instance "
f"uses `interface {instance['interface']}`; the render always uses the instance "
f"interface")
continue
additional_ips.append({"address": addr, "prefix_length": int(prefix)})
if instance.get("virtual_router_id") is None:
blockers.append("no virtual_router_id — it cannot be guessed: a wrong VRID puts the "
"nodes in separate VRRP domains and both would claim the VIP")
@@ -468,6 +522,7 @@ def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -
"use_unicast": bool(peers),
"track_haproxy": OUR_CHECK_SCRIPT_NAME in tracked,
"auth_pass": instance.get("auth_pass"),
"additional_ips": additional_ips,
},
"member": {
"network_interface": instance.get("interface"),
@@ -526,6 +581,7 @@ def analyse_keepalived_conf(text: str) -> Dict[str, Any]:
"""Parse + map in one call: the shape the discovery endpoint stores and the UI renders."""
parsed = parse_keepalived_conf(text)
return {
"analysis_version": ANALYSIS_VERSION,
"instance_count": len(parsed["instances"]),
"sync_groups": parsed["sync_groups"],
"global_defs": parsed["global_defs"],
+48 -7
View File
@@ -18,7 +18,7 @@ os.environ.setdefault("SECRET_KEY", "test-secret-key-for-keepalived-parser-tests
from services import keepalived_config as kc # noqa: E402
from services.keepalived_parser import ( # noqa: E402
KeepalivedParseError, analyse_keepalived_conf, build_adoption_candidate,
parse_keepalived_conf,
parse_keepalived_conf, remaining_blockers,
)
@@ -78,6 +78,7 @@ def test_parses_a_handwritten_config_into_model_fields():
"use_unicast": True,
"track_haproxy": True,
"auth_pass": "s3cr3t",
"additional_ips": [],
}
assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
assert cand["peers"] == ["10.0.0.12"] and cand["unicast_src_ip"] == "10.0.0.11"
@@ -141,11 +142,46 @@ def test_directives_we_cannot_render_block_adoption():
assert _only_candidate(text)["adoptable"] is False
def test_multiple_addresses_in_one_instance_block_adoption():
text = HANDWRITTEN.replace(" 10.0.0.100/24 dev eth0",
" 10.0.0.100/24 dev eth0\n 10.0.0.101/24 dev eth0")
blockers = _blockers_for(text)
assert "2 addresses" in blockers and "10.0.0.101" in blockers
def _with_extra(line):
return HANDWRITTEN.replace(" 10.0.0.100/24 dev eth0",
" 10.0.0.100/24 dev eth0\n" + line)
def test_multiple_addresses_in_one_instance_are_adopted_as_additional_ips():
"""Issue #73: the addresses after the first become the VIP's additional addresses."""
cand = _only_candidate(_with_extra(" 10.0.0.101/25 dev eth0\n 10.0.0.102/24"))
assert cand["adoptable"] is True, cand["blockers"]
assert cand["vip"]["virtual_ip"] == "10.0.0.100"
assert cand["vip"]["additional_ips"] == [
{"address": "10.0.0.101", "prefix_length": 25},
{"address": "10.0.0.102", "prefix_length": 24},
]
def test_an_additional_address_that_cannot_render_back_exactly_blocks_for_good():
"""No prefix, another dev, IPv6 or a repeat cannot be reproduced. None of them may be
waivable: their wording must carry neither the loss nor the prefix marker."""
cases = {
" 10.0.0.101": "no /prefix",
" 10.0.0.101/24 dev eth1": "dev eth1",
" fd00::1/64 dev eth0": "IPv6",
" 10.0.0.100/24 dev eth0": "more than once",
" 10.0.0.101/40 dev eth0": "invalid prefix",
}
for line, reason in cases.items():
cand = _only_candidate(_with_extra(line))
assert cand["adoptable"] is False, line
assert reason in " | ".join(cand["blockers"]), (line, cand["blockers"])
assert remaining_blockers(cand["blockers"], prefix_supplied=True,
accept_data_loss=True), line
def test_too_many_addresses_block_adoption():
from models.vip import MAX_ADDITIONAL_IPS
lines = "\n".join(f" 10.0.1.{i}/24 dev eth0" for i in range(1, MAX_ADDITIONAL_IPS + 2))
cand = _only_candidate(_with_extra(lines))
assert cand["adoptable"] is False
assert f"at most {MAX_ADDITIONAL_IPS + 1}" in " | ".join(cand["blockers"])
def test_missing_vrid_blocks_adoption_with_the_split_brain_reason():
@@ -237,7 +273,12 @@ vrrp_instance VI_2 {
names = [c["instance_name"] for c in analysed["candidates"]]
assert names == ["VI_1", "VI_2"]
assert [c["vip"]["virtual_ip"] for c in analysed["candidates"]] == ["10.0.0.100", "10.0.0.200"]
assert all(c["adoptable"] for c in analysed["candidates"])
# Issue #73: adoption takes over the whole file, so adopting either instance would remove
# the other from the node. Neither may be adoptable, and no flag may waive it.
for c in analysed["candidates"]:
assert c["adoptable"] is False
assert remaining_blockers(c["blockers"], prefix_supplied=True, accept_data_loss=True)
assert any("2 vrrp_instance blocks" in b for b in c["blockers"])
def test_unbalanced_braces_raise():
+2 -1
View File
@@ -460,7 +460,8 @@ def test_discovery_cache_defers_to_the_server():
assert agent_router.count('AS discovery_known') == 1, (
"the keepalived-config query must report whether a discovery row exists"
)
assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 5, (
# 7 = the five original paths + the v1.12.0 merged-delivery and compose_error paths.
assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 7, (
"every response path that knows the agent must carry the flag — the adoptable nodes are "
"precisely the not_configured ones"
)
+231
View File
@@ -0,0 +1,231 @@
"""Issue #73 (v1.12.0) — several VIPs on one node, several addresses on one VIP.
Pure tests for the merge of a node's applied single-VIP files, the extra-address rendering
and validation, plus source-level guards for the delivery / ack / adoption / migration
invariants that need a live database to exercise end to end.
"""
from __future__ import annotations
import hashlib
import os
import pathlib
import sys
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from models.vip import MAX_ADDITIONAL_IPS, VIPCreate, VIPUpdate # noqa: E402
from services import keepalived_config as kc # noqa: E402
from services.keepalived_parser import ( # noqa: E402
ANALYSIS_VERSION, analyse_keepalived_conf, parse_keepalived_conf,
)
BACKEND = pathlib.Path(__file__).resolve().parents[1]
AGENT_ROUTER = (BACKEND / "routers" / "agent.py").read_text()
VIP_ROUTER = (BACKEND / "routers" / "vip.py").read_text()
MIGRATIONS = (BACKEND / "database" / "migrations.py").read_text()
NODE = {"role": "MASTER", "priority": 150, "network_interface": "eth0",
"agent_id": 1, "ip_address": "10.0.0.11"}
PEER = {"role": "BACKUP", "priority": 100, "network_interface": "eth0",
"agent_id": 2, "ip_address": "10.0.0.12"}
def _vip(vid, ip, vrid, **over):
base = {"id": vid, "name": f"vip-{vid}", "virtual_ip": ip, "prefix_length": 24,
"virtual_router_id": vrid, "advert_int": 1, "use_unicast": True,
"track_haproxy": True}
base.update(over)
return base
def _applied(vip, auth="s3cr3t", this=NODE, members=(NODE, PEER)):
peers = [m["ip_address"] for m in members if m["agent_id"] != this["agent_id"]]
return kc.render_keepalived_conf(vip=vip, members=list(members), this_agent=this,
peer_ips=peers, auth_pass_plain=auth)
A = _vip(3, "10.0.0.100", 51)
B = _vip(7, "10.0.0.200", 52, track_haproxy=False)
class TestCompose:
def test_single_vip_is_served_byte_identical(self):
conf = _applied(A)
assert kc.compose_node_keepalived_conf([(3, conf)]) == conf
def test_two_vips_merge_into_one_valid_file(self):
out = kc.compose_node_keepalived_conf([(3, _applied(A)), (7, _applied(B))])
assert out.count("global_defs {") == 1
assert out.startswith(kc.OWNERSHIP_MARKER)
assert "vrrp_script chk_haproxy_3 {" in out
assert "vrrp_script chk_haproxy {" not in out
assert "vrrp_instance VI_3 {" in out and "vrrp_instance VI_7 {" in out
assert out.index("VI_3") < out.index("VI_7")
assert " chk_haproxy_3\n" in out
parsed = parse_keepalived_conf(out)
assert [i["instance_name"] for i in parsed["instances"]] == ["VI_3", "VI_7"]
assert [i["track_scripts"] for i in parsed["instances"]] == [["chk_haproxy_3"], []]
def test_order_follows_vip_id_not_input_order(self):
a, b = (3, _applied(A)), (7, _applied(B))
assert kc.compose_node_keepalived_conf([b, a]) == kc.compose_node_keepalived_conf([a, b])
def test_each_instance_keeps_its_own_failover_weight(self):
a2 = _vip(9, "10.0.0.150", 53)
low_peer = {**PEER, "priority": 50}
out = kc.compose_node_keepalived_conf([
(3, _applied(A)), (9, _applied(a2, members=(NODE, low_peer)))])
assert "weight -51" in out and "weight -101" in out
def test_braces_in_the_secret_do_not_break_the_merge(self):
out = kc.compose_node_keepalived_conf([(3, _applied(A, auth="a}b{")),
(7, _applied(B, auth="}x"))])
assert "auth_pass a}b{" in out and "auth_pass }x" in out
assert out.count("vrrp_instance") == 2
@pytest.mark.parametrize("bad", [
"",
"global_defs {\n}\n",
"vrrp_instance VI_3 {\n state MASTER\n",
"something else\n",
])
def test_unexpected_snapshot_fails_closed(self, bad):
with pytest.raises(kc.KeepalivedComposeError):
kc.compose_node_keepalived_conf([(3, _applied(A)), (7, bad)])
def test_foreign_block_fails_closed(self):
tampered = _applied(B) + "virtual_server 10.0.0.1 80 {\n}\n"
with pytest.raises(kc.KeepalivedComposeError):
kc.compose_node_keepalived_conf([(3, _applied(A)), (7, tampered)])
def test_snapshot_of_another_vip_id_fails_closed(self):
with pytest.raises(kc.KeepalivedComposeError):
kc.compose_node_keepalived_conf([(3, _applied(A)), (8, _applied(B))])
def test_nothing_to_compose_raises(self):
with pytest.raises(kc.KeepalivedComposeError):
kc.compose_node_keepalived_conf([])
class TestAdditionalAddresses:
def test_empty_list_renders_byte_identical(self):
assert _applied({**A, "additional_ips": []}) == _applied(A)
def test_extras_follow_the_primary_on_the_same_interface(self):
conf = _applied({**A, "additional_ips": [
{"address": "10.0.0.101", "prefix_length": 24},
{"address": "192.168.5.9", "prefix_length": 32}]})
lines = [l.strip() for l in conf.splitlines()]
i = lines.index("10.0.0.100/24 dev eth0")
assert lines[i + 1:i + 3] == ["10.0.0.101/24 dev eth0", "192.168.5.9/32 dev eth0"]
def test_rendered_extras_are_adopted_back_unchanged(self):
extras = [{"address": "10.0.0.101", "prefix_length": 25}]
conf = _applied({**A, "additional_ips": extras})
cand = analyse_keepalived_conf(conf)["candidates"][0]
assert cand["adoptable"] is True, cand["blockers"]
assert cand["vip"]["additional_ips"] == extras
def _members():
return [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150}]
class TestModels:
def test_default_is_empty(self):
v = VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members())
assert v.additional_ips == []
def test_address_is_normalised_and_prefix_defaults(self):
v = VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members(),
additional_ips=[{"address": " 10.0.0.101 "}])
assert v.additional_ips[0].model_dump() == {"address": "10.0.0.101", "prefix_length": 24}
@pytest.mark.parametrize("extras", [
[{"address": "10.0.0.100"}],
[{"address": "10.0.0.101"}, {"address": "10.0.0.101"}],
[{"address": "fd00::1"}],
[{"address": "nope"}],
[{"address": "10.0.0.101", "prefix_length": 33}],
[{"address": f"10.0.1.{i}"} for i in range(1, MAX_ADDITIONAL_IPS + 2)],
])
def test_invalid_extras_are_rejected(self, extras):
with pytest.raises(ValueError):
VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members(),
additional_ips=extras)
def test_update_omitted_keeps_and_empty_clears(self):
assert VIPUpdate().additional_ips is None
assert VIPUpdate(additional_ips=[]).additional_ips == []
class TestDeliveryAndAckGuards:
def test_failures_answer_with_statuses_the_agent_ignores(self):
assert '"status": "compose_error"' in AGENT_ROUTER
generic = AGENT_ROUTER.split("keepalived-config delivery failed", 1)[1][:600]
assert '"status": "unavailable"' in generic
assert "not_configured" not in generic.split("return", 1)[1].split("}", 1)[0]
def test_merged_delivery_hash_is_md5_of_the_composed_text(self):
assert 'hashlib.md5(composed.encode("utf-8")).hexdigest()' in AGENT_ROUTER
out = kc.compose_node_keepalived_conf([(3, _applied(A)), (7, _applied(B))])
assert len(hashlib.md5(out.encode("utf-8")).hexdigest()) == 32
def test_only_applied_snapshots_are_merged(self):
q = AGENT_ROUTER.split("async def _node_applied_vips", 1)[1].split("async def ", 1)[0]
assert "v.is_active = TRUE" in q
assert "m.applied_config_content IS NOT NULL" in q
def test_ack_failure_to_merge_is_not_a_500(self):
ack = AGENT_ROUTER.split("acks the merged file once", 1)[1][:1200]
assert "except Exception" in ack and "composed_hash = None" in ack
def test_discovery_known_asks_old_analyses_to_be_reported_again(self):
assert "vd.analysis_version >= $2" in AGENT_ROUTER
assert ANALYSIS_VERSION >= 1
assert analyse_keepalived_conf(_applied(A))["analysis_version"] == ANALYSIS_VERSION
class TestAdoptionGuards:
def test_apply_refuses_an_inflight_adoption_of_a_multi_instance_file(self):
apply = VIP_ROUTER.split("async def apply_vip", 1)[1].split("\nasync def ", 1)[0]
assert "d.config_hash = m.takeover_expected_hash" in apply
assert "(d.analysis->>'instance_count')::int, 0) > 1" in apply
assert "status_code=409" in apply
def test_old_analyses_get_the_blocker_in_the_list_and_on_adopt(self):
assert VIP_ROUTER.count("multi_instance_blocker(") >= 2
row_api = VIP_ROUTER.split("def _discovery_row_to_api", 1)[1].split("\ndef ", 1)[0]
assert 'cand["adoptable"] = False' in row_api
def test_address_writes_take_the_address_lock(self):
for fn in ("create_vip", "update_vip", "adopt_vip"):
body = VIP_ROUTER.split(f"async def {fn}", 1)[1].split("\nasync def ", 1)[0]
assert "_lock_and_check_addresses(" in body, fn
class TestDeltaMigration:
def test_versions(self):
assert "SCHEMA_VERSION = 13" in MIGRATIONS
assert "_DELTA_BASE_VERSION = 12" in MIGRATIONS
assert "return [(13, ensure_vip_multi_address_columns)]" in MIGRATIONS
def test_full_sequence_also_creates_the_columns(self):
inner = MIGRATIONS.split("async def _run_all_migrations_inner", 1)[1].split("\nasync def ", 1)[0]
assert inner.index("await ensure_vip_tables()") < inner.index(
"await ensure_vip_multi_address_columns()")
def test_delta_path_does_not_reseed_roles(self):
run = MIGRATIONS.split("async def run_all_migrations", 1)[1].split("\nasync def ", 1)[0]
delta = run.split("applied_version >= _DELTA_BASE_VERSION", 1)[1].split("else:", 1)[0]
assert "_delta_steps()" in delta
assert "_run_all_migrations_inner" not in delta
assert "update_system_roles" not in delta
def test_columns_are_additive_with_safe_defaults(self):
body = MIGRATIONS.split("async def ensure_vip_multi_address_columns", 1)[1].split(
"\nasync def ", 1)[0].replace('"\n "', "")
assert "ADD COLUMN IF NOT EXISTS additional_ips JSONB NOT NULL DEFAULT '[]'::jsonb" in body
assert "ADD COLUMN IF NOT EXISTS analysis_version SMALLINT NOT NULL DEFAULT 0" in body
+2 -2
View File
@@ -1,5 +1,5 @@
{
"version": "1.11.3",
"releaseName": "Config Import keeps stick-table and stick rules instead of dropping them",
"version": "1.12.0",
"releaseName": "Several keepalived VIPs on one node, and several addresses on one VIP",
"releaseDate": "2026-10-16"
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "haproxy-openmanager-frontend",
"version": "1.11.3",
"version": "1.12.0",
"description": "HAProxy Load Balancer Management UI",
"license": "AGPL-3.0-or-later",
"dependencies": {
+53 -4
View File
@@ -6,7 +6,7 @@ import {
import {
PlusOutlined, EditOutlined, DeleteOutlined, ReloadOutlined, WarningOutlined,
PlayCircleOutlined, SyncOutlined, CrownOutlined, ClockCircleOutlined, FileSearchOutlined,
InfoCircleOutlined
InfoCircleOutlined, MinusCircleOutlined
} from '@ant-design/icons';
import { useCluster } from '../contexts/ClusterContext';
import { extractApiError } from '../utils/apiError';
@@ -76,9 +76,13 @@ const mergeBlockers = (lists) => {
return Array.from(seen.values());
};
// Mirrors backend/models/vip.py MAX_ADDITIONAL_IPS.
const MAX_ADDITIONAL_IPS = 19;
const IPV4_RE = /^(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}$/;
// This component uses raw fetch(), but extractApiError expects an axios-shaped error
// (err.response.data). Read the fetch Response body and reuse the envelope-aware extractor
// so backend messages — e.g. the 409 "node already in VIP X" — actually reach the user.
// so backend messages — e.g. the 409 "address already used by VIP X" — actually reach the user.
const fetchApiError = async (res, fallback) => {
try { const data = await res.json(); return extractApiError({ response: { data } }, fallback); }
catch { return fallback; }
@@ -351,7 +355,8 @@ const VIPManagement = () => {
setSelectedPoolId(null);
setMemberRows([]);
form.resetFields();
form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true });
form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true,
additional_ips: [] });
setModalVisible(true);
};
@@ -365,6 +370,7 @@ const VIPManagement = () => {
virtual_ip: vip.virtual_ip, prefix_length: vip.prefix_length,
virtual_router_id: vip.virtual_router_id, advert_int: vip.advert_int,
use_unicast: vip.use_unicast, track_haproxy: vip.track_haproxy,
additional_ips: (vip.additional_ips || []).map((a) => ({ ...a })),
});
setModalVisible(true);
};
@@ -478,7 +484,19 @@ const VIPManagement = () => {
const columns = [
{ title: 'Name', dataIndex: 'name', key: 'name' },
{ title: 'Virtual IP', key: 'vip', render: (_, v) => <Text code>{v.virtual_ip}/{v.prefix_length}</Text> },
{ title: 'Virtual IP', key: 'vip', render: (_, v) => {
const extra = v.additional_ips || [];
return (
<Space size={4}>
<Text code>{v.virtual_ip}/{v.prefix_length}</Text>
{extra.length > 0 && (
<Tooltip title={extra.map((a) => `${a.address}/${a.prefix_length}`).join(', ')}>
<Tag>+{extra.length}</Tag>
</Tooltip>
)}
</Space>
);
} },
{ title: 'Pool', dataIndex: 'pool_name', key: 'pool' },
{ title: 'VRID', dataIndex: 'virtual_router_id', key: 'vrid' },
{ title: 'Members / Live state', key: 'members', render: renderMembers },
@@ -843,6 +861,37 @@ const VIPManagement = () => {
</Form.Item>
</Space>
<Form.List name="additional_ips">
{(fields, { add, remove }) => (
<div style={{ marginBottom: 16 }}>
<Text strong>Additional addresses</Text>
<div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
Optional. Extra IPv4 addresses carried by the same VRRP instance — they move between
nodes together with the virtual IP, on the same interface (up to {MAX_ADDITIONAL_IPS}).
</div>
{fields.map(({ key, name, ...rest }) => (
<Space key={key} align="baseline">
<Form.Item {...rest} name={[name, 'address']}
rules={[{ required: true, message: 'Address required' },
{ pattern: IPV4_RE, message: 'Enter an IPv4 address' }]}>
<Input placeholder="10.0.0.101" style={{ width: 180 }} />
</Form.Item>
<Form.Item {...rest} name={[name, 'prefix_length']}
rules={[{ required: true, message: 'Prefix required' }]}>
<InputNumber min={1} max={32} addonBefore="/" />
</Form.Item>
<MinusCircleOutlined onClick={() => remove(name)} />
</Space>
))}
<Button type="dashed" size="small" icon={<PlusOutlined />}
disabled={fields.length >= MAX_ADDITIONAL_IPS}
onClick={() => add({ prefix_length: form.getFieldValue('prefix_length') || 24 })}>
Add address
</Button>
</div>
)}
</Form.List>
<Text strong>Participating nodes</Text>
<div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
These are the HAProxy servers (agents) already enrolled in this pool — toggle which join the VIP.