mirror of
https://github.com/taylanbakircioglu/haproxy-openmanager.git
synced 2026-10-23 20:45:41 +00:00
Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 78641d87ff | |||
| 349434fc4d |
@@ -1096,6 +1096,7 @@ The HA / VIP page manages **highly-available virtual IPs** backed by Keepalived
|
||||
- **VIP dashboard**: lists each virtual IP with its pool, VRID, per-node interface, and a **live MASTER / BACKUP / FAULT** column per node (refreshed every 30s from the existing keepalive-state heartbeat pipeline).
|
||||
- **Creation form**: enter the virtual IP (+ prefix), pick the pool, and select which of the pool's HAProxy nodes participate — each with a network interface (from the node's reported interfaces), a VRRP role (MASTER/BACKUP) and a priority. VRID auto-allocates per pool; an optional VRRP secret can be set.
|
||||
- **Hands-off automation**: on **Apply**, each member's `keepalived.conf` is rendered and delivered; the agent installs Keepalived if missing, writes the config + a HAProxy health-check (`vrrp_script` + `track_script`), validates with `keepalived -t`, and starts the service. When HAProxy drops on the active node, the health-check lowers VRRP priority and the **VIP fails over automatically** to a backup.
|
||||
- **Several VIPs per node, several addresses per VIP** (v1.12.0): a node can belong to more than one VIP — each VIP is its own `vrrp_instance` (its own VRID, role and priority) and the node's `keepalived.conf` holds all of them. A VIP can also carry up to 19 **additional addresses** next to its virtual IP; they sit in the same `virtual_ipaddress` block, on the same interface, and fail over together.
|
||||
|
||||
**How it works (pull-based, isolated):**
|
||||
- New tables `vip_instances` + `vip_members`; the agent polls `GET /api/agents/{name}/keepalived-config` (key-authenticated, agent-bound) and converges. It is fully **isolated** from the global HAProxy apply flow — VIP changes never regenerate `haproxy.cfg`.
|
||||
@@ -2475,6 +2476,7 @@ Developed with ❤️ for the HAProxy community
|
||||
|
||||
## Release Notes
|
||||
|
||||
- **v1.12.0** (2026-10-16) — **Several keepalived VIPs on one node, and several addresses on one VIP**: a node could belong to only one VIP (the form refused a second one with "node already in VIP …") and a VIP held exactly one address. A node may now be a member of several VIPs: each VIP stays its own `vrrp_instance` with its own VRID, role and priority, and the delivery endpoint merges the node's applied VIPs into the one `keepalived.conf` the node deploys — one `global_defs`, then per VIP a `vrrp_script chk_haproxy_<id>` (each VIP keeps its own failover weight) and its `vrrp_instance`, ordered by VIP id. A node with a single VIP receives exactly the bytes and hash it received before, so existing installs see no change and no reload. A VIP can also carry up to 19 **additional addresses** (IPv4 + prefix) in the same `virtual_ipaddress` block and on the same interface; they are checked against node IPs and every other VIP's addresses under a lock, kept in the applied snapshot, and restored by Reject. The merged file passes `keepalived -t` with no output (which older agents require) on keepalived 2.0.19, 2.1.5, 2.2.4, 2.2.7 and 2.2.8; on 2.0.19, 2.1.5 and 2.2.8 a reload that adds a second VIP did not drop the first one's address, and a VIP with 20 addresses came up. If a node's VIPs cannot be merged, the node is left untouched and its members show the reason. A database error during delivery now answers a status the agent ignores instead of `not_configured`, which on a managed node meant teardown. **Adoption** refuses a `keepalived.conf` with several `vrrp_instance` blocks (taking it over would remove the others; earlier versions allowed it), now accepts an instance with several addresses (the extra ones become additional addresses), and asks agents to re-report files analysed by the older parser once. Schema `13` is a **delta** from `12`: two additive columns, built-in roles are not re-seeded. No agent change. See the upgrade notes for rolling-deploy and downgrade rules.
|
||||
- **v1.11.3** (2026-10-16) — **Config Import keeps `stick-table` and `stick` rules instead of dropping them**: the import parser had no branch for `stick-table` (frontend or backend) or for backend `stick on|match|store-request|store-response`, so those lines were discarded **without a warning**. A frontend that kept its `track-sc0` / `sc_*` counter rules then received the renderer's fallback table (`type ip size 100k expire 30s store http_req_rate(10s)`), so a re-defined table such as `size 200k expire 3h store http_req_rate(10s),gpc0_rate(10s),gpt0` was silently replaced by the default while the `gpc0` rules imported alongside it referenced counters the table did not store. These lines are now imported as-is into the frontend/backend **Options** field (no schema change) with an import warning, and an import that carries only a stick-table keeps the existing option lines. The table the platform generates itself for Rate Limit / WAF `rate_limit` rules is recognised and not stored, so a round-trip of a generated config shows no change and later rule edits still take effect. The import preview also deleted every `http-request track-sc0 src` / `sc_http_req_rate(0)` line as "auto-generated"; it now does so only where the platform regenerates them (frontend Rate Limit or a WAF `rate_limit` rule), so operator trackers on other frontends survive. Backends now render a single `stick-table` (the first; a second one is a fatal HAProxy error) and the preview warns about a table that does not store `http_req_rate` while Rate Limit is active, a duplicate table left in Request Headers, `stick on` without a table, and `peers` references. Configs without stick lines import and render byte-for-byte as before. No schema change, no agent change.
|
||||
- **v1.11.2** (2026-10-01) — **Read-only roles read SSL certificates, and private keys need `ssl.download`**: the UI never fills in `user_pool_access`, so every non-admin user created there has no pool access, and the SSL list (`?cluster_id=`) and SSL detail answered **403** on read while clusters, frontends, backends, WAF rules, agents and VIPs were readable. A role that grants `ssl.read` (or `waf.read` for WAF stats) now reads that resource on every cluster; without it the pool check still applies, and write endpoints are unchanged. Separately, `GET /api/ssl/certificates/{id}` and the SSL diff in Apply Management returned the **private key to any authenticated user**, including `viewer`. The key is now included only with `ssl.download` plus pool access; everyone else gets `private_key_content: null` with `private_key_redacted: true`, and the UI shows the key as hidden while editing keeps the stored key. Custom roles with `ssl.update` but no `ssl.download` lose sight of the key. No schema change, no agent change.
|
||||
- **v1.11.1** (2026-08-15) — **A node can no longer be hidden from adoption for good, and Config Import reaches a freshly installed agent**: the agent posts an unmanaged `keepalived.conf` for adoption and caches the hash of what it sent, so the file (which carries the VRRP password) is re-posted only when it changes. Delivery was judged by `curl`'s exit code, which is **0 for 5xx as well**, so a report the server *rejected* was recorded as delivered — and because a hand-maintained config does not change on its own, that node dropped out of *Unmanaged keepalived detected* permanently, curable only by deleting a cache file on the node by hand. Now: the report is cached only on a 2xx; `GET /agents/{name}/keepalived-config` tells the agent whether the server actually holds a discovery for it, so nodes stuck from earlier releases recover by themselves on the next poll; a **400/413/422** records the refusal so identical bytes are not re-posted forever (4xx and 5xx are never sampled out of the request log, so an unattended loop would write a row carrying the whole config every cycle), while **401 and 404 keep retrying** because in this system they mean a token rotation or an agent row briefly absent, not a bad payload. The same exit-code mistake in the *clear* path is fixed too, where it left a managed node still being offered for adoption. Separately, **Config Import silently did nothing on any agent that had never self-upgraded**: `check_config_requests` was defined in the installer and in the self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by `type`, so the operator asked a node for its `haproxy.cfg` and nothing arrived, with no error anywhere. And the cluster whose `keepalived_config_path` is handed to an agent is now resolved deterministically — a pool may hold several clusters, and the unordered join could return a different one between polls, pointing the agent at a file that does not exist. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema change.
|
||||
|
||||
@@ -1,3 +1,59 @@
|
||||
# Upgrade Notes — v1.12.0 (several keepalived VIPs on one node, several addresses on one VIP)
|
||||
|
||||
**Backend + UI change, schema `13` (delta), no agent change.** Agents of every version keep working;
|
||||
no agent upgrade is needed.
|
||||
|
||||
- **Schema 12 → 13 is a delta.** A database at 12 adds two columns —
|
||||
`vip_instances.additional_ips` (JSONB, default `[]`) and `vip_discoveries.analysis_version`
|
||||
(SMALLINT, default `0`) — and nothing else runs. **Built-in roles are not re-seeded**, so role
|
||||
customizations survive this upgrade (unlike v1.10.4 and v1.11.0). A database older than 12 runs the
|
||||
full sequence as before, including the role re-seed.
|
||||
- **Nodes with one VIP see no change.** Their `keepalived.conf` keeps the same bytes and hash, so the
|
||||
agent does not rewrite or reload anything.
|
||||
- **Check for nodes that already belong to several VIPs before upgrading.** Earlier versions refused
|
||||
this when a VIP was saved, but without a lock, so two saves at the same moment could still create
|
||||
it. Such a node switches from one VIP's file to the merged file on its first poll after the upgrade:
|
||||
```sql
|
||||
SELECT a.name, count(*) FROM vip_members m
|
||||
JOIN vip_instances v ON v.id = m.vip_id JOIN agents a ON a.id = m.agent_id
|
||||
WHERE v.is_active GROUP BY a.name HAVING count(*) > 1;
|
||||
```
|
||||
- **Rolling deploy:** do not add a second VIP to a node until **every backend replica** runs v1.12.0.
|
||||
An older replica serves a node only one of its VIPs, so the node would alternate between two files.
|
||||
An older replica applying a VIP also drops that VIP's additional addresses.
|
||||
- **Rollback / downgrade:** first reduce every node to one VIP and remove the additional addresses
|
||||
(edit, then Apply), then downgrade. The two new columns can stay; older versions ignore them.
|
||||
- **How a node with several VIPs is served.** Each VIP is still rendered and applied on its own; the
|
||||
delivery endpoint merges the node's applied VIPs into one file: `global_defs` once, then per VIP a
|
||||
`vrrp_script chk_haproxy_<id>` and a `vrrp_instance VI_<id>`, ordered by VIP id. VRIDs are unique per
|
||||
pool, so the instances never collide. One acknowledgement from the node converges all of its VIPs.
|
||||
Applying two VIPs one after the other reloads keepalived twice; measured on keepalived 2.0.19,
|
||||
2.1.5 and 2.2.8, a reload that adds or removes a VIP does not drop the other VIPs' addresses.
|
||||
- **If the merge fails** (an applied snapshot that is not in the platform's own format), the node keeps
|
||||
running what it has: the agent is told nothing new, and the node's members show
|
||||
"cannot merge this node's VIPs …" with an ERROR status. Re-apply the VIPs to fix it.
|
||||
- **Delivery errors no longer tear keepalived down.** A database error while serving
|
||||
`keepalived-config` answered `not_configured`, which on a node whose file is ours meant teardown. It
|
||||
now answers `unavailable`, which agents of every version ignore.
|
||||
- **Additional addresses:** up to 19 per VIP (IPv4, prefix 1–32), on the VIP's interface on every node.
|
||||
They must not be a node's IP or any address of another active VIP.
|
||||
- **Live state is per node, not per VIP.** The MASTER/BACKUP column comes from the node's heartbeat,
|
||||
which reports one state for the node. A node that is MASTER for one VIP and BACKUP for another shows
|
||||
the same state in both rows; with several VIPs on a node, the heartbeat's config-based fallback can
|
||||
also report MASTER on a BACKUP node. Check `ip addr` on the node when it matters. The VIPs themselves
|
||||
fail over correctly; only the display is affected.
|
||||
- **Adoption:** a `keepalived.conf` with several `vrrp_instance` blocks can no longer be adopted (it
|
||||
replaces the whole file, so the other instances would be removed — earlier versions allowed it). An
|
||||
adoption made before the upgrade from such a file is refused at Apply; reject it. An instance with
|
||||
several addresses is now adoptable; the extra addresses become additional addresses. Files analysed
|
||||
by an older version are re-reported once by v1.11.1+ agents; until then the list shows them with the
|
||||
new blocker.
|
||||
|
||||
**Rollback:** see the downgrade rule above. With one VIP per node and no additional addresses,
|
||||
rolling back is safe.
|
||||
|
||||
---
|
||||
|
||||
# Upgrade Notes — v1.11.3 (Config Import keeps `stick-table` and `stick` rules)
|
||||
|
||||
**Backend change, no schema change, no agent change.** No `SCHEMA_VERSION` bump and no agent
|
||||
|
||||
@@ -1796,7 +1796,24 @@ async def ensure_agent_activity_logs_table():
|
||||
# Additive + idempotent; no existing table is altered, agents never read this
|
||||
# table. Same caveat as v1.10.4: this bump re-seeds the four built-in roles to
|
||||
# their defaults, so export role customizations before upgrading.
|
||||
SCHEMA_VERSION = 12
|
||||
#
|
||||
# v1.12.0 (issue #73 — several VIPs per node, several addresses per VIP): bumped
|
||||
# 12 -> 13 as a DELTA. A database already at 12 runs only the steps listed in
|
||||
# _delta_steps() — two additive columns — and keeps its role customizations; the full
|
||||
# sequence (and with it the built-in role re-seed) runs only for databases older than
|
||||
# _DELTA_BASE_VERSION or without a marker, exactly as before. Every delta step is ALSO
|
||||
# called from _run_all_migrations_inner(), so both paths end at the same schema.
|
||||
# A future change that alters seeded/role data needs a full bump: raise
|
||||
# _DELTA_BASE_VERSION to the new SCHEMA_VERSION instead of adding a delta step.
|
||||
SCHEMA_VERSION = 13
|
||||
_DELTA_BASE_VERSION = 12
|
||||
|
||||
|
||||
def _delta_steps():
|
||||
"""(version, step) pairs a database at >= _DELTA_BASE_VERSION runs instead of the full
|
||||
sequence. Each step takes the migration connection and raises on failure, so the version
|
||||
marker is not written and the next start retries."""
|
||||
return [(13, ensure_vip_multi_address_columns)]
|
||||
|
||||
|
||||
async def run_all_migrations():
|
||||
@@ -1845,7 +1862,13 @@ async def run_all_migrations():
|
||||
logger.info(f"Schema already at version {applied_version} (>= {SCHEMA_VERSION}); skipping migration run.")
|
||||
return
|
||||
|
||||
await _run_all_migrations_inner()
|
||||
if applied_version is not None and applied_version >= _DELTA_BASE_VERSION:
|
||||
for version, step in _delta_steps():
|
||||
if version > applied_version:
|
||||
logger.info(f"Applying schema delta {applied_version} -> {version}")
|
||||
await step(lock_conn)
|
||||
else:
|
||||
await _run_all_migrations_inner()
|
||||
|
||||
try:
|
||||
await lock_conn.execute("""
|
||||
@@ -1935,6 +1958,7 @@ async def _run_all_migrations_inner():
|
||||
# Issue #27 — HA/VIP Keepalived management (v1.7.0): two brand-new tables.
|
||||
# MUST run after its FK targets (haproxy_cluster_pools/agents/users), all created above.
|
||||
await ensure_vip_tables()
|
||||
await ensure_vip_multi_address_columns()
|
||||
|
||||
# v1.9.0 — CSR creation: brand-new ssl_csrs table. FK-references
|
||||
# ssl_certificates/users, both created above.
|
||||
@@ -2408,6 +2432,36 @@ async def ensure_vip_tables():
|
||||
if conn:
|
||||
await close_database_connection(conn)
|
||||
|
||||
|
||||
async def ensure_vip_multi_address_columns(conn=None):
|
||||
"""v1.12.0 (issue #73). Additive + idempotent:
|
||||
|
||||
* vip_instances.additional_ips — the VIP's addresses beyond virtual_ip, as a JSON list
|
||||
of {"address", "prefix_length"}. Empty for every existing VIP, which renders exactly
|
||||
as before.
|
||||
* vip_discoveries.analysis_version — which parser produced the stored analysis. Rows
|
||||
written by an older parser read 0, and the agent is asked to re-report them once.
|
||||
|
||||
Raises on failure (unlike ensure_vip_tables): code reads these columns unconditionally,
|
||||
so the schema version must not be recorded without them.
|
||||
"""
|
||||
own = conn is None
|
||||
if own:
|
||||
conn = await get_database_connection()
|
||||
try:
|
||||
# IF EXISTS: ensure_vip_tables() tolerates its own failure, and a missing VIP table
|
||||
# must keep degrading only the VIP feature, not stop the backend from starting.
|
||||
await conn.execute(
|
||||
"ALTER TABLE IF EXISTS vip_instances ADD COLUMN IF NOT EXISTS additional_ips "
|
||||
"JSONB NOT NULL DEFAULT '[]'::jsonb;")
|
||||
await conn.execute(
|
||||
"ALTER TABLE IF EXISTS vip_discoveries ADD COLUMN IF NOT EXISTS analysis_version "
|
||||
"SMALLINT NOT NULL DEFAULT 0;")
|
||||
logger.info("✅ VIP multi-address columns ensured (issue #73)")
|
||||
finally:
|
||||
if own:
|
||||
await close_database_connection(conn)
|
||||
|
||||
async def add_ssl_certificate_id_to_backend_servers():
|
||||
"""Add ssl_certificate_id column to backend_servers table for SSL certificate management"""
|
||||
conn = None
|
||||
|
||||
@@ -50,6 +50,38 @@ def _validate_ipv4(v: str) -> str:
|
||||
return s
|
||||
|
||||
|
||||
# Issue #73: a VIP may carry further addresses in the same VRRP instance. 20 addresses in
|
||||
# total keeps every advertisement within what VRRPv2 peers are known to accept.
|
||||
MAX_ADDITIONAL_IPS = 19
|
||||
|
||||
|
||||
class VIPAddressIn(BaseModel):
|
||||
address: str
|
||||
prefix_length: int = 24
|
||||
|
||||
@field_validator('address')
|
||||
@classmethod
|
||||
def _addr(cls, v):
|
||||
return str(ipaddress.ip_address(_validate_ipv4(v)))
|
||||
|
||||
@field_validator('prefix_length')
|
||||
@classmethod
|
||||
def _prefix(cls, v):
|
||||
if not isinstance(v, int) or not (1 <= v <= 32):
|
||||
raise ValueError('prefix_length must be an integer between 1 and 32 (IPv4)')
|
||||
return v
|
||||
|
||||
|
||||
def _validate_additional_ips(extras: List['VIPAddressIn'], primary: Optional[str]) -> None:
|
||||
if len(extras) > MAX_ADDITIONAL_IPS:
|
||||
raise ValueError(f'a VIP may carry at most {MAX_ADDITIONAL_IPS} additional addresses')
|
||||
addrs = [e.address for e in extras]
|
||||
if len(set(addrs)) != len(addrs):
|
||||
raise ValueError('each additional address may appear only once')
|
||||
if primary is not None and str(ipaddress.ip_address(primary)) in addrs:
|
||||
raise ValueError('an additional address must differ from virtual_ip')
|
||||
|
||||
|
||||
class VIPMemberIn(BaseModel):
|
||||
agent_id: int
|
||||
network_interface: str
|
||||
@@ -122,6 +154,7 @@ class VIPCreate(BaseModel):
|
||||
auth_pass: Optional[str] = None # plaintext on the wire; stored Fernet-encrypted
|
||||
use_unicast: bool = True
|
||||
track_haproxy: bool = True
|
||||
additional_ips: List[VIPAddressIn] = []
|
||||
members: List[VIPMemberIn]
|
||||
|
||||
@field_validator('name')
|
||||
@@ -172,6 +205,7 @@ class VIPCreate(BaseModel):
|
||||
@model_validator(mode='after')
|
||||
def _members_consistent(self):
|
||||
_validate_members(self.members)
|
||||
_validate_additional_ips(self.additional_ips, self.virtual_ip)
|
||||
return self
|
||||
|
||||
|
||||
@@ -187,6 +221,7 @@ class VIPUpdate(BaseModel):
|
||||
auth_pass: Optional[str] = None # provide only to rotate; omit to keep existing
|
||||
use_unicast: Optional[bool] = None
|
||||
track_haproxy: Optional[bool] = None
|
||||
additional_ips: Optional[List[VIPAddressIn]] = None # None keeps them; [] clears them
|
||||
members: Optional[List[VIPMemberIn]] = None
|
||||
|
||||
@field_validator('name')
|
||||
@@ -242,4 +277,6 @@ class VIPUpdate(BaseModel):
|
||||
def _members_consistent(self):
|
||||
if self.members is not None:
|
||||
_validate_members(self.members)
|
||||
if self.additional_ips is not None:
|
||||
_validate_additional_ips(self.additional_ips, self.virtual_ip)
|
||||
return self
|
||||
|
||||
+153
-24
@@ -931,16 +931,17 @@ async def delete_agent(agent_id: int, authorization: str = Header(None)):
|
||||
# with no way to be told to tear down (review MED-3). Make the operator remove it
|
||||
# from the VIP first — that stages a clean PENDING change they can apply.
|
||||
try:
|
||||
vip_member = await conn.fetchrow(
|
||||
# A node may belong to several VIPs (issue #73); name them all.
|
||||
vip_names = [r["name"] for r in await conn.fetch(
|
||||
"SELECT v.name FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
|
||||
"WHERE vm.agent_id = $1 AND v.is_active = TRUE LIMIT 1", agent_id)
|
||||
"WHERE vm.agent_id = $1 AND v.is_active = TRUE ORDER BY v.name", agent_id)]
|
||||
except Exception: # noqa: BLE001 — vip_* may not exist on older schemas; don't block delete
|
||||
vip_member = None
|
||||
if vip_member:
|
||||
vip_names = []
|
||||
if vip_names:
|
||||
await close_database_connection(conn)
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(f"This node is a member of VIP '{vip_member['name']}'. Remove it from the "
|
||||
detail=(f"This node is a member of VIP '{', '.join(vip_names)}'. Remove it from the "
|
||||
f"VIP on the HA / VIP page (and apply) before deleting the agent."))
|
||||
|
||||
await conn.execute("DELETE FROM agents WHERE id = $1", agent_id)
|
||||
@@ -2281,6 +2282,37 @@ async def get_agent_ssl_certificates(agent_name: str, since: Optional[str] = Non
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
async def _node_applied_vips(conn, agent_id: int):
|
||||
"""Issue #73 — the active VIPs this node runs: every active membership that has an applied
|
||||
snapshot, ordered by VIP id (the order compose_node_keepalived_conf writes them in)."""
|
||||
return await conn.fetch("""
|
||||
SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
|
||||
v.purge_on_teardown,
|
||||
m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
|
||||
FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
|
||||
WHERE m.agent_id = $1 AND v.is_active = TRUE
|
||||
AND m.applied_config_content IS NOT NULL AND m.applied_config_content <> ''
|
||||
ORDER BY v.id
|
||||
""", agent_id)
|
||||
|
||||
|
||||
async def _flag_compose_error(conn, agent_id: int, vip_ids, exc: Exception) -> None:
|
||||
"""Show a merge failure on the node's members. The agent never acks a status it does not
|
||||
know, so without this the VIPs would sit at SYNCING with nothing saying why."""
|
||||
if not vip_ids:
|
||||
return
|
||||
msg = f"cannot merge this node's VIPs into one keepalived.conf ({exc}); re-apply the VIPs"[:500]
|
||||
try:
|
||||
await conn.execute("""
|
||||
UPDATE vip_members SET last_deploy_state='error', last_deploy_message=$3,
|
||||
last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
|
||||
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
|
||||
AND (last_deploy_state IS DISTINCT FROM 'error' OR last_deploy_message IS DISTINCT FROM $4)
|
||||
""", agent_id, list(vip_ids), msg, msg)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.error(f"could not record the keepalived compose failure: {e}")
|
||||
|
||||
|
||||
@router.get("/{agent_name}/keepalived-config")
|
||||
async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str] = Header(None)):
|
||||
"""Issue #27 (v1.7.0) — deliver this agent's APPLIED keepalived snapshot, or a
|
||||
@@ -2293,8 +2325,8 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
the agent API key is a SHARED/global install token, so the config is resolved by the
|
||||
requested agent_name and a token/name mismatch is an advisory audit log, NOT a 403
|
||||
(a hard 403 would break every VIP member whose name isn't the one row the shared token
|
||||
resolves to — review HIGH-1). Any unexpected error degrades to not_configured (B-7) so
|
||||
the agent stays inert; a node with no membership row always gets not_configured.
|
||||
resolves to — review HIGH-1). Any unexpected error degrades to a status the agent ignores
|
||||
(B-7) so the agent stays inert; a node with no membership row always gets not_configured.
|
||||
"""
|
||||
conn = None
|
||||
try:
|
||||
@@ -2303,6 +2335,7 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
# name mismatch — do not 403 (HIGH-1). Done before the agent lookup so an
|
||||
# unauthenticated caller can't probe which agent names exist.
|
||||
from auth_middleware import validate_agent_api_key
|
||||
from services.keepalived_parser import ANALYSIS_VERSION
|
||||
if not x_api_key:
|
||||
raise HTTPException(status_code=401, detail="Agent API key required")
|
||||
agent_auth = await validate_agent_api_key(x_api_key)
|
||||
@@ -2333,7 +2366,12 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
-- good: the file never changes, so the agent never speaks again. Telling it
|
||||
-- what we actually hold lets it recover on its own, with no extra request and
|
||||
-- no one having to touch the node.
|
||||
EXISTS (SELECT 1 FROM vip_discoveries vd WHERE vd.agent_id = a.id)
|
||||
--
|
||||
-- v1.12.0: only a discovery analysed by the CURRENT parser counts. An older
|
||||
-- analysis reads as unknown, so the agent re-reports once and the stored
|
||||
-- blockers/candidates catch up without anyone touching the node.
|
||||
EXISTS (SELECT 1 FROM vip_discoveries vd
|
||||
WHERE vd.agent_id = a.id AND vd.analysis_version >= $2)
|
||||
AS discovery_known
|
||||
FROM agents a
|
||||
LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
|
||||
@@ -2354,7 +2392,7 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
OR hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'),
|
||||
hc.id
|
||||
LIMIT 1
|
||||
""", agent_name)
|
||||
""", agent_name, ANALYSIS_VERSION)
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
|
||||
config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
|
||||
@@ -2367,13 +2405,57 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
m.applied_config_content, m.applied_config_hash, m.takeover_expected_hash
|
||||
FROM vip_members m JOIN vip_instances v ON v.id = m.vip_id
|
||||
WHERE m.agent_id = $1
|
||||
-- Active VIP first (an agent has at most one). With NO active VIP, pick the most
|
||||
-- RECENTLY updated inactive membership so a teardown reflects the latest delete
|
||||
-- (incl. its purge flag) — not a stale older VIP the node was once part of.
|
||||
-- Active VIP first. With NO active VIP, pick the most RECENTLY updated inactive
|
||||
-- membership so a teardown reflects the latest delete (incl. its purge flag) —
|
||||
-- not a stale older VIP the node was once part of.
|
||||
ORDER BY v.is_active DESC, v.updated_at DESC, v.id DESC
|
||||
LIMIT 1
|
||||
""", agent['id'])
|
||||
|
||||
# Issue #73: a node may belong to several active VIPs. Their applied snapshots are
|
||||
# merged into the one keepalived.conf the node deploys; a single VIP is served exactly
|
||||
# as before (same content, same hash).
|
||||
# Any failure in this block must answer with a status the agent ignores: on a node
|
||||
# whose file is ours the agent treats not_configured as "tear keepalived down". Here
|
||||
# the members are also flagged, so the operator sees why the node is not converging.
|
||||
node_vips = []
|
||||
merged = None
|
||||
if row and row['is_active']:
|
||||
try:
|
||||
node_vips = await _node_applied_vips(conn, agent['id'])
|
||||
if len(node_vips) > 1:
|
||||
from services.keepalived_config import (
|
||||
build_haproxy_check_script, compose_node_keepalived_conf)
|
||||
composed = compose_node_keepalived_conf(
|
||||
[(r['vip_id'], r['applied_config_content']) for r in node_vips])
|
||||
takeover = next((r['takeover_expected_hash'] for r in node_vips
|
||||
if r['takeover_expected_hash']), None)
|
||||
merged = {
|
||||
"desired_state": "enabled",
|
||||
"install_if_missing": True,
|
||||
"vip_id": node_vips[0]['vip_id'],
|
||||
"vip_ids": [r['vip_id'] for r in node_vips],
|
||||
"vip_name": ", ".join(str(r['vip_name']) for r in node_vips),
|
||||
"config_content": composed,
|
||||
"config_hash": hashlib.md5(composed.encode("utf-8")).hexdigest(),
|
||||
"check_script": build_haproxy_check_script()
|
||||
if any(r['track_haproxy'] for r in node_vips) else "",
|
||||
"allow_takeover": bool(takeover),
|
||||
"takeover_expected_hash": takeover,
|
||||
}
|
||||
except Exception as ce: # noqa: BLE001
|
||||
logger.error(f"keepalived compose failed for '{agent_name}': {ce}")
|
||||
await _flag_compose_error(conn, agent['id'], [r['vip_id'] for r in node_vips], ce)
|
||||
return {"agent_name": agent_name, "status": "compose_error", "config_path": config_path,
|
||||
"discovery_known": bool(agent["discovery_known"]), "keepalived": None}
|
||||
if merged is not None:
|
||||
return {"agent_name": agent_name, "status": "available", "config_path": config_path,
|
||||
"discovery_known": bool(agent["discovery_known"]), "keepalived": merged}
|
||||
if len(node_vips) == 1:
|
||||
# The most recently updated active VIP may be a pending one with nothing applied
|
||||
# yet; serving it would read as not_configured and tear down the running VIP.
|
||||
row = node_vips[0]
|
||||
|
||||
if not row:
|
||||
return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
|
||||
if not row['is_active']:
|
||||
@@ -2413,8 +2495,10 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"keepalived-config delivery failed for '{agent_name}': {e}")
|
||||
# Degrade to no-op rather than 500 (B-7) — keeps the fleet inert on any error.
|
||||
return {"agent_name": agent_name, "status": "not_configured",
|
||||
# Degrade to a no-op rather than 500 (B-7). The status must be one the agent does not
|
||||
# act on: not_configured is a teardown on a node whose file is ours, so a transient
|
||||
# database error would have taken a running VIP down.
|
||||
return {"agent_name": agent_name, "status": "unavailable",
|
||||
"config_path": "/etc/keepalived/keepalived.conf", "keepalived": None}
|
||||
finally:
|
||||
if conn:
|
||||
@@ -2476,12 +2560,54 @@ async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key:
|
||||
WHERE agent_id=$1
|
||||
""", agent['id'], state, message, config_hash, config_hash)
|
||||
else:
|
||||
await conn.execute(f"""
|
||||
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
|
||||
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
|
||||
{_retire_takeover.format(p="$6")}
|
||||
WHERE agent_id=$1 AND vip_id=$2
|
||||
""", agent['id'], int(vip_id), state, message, config_hash, config_hash)
|
||||
# Issue #73: a node running several VIPs acks the merged file once, with the first
|
||||
# VIP's id and the merged file's hash. Spread the ack over every VIP in that file;
|
||||
# each member is converged against its OWN applied hash.
|
||||
node_vips = await _node_applied_vips(conn, agent['id'])
|
||||
node_vip_ids = [r['vip_id'] for r in node_vips]
|
||||
converged = False
|
||||
if len(node_vips) > 1 and int(vip_id) in node_vip_ids:
|
||||
from services.keepalived_config import compose_node_keepalived_conf
|
||||
try:
|
||||
composed = compose_node_keepalived_conf(
|
||||
[(r['vip_id'], r['applied_config_content']) for r in node_vips])
|
||||
composed_hash = hashlib.md5(composed.encode("utf-8")).hexdigest()
|
||||
except Exception: # noqa: BLE001 — unmergeable means not converged, never a 500
|
||||
composed_hash = None
|
||||
converged = bool(composed_hash) and config_hash == composed_hash
|
||||
if converged:
|
||||
await conn.execute("""
|
||||
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
|
||||
last_deploy_hash=applied_config_hash, last_deploy_at=CURRENT_TIMESTAMP,
|
||||
updated_at=CURRENT_TIMESTAMP, takeover_expected_hash=NULL
|
||||
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
|
||||
""", agent['id'], node_vip_ids, state, message)
|
||||
else:
|
||||
await conn.execute("""
|
||||
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
|
||||
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
|
||||
WHERE agent_id=$1 AND vip_id = ANY($2::int[])
|
||||
""", agent['id'], node_vip_ids, state, message, config_hash)
|
||||
else:
|
||||
await conn.execute(f"""
|
||||
UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
|
||||
last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
|
||||
{_retire_takeover.format(p="$6")}
|
||||
WHERE agent_id=$1 AND vip_id=$2
|
||||
""", agent['id'], int(vip_id), state, message, config_hash, config_hash)
|
||||
converged = any(r['vip_id'] == int(vip_id) and r['applied_config_hash'] == config_hash
|
||||
for r in node_vips)
|
||||
if state == "enabled" and converged:
|
||||
# The file the node now runs holds only its active VIPs, so a deleted VIP's
|
||||
# membership on this node is torn down even though no teardown was sent for it.
|
||||
await conn.execute("""
|
||||
UPDATE vip_members m SET last_deploy_state='disabled',
|
||||
last_deploy_message='removed from this node''s keepalived.conf',
|
||||
last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP
|
||||
FROM vip_instances v
|
||||
WHERE v.id = m.vip_id AND m.agent_id=$1 AND v.is_active = FALSE
|
||||
AND m.last_deploy_state IS DISTINCT FROM 'disabled'
|
||||
""", agent['id'])
|
||||
return {"status": "ok"}
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -2539,7 +2665,8 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
|
||||
is_managed = bool(payload.get("is_managed"))
|
||||
config_hash = hashlib.md5(content.encode("utf-8", "replace")).hexdigest()
|
||||
|
||||
from services.keepalived_parser import analyse_keepalived_conf, KeepalivedParseError
|
||||
from services.keepalived_parser import (
|
||||
ANALYSIS_VERSION, analyse_keepalived_conf, KeepalivedParseError)
|
||||
from services.keepalived_config import encrypt_vrrp_secret
|
||||
|
||||
parse_error = None
|
||||
@@ -2562,8 +2689,8 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
|
||||
await conn.execute("""
|
||||
INSERT INTO vip_discoveries
|
||||
(agent_id, config_path, config_hash, is_managed, raw_config_masked,
|
||||
auth_pass_encrypted, analysis, parse_error, reported_at)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,CURRENT_TIMESTAMP)
|
||||
auth_pass_encrypted, analysis, parse_error, analysis_version, reported_at)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,CURRENT_TIMESTAMP)
|
||||
ON CONFLICT (agent_id) DO UPDATE SET
|
||||
config_path = EXCLUDED.config_path,
|
||||
config_hash = EXCLUDED.config_hash,
|
||||
@@ -2572,9 +2699,11 @@ async def agent_keepalived_discovery(agent_name: str, payload: dict, x_api_key:
|
||||
auth_pass_encrypted = EXCLUDED.auth_pass_encrypted,
|
||||
analysis = EXCLUDED.analysis,
|
||||
parse_error = EXCLUDED.parse_error,
|
||||
analysis_version = EXCLUDED.analysis_version,
|
||||
reported_at = CURRENT_TIMESTAMP
|
||||
""", agent['id'], config_path, config_hash, is_managed, masked, auth_enc,
|
||||
json.dumps(analysis) if analysis is not None else None, parse_error)
|
||||
json.dumps(analysis) if analysis is not None else None, parse_error,
|
||||
ANALYSIS_VERSION)
|
||||
return {"status": "recorded", "config_hash": config_hash}
|
||||
except HTTPException:
|
||||
raise
|
||||
|
||||
+157
-60
@@ -16,6 +16,7 @@ Every DB access is wrapped so a (pathological) missing vip_* relation degrades t
|
||||
empty/None result instead of a 500 (B-7) — preserving the fleet-wide no-op guarantee.
|
||||
"""
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
@@ -171,7 +172,7 @@ async def render_vip_config_masked(conn, vip_id: int):
|
||||
vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
|
||||
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
|
||||
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
|
||||
"track_haproxy": v["track_haproxy"]}
|
||||
"track_haproxy": v["track_haproxy"], "additional_ips": _extra_ips(v["additional_ips"])}
|
||||
member_dicts = [{"role": m["role"], "priority": m["priority"],
|
||||
"network_interface": m["network_interface"], "agent_id": m["agent_id"],
|
||||
"ip_address": str(m["ip_address"]) if m["ip_address"] else ""} for m in members]
|
||||
@@ -241,7 +242,8 @@ async def _capture_vip_pending_state(conn, vip_id: int):
|
||||
"vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
|
||||
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
|
||||
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
|
||||
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
|
||||
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"],
|
||||
"additional_ips": _extra_ips(v["additional_ips"])},
|
||||
"members": [{"agent_id": m["agent_id"], "network_interface": m["network_interface"],
|
||||
"role": m["role"], "priority": m["priority"]} for m in members],
|
||||
}
|
||||
@@ -282,37 +284,31 @@ async def restore_vip_from_rejected_version(conn, version_id: int):
|
||||
if not await conn.fetchrow("SELECT 1 FROM vip_instances WHERE id=$1", vip_id):
|
||||
return "the VIP no longer exists — re-create it from the HA/VIP page"
|
||||
sv, sm = ps["vip"], ps.get("members", [])
|
||||
# One-VIP-per-agent must STILL hold after reactivation: if a member was meanwhile added
|
||||
# to another active VIP (while this one was rejected/soft-deleted), undoing here would
|
||||
# create a double-membership — and since the delivery endpoint serves one VIP per agent,
|
||||
# reactivating this (never-applied → not_configured) VIP would tear down the other live
|
||||
# VIP on that node. Block it with a clear message (review round-3 FINDING 2).
|
||||
member_ids = [m["agent_id"] for m in sm]
|
||||
if member_ids:
|
||||
dup = await conn.fetchrow(
|
||||
"SELECT a.name AS agent_name, v.name AS vip_name FROM vip_members vm "
|
||||
"JOIN vip_instances v ON v.id = vm.vip_id JOIN agents a ON a.id = vm.agent_id "
|
||||
"WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE AND v.id <> $2 LIMIT 1",
|
||||
member_ids, vip_id)
|
||||
if dup:
|
||||
return (f"node '{dup['agent_name']}' now belongs to active VIP '{dup['vip_name']}' — "
|
||||
f"remove it there first, then undo")
|
||||
# Issue #73: a member that meanwhile joined another active VIP keeps it. The delivery
|
||||
# endpoint serves a node every active VIP that has an applied snapshot, so reactivating
|
||||
# this one (never applied, or applied earlier) cannot displace the other.
|
||||
# Preserve the running (applied) snapshot for members that remain, so undo of an edit
|
||||
# never momentarily flips a live node to not_configured (T-1).
|
||||
prev = {r["agent_id"]: r for r in await conn.fetch(
|
||||
"SELECT agent_id, applied_config_content, applied_config_hash, last_deploy_state, "
|
||||
"last_deploy_message, last_deploy_hash, last_deploy_at FROM vip_members WHERE vip_id=$1", vip_id)}
|
||||
# Captured before v1.12.0 → no additional addresses.
|
||||
extras = _extra_ips(sv.get("additional_ips") or [])
|
||||
try:
|
||||
async with conn.transaction():
|
||||
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
|
||||
reason = await _address_clash(conn, sv["virtual_ip"], extras, vip_id)
|
||||
if reason:
|
||||
return f"cannot restore — {reason}"
|
||||
await conn.execute("""
|
||||
UPDATE vip_instances SET name=$2, description=$3, virtual_ip=$4, prefix_length=$5,
|
||||
virtual_router_id=$6, advert_int=$7, use_unicast=$8, track_haproxy=$9,
|
||||
auth_pass_encrypted=$10, is_active=TRUE, last_config_status='PENDING',
|
||||
updated_at=CURRENT_TIMESTAMP
|
||||
additional_ips=$11::jsonb, updated_at=CURRENT_TIMESTAMP
|
||||
WHERE id=$1
|
||||
""", vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
|
||||
sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
|
||||
sv.get("auth_pass_encrypted"))
|
||||
sv.get("auth_pass_encrypted"), json.dumps(extras))
|
||||
await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
|
||||
for mm in sm:
|
||||
o = prev.get(mm["agent_id"])
|
||||
@@ -395,7 +391,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
|
||||
if cluster_id is not None:
|
||||
vips = await conn.fetch(f"""
|
||||
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
|
||||
v.is_active, v.last_config_status, v.pending_delete,
|
||||
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
|
||||
v.created_at, v.updated_at, p.name AS pool_name
|
||||
@@ -408,7 +404,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
|
||||
else:
|
||||
vips = await conn.fetch(f"""
|
||||
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
|
||||
v.is_active, v.last_config_status, v.pending_delete,
|
||||
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
|
||||
v.created_at, v.updated_at, p.name AS pool_name
|
||||
@@ -434,6 +430,7 @@ async def list_vips(cluster_id: Optional[int] = None, authorization: str = Heade
|
||||
v["last_config_status"], members, v["pending_delete"], v["is_active"])
|
||||
result.append({
|
||||
**dict(v),
|
||||
"additional_ips": _extra_ips(v["additional_ips"]),
|
||||
# Convergence-aware status for the table (issue #27 follow-up); the raw
|
||||
# last_config_status is kept above for the PENDING gate / Apply button.
|
||||
"deploy_status": deploy_status,
|
||||
@@ -505,7 +502,7 @@ async def get_vip(vip_id: int, authorization: str = Header(None)):
|
||||
try:
|
||||
v = await conn.fetchrow("""
|
||||
SELECT v.id, v.name, v.description, v.pool_id, v.virtual_ip, v.prefix_length,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy,
|
||||
v.virtual_router_id, v.advert_int, v.use_unicast, v.track_haproxy, v.additional_ips,
|
||||
v.is_active, v.last_config_status,
|
||||
(v.auth_pass_encrypted IS NOT NULL) AS auth_pass_set,
|
||||
v.created_at, v.updated_at
|
||||
@@ -520,7 +517,8 @@ async def get_vip(vip_id: int, authorization: str = Header(None)):
|
||||
FROM vip_members m LEFT JOIN agents a ON a.id = m.agent_id
|
||||
WHERE m.vip_id = $1 ORDER BY m.priority DESC
|
||||
""", vip_id)
|
||||
return {**dict(v), "members": [dict(m) for m in members]}
|
||||
return {**dict(v), "additional_ips": _extra_ips(v["additional_ips"]),
|
||||
"members": [dict(m) for m in members]}
|
||||
finally:
|
||||
await close_database_connection(conn)
|
||||
|
||||
@@ -540,24 +538,9 @@ async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_
|
||||
if row["pool_id"] != pool_id:
|
||||
raise HTTPException(status_code=400,
|
||||
detail=f"agent {m.agent_id} is not in pool {pool_id}")
|
||||
# One active VIP per agent (v1): a node deploys a single keepalived.conf, and the
|
||||
# delivery endpoint serves one VIP per agent — a second active membership would never
|
||||
# converge (stuck SYNCING) instead of erroring. Reject it up front (review MED-2).
|
||||
agent_ids = [m.agent_id for m in members]
|
||||
if agent_ids:
|
||||
dq = ("SELECT a.name AS agent_name, v.name AS vip_name "
|
||||
"FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id "
|
||||
"JOIN agents a ON a.id = vm.agent_id "
|
||||
"WHERE vm.agent_id = ANY($1) AND v.is_active = TRUE")
|
||||
dparams = [agent_ids]
|
||||
if exclude_vip_id is not None:
|
||||
dq += " AND v.id <> $2"
|
||||
dparams.append(exclude_vip_id)
|
||||
dup = await conn.fetchrow(dq + " LIMIT 1", *dparams)
|
||||
if dup:
|
||||
raise HTTPException(status_code=409,
|
||||
detail=f"node '{dup['agent_name']}' is already a member of VIP "
|
||||
f"'{dup['vip_name']}' — a node can belong to only one VIP")
|
||||
# Issue #73: a node may belong to several VIPs. The delivery endpoint merges their applied
|
||||
# snapshots into the node's one keepalived.conf; the pool-scoped VRID uniqueness
|
||||
# (uq_vip_vrid_active) keeps their VRRP instances apart.
|
||||
# VIP must not be an existing agent's primary IP (INET cast, SQL-1)
|
||||
clash = await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", vip_virtual_ip)
|
||||
if clash:
|
||||
@@ -572,6 +555,57 @@ async def _validate_members_against_pool(conn, pool_id: int, members: List, vip_
|
||||
raise HTTPException(status_code=409, detail=f"{vip_virtual_ip} is already used by another VIP")
|
||||
|
||||
|
||||
# Serialises every write that sets a VIP's addresses. The primary address has a partial unique
|
||||
# index; the additional ones live in a JSON list no index can cover, so two concurrent writes
|
||||
# could otherwise both pass the clash check below with the same address.
|
||||
_VIP_ADDRESS_LOCK = (18181873, 0)
|
||||
|
||||
|
||||
def _norm_ip(value) -> str:
|
||||
try:
|
||||
return str(ipaddress.ip_address(str(value).strip()))
|
||||
except ValueError:
|
||||
return str(value)
|
||||
|
||||
|
||||
def _extra_ips(value) -> List[dict]:
|
||||
"""vip_instances.additional_ips (JSONB; asyncpg hands it over as a string) as a list."""
|
||||
items = _jsonb_list(value)
|
||||
return [{"address": str(i["address"]), "prefix_length": int(i["prefix_length"])}
|
||||
for i in items if isinstance(i, dict) and i.get("address")]
|
||||
|
||||
|
||||
async def _address_clash(conn, primary: str, extras: List[dict],
|
||||
exclude_vip_id: Optional[int]) -> Optional[str]:
|
||||
"""Issue #73 — why these addresses cannot be used, or None. Covers what the primary-address
|
||||
checks do not: every additional address against node IPs and every active VIP's addresses,
|
||||
and the primary against the other VIPs' additional addresses. Call it inside the write's
|
||||
transaction after taking _VIP_ADDRESS_LOCK."""
|
||||
other = ("SELECT name FROM vip_instances WHERE is_active = TRUE "
|
||||
"AND ($2::int IS NULL OR id <> $2::int) AND {cond} LIMIT 1")
|
||||
in_extras = "additional_ips @> jsonb_build_array(jsonb_build_object('address', $1::text))"
|
||||
hit = await conn.fetchval(other.format(cond=in_extras), primary, exclude_vip_id)
|
||||
if hit:
|
||||
return f"{primary} is already an additional address of VIP '{hit}'"
|
||||
for e in extras:
|
||||
addr = e["address"]
|
||||
if await conn.fetchval("SELECT 1 FROM agents WHERE ip_address = $1::inet LIMIT 1", addr):
|
||||
return f"{addr} is already a node's IP"
|
||||
hit = await conn.fetchval(
|
||||
other.format(cond=f"(virtual_ip = $1::text OR {in_extras})"), addr, exclude_vip_id)
|
||||
if hit:
|
||||
return f"{addr} is already used by VIP '{hit}'"
|
||||
return None
|
||||
|
||||
|
||||
async def _lock_and_check_addresses(conn, primary: str, extras: List[dict],
|
||||
exclude_vip_id: Optional[int]) -> None:
|
||||
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
|
||||
reason = await _address_clash(conn, primary, extras, exclude_vip_id)
|
||||
if reason:
|
||||
raise HTTPException(status_code=409, detail=reason)
|
||||
|
||||
|
||||
@router.post("")
|
||||
async def create_vip(payload: VIPCreate, request: Request, authorization: str = Header(None)):
|
||||
current_user = await _require(authorization, "create")
|
||||
@@ -583,22 +617,25 @@ async def create_vip(payload: VIPCreate, request: Request, authorization: str =
|
||||
await _validate_members_against_pool(conn, payload.pool_id, payload.members, payload.virtual_ip)
|
||||
|
||||
enc = encrypt_vrrp_secret(payload.auth_pass) if payload.auth_pass else None
|
||||
extras = [e.model_dump() for e in payload.additional_ips]
|
||||
# Allocate VRID + insert, retrying once on a unique-violation race (B-1).
|
||||
last_err = None
|
||||
for _attempt in range(2):
|
||||
vrid = await _alloc_free_vrid(conn, payload.pool_id, payload.virtual_router_id)
|
||||
try:
|
||||
async with conn.transaction():
|
||||
await _lock_and_check_addresses(conn, payload.virtual_ip, extras, None)
|
||||
vip_id = await conn.fetchval("""
|
||||
INSERT INTO vip_instances
|
||||
(name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
|
||||
advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
|
||||
is_active, last_config_status, created_by)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11)
|
||||
is_active, last_config_status, created_by, additional_ips)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',$11,$12::jsonb)
|
||||
RETURNING id
|
||||
""", payload.name, payload.description, payload.pool_id, payload.virtual_ip,
|
||||
payload.prefix_length, vrid, payload.advert_int, enc,
|
||||
payload.use_unicast, payload.track_haproxy, current_user["id"])
|
||||
payload.use_unicast, payload.track_haproxy, current_user["id"],
|
||||
json.dumps(extras))
|
||||
for m in payload.members:
|
||||
await conn.execute("""
|
||||
INSERT INTO vip_members (vip_id, agent_id, network_interface, role, priority)
|
||||
@@ -606,6 +643,8 @@ async def create_vip(payload: VIPCreate, request: Request, authorization: str =
|
||||
""", vip_id, m.agent_id, m.network_interface, m.role, m.priority)
|
||||
last_err = None
|
||||
break
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as ie: # noqa: BLE001
|
||||
if "unique" in str(ie).lower() or "duplicate" in str(ie).lower():
|
||||
last_err = ie
|
||||
@@ -650,10 +689,17 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
|
||||
"SELECT 1 FROM vip_instances WHERE virtual_ip=$1 AND is_active=TRUE AND id<>$2", new_ip, vip_id)
|
||||
if dup:
|
||||
raise HTTPException(status_code=409, detail=f"{new_ip} is already used by another VIP")
|
||||
new_extras = ([e.model_dump() for e in payload.additional_ips]
|
||||
if payload.additional_ips is not None else None)
|
||||
extras = new_extras if new_extras is not None else _extra_ips(v["additional_ips"])
|
||||
if _norm_ip(new_ip) in {e["address"] for e in extras}:
|
||||
raise HTTPException(status_code=400, detail="an additional address must differ from virtual_ip")
|
||||
|
||||
enc_set = payload.auth_pass is not None and payload.auth_pass != ""
|
||||
try:
|
||||
async with conn.transaction():
|
||||
if payload.virtual_ip or new_extras is not None:
|
||||
await _lock_and_check_addresses(conn, new_ip, extras, vip_id)
|
||||
await conn.execute("""
|
||||
UPDATE vip_instances SET
|
||||
name = COALESCE($2, name),
|
||||
@@ -665,6 +711,7 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
|
||||
use_unicast = COALESCE($8, use_unicast),
|
||||
track_haproxy = COALESCE($9, track_haproxy),
|
||||
auth_pass_encrypted = CASE WHEN $10 THEN $11 ELSE auth_pass_encrypted END,
|
||||
additional_ips = COALESCE($12::jsonb, additional_ips),
|
||||
last_config_status = 'PENDING',
|
||||
-- Editing a VIP means you are KEEPING and changing it, so it cancels any
|
||||
-- staged deletion (otherwise a later Apply would delete instead of applying
|
||||
@@ -676,7 +723,8 @@ async def update_vip(vip_id: int, payload: VIPUpdate, request: Request, authoriz
|
||||
""", vip_id, payload.name, payload.description, payload.virtual_ip,
|
||||
payload.prefix_length, payload.virtual_router_id, payload.advert_int,
|
||||
payload.use_unicast, payload.track_haproxy,
|
||||
enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None)
|
||||
enc_set, encrypt_vrrp_secret(payload.auth_pass) if enc_set else None,
|
||||
json.dumps(new_extras) if new_extras is not None else None)
|
||||
if members is not None:
|
||||
# T-1: preserve the APPLIED snapshot + deploy ack for members that REMAIN,
|
||||
# so a pending member edit never momentarily flips a running node to
|
||||
@@ -826,6 +874,22 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
|
||||
master_prio = masters[0]["priority"]
|
||||
if any(m["role"] == "BACKUP" and m["priority"] >= master_prio for m in members):
|
||||
raise HTTPException(status_code=400, detail="MASTER priority must exceed every BACKUP")
|
||||
# Issue #73: an adoption made before v1.12.0 may still be waiting for this Apply. Taking
|
||||
# over a file with several vrrp_instance blocks would remove the others from the node.
|
||||
multi = await conn.fetchrow("""
|
||||
SELECT a.name AS agent_name, (d.analysis->>'instance_count')::int AS n
|
||||
FROM vip_members m
|
||||
JOIN vip_discoveries d ON d.agent_id = m.agent_id
|
||||
JOIN agents a ON a.id = m.agent_id
|
||||
WHERE m.vip_id = $1 AND m.takeover_expected_hash IS NOT NULL
|
||||
AND d.config_hash = m.takeover_expected_hash
|
||||
AND COALESCE((d.analysis->>'instance_count')::int, 0) > 1
|
||||
LIMIT 1
|
||||
""", vip_id)
|
||||
if multi:
|
||||
raise HTTPException(status_code=409, detail=(
|
||||
f"{multi['agent_name']}'s keepalived.conf declares {multi['n']} vrrp_instance blocks; "
|
||||
f"taking it over would remove the others from the node. Reject this adoption."))
|
||||
|
||||
auth_plain = decrypt_vrrp_secret(v["auth_pass_encrypted"]) if v["auth_pass_encrypted"] else None
|
||||
# If a secret is set but can't be decrypted (SECRET_KEY/VIP_ENCRYPTION_KEY rotated or
|
||||
@@ -844,7 +908,7 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
|
||||
vip_dict = {"id": v["id"], "name": v["name"], "virtual_ip": v["virtual_ip"],
|
||||
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
|
||||
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
|
||||
"track_haproxy": v["track_haproxy"]}
|
||||
"track_haproxy": v["track_haproxy"], "additional_ips": _extra_ips(v["additional_ips"])}
|
||||
|
||||
snapshot_members = []
|
||||
async with conn.transaction():
|
||||
@@ -870,7 +934,8 @@ async def apply_vip(vip_id: int, request: Request, authorization: str = Header(N
|
||||
"vip": {"name": v["name"], "description": v["description"], "virtual_ip": v["virtual_ip"],
|
||||
"prefix_length": v["prefix_length"], "virtual_router_id": v["virtual_router_id"],
|
||||
"advert_int": v["advert_int"], "use_unicast": v["use_unicast"],
|
||||
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"]},
|
||||
"track_haproxy": v["track_haproxy"], "auth_pass_encrypted": v["auth_pass_encrypted"],
|
||||
"additional_ips": vip_dict["additional_ips"]},
|
||||
"members": snapshot_members}
|
||||
await conn.execute(
|
||||
"UPDATE vip_instances SET last_config_status='APPLIED', "
|
||||
@@ -944,17 +1009,24 @@ async def reject_vip(vip_id: int, request: Request, authorization: str = Header(
|
||||
|
||||
sv = snap["vip"]
|
||||
sm = snap.get("members", [])
|
||||
# Applied before v1.12.0 → no additional addresses.
|
||||
extras = _extra_ips(sv.get("additional_ips") or [])
|
||||
try:
|
||||
async with conn.transaction():
|
||||
await conn.execute("SELECT pg_advisory_xact_lock($1, $2)", *_VIP_ADDRESS_LOCK)
|
||||
reason = await _address_clash(conn, sv["virtual_ip"], extras, vip_id)
|
||||
if reason:
|
||||
raise HTTPException(status_code=409, detail=f"Cannot restore — {reason}")
|
||||
await conn.execute("""
|
||||
UPDATE vip_instances SET
|
||||
name=$2, description=$3, virtual_ip=$4, prefix_length=$5, virtual_router_id=$6,
|
||||
advert_int=$7, use_unicast=$8, track_haproxy=$9, auth_pass_encrypted=$10,
|
||||
additional_ips=$11::jsonb,
|
||||
last_config_status='APPLIED', updated_at=CURRENT_TIMESTAMP
|
||||
WHERE id=$1
|
||||
""", vip_id, sv["name"], sv.get("description"), sv["virtual_ip"], sv["prefix_length"],
|
||||
sv["virtual_router_id"], sv["advert_int"], sv["use_unicast"], sv["track_haproxy"],
|
||||
sv.get("auth_pass_encrypted"))
|
||||
sv.get("auth_pass_encrypted"), json.dumps(extras))
|
||||
await conn.execute("DELETE FROM vip_members WHERE vip_id=$1", vip_id)
|
||||
for m in sm:
|
||||
await conn.execute("""
|
||||
@@ -1049,6 +1121,16 @@ def _discovery_row_to_api(row) -> dict:
|
||||
analysis = json.loads(analysis)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
analysis = None
|
||||
# Issue #73: an analysis stored by an older parser does not carry the multi-instance
|
||||
# blocker yet; show it here exactly as adopt_vip enforces it.
|
||||
count = int((analysis or {}).get("instance_count") or 0) if isinstance(analysis, dict) else 0
|
||||
if count > 1:
|
||||
from services.keepalived_parser import MULTI_INSTANCE_MARKER, multi_instance_blocker
|
||||
for cand in analysis.get("candidates") or []:
|
||||
blockers = cand.setdefault("blockers", [])
|
||||
if not any(MULTI_INSTANCE_MARKER in b for b in blockers):
|
||||
blockers.append(multi_instance_blocker(count))
|
||||
cand["adoptable"] = False
|
||||
return {
|
||||
"agent_id": row["agent_id"],
|
||||
"agent_name": row["agent_name"],
|
||||
@@ -1133,6 +1215,7 @@ async def _collect_instance_participants(conn, *, pool_id: int, vrid: int, virtu
|
||||
"agent_id": r["agent_id"], "agent_name": r["agent_name"],
|
||||
"ip_address": r["ip_address"], "config_hash": r["config_hash"],
|
||||
"auth_pass_encrypted": r["auth_pass_encrypted"],
|
||||
"instance_count": int((analysis or {}).get("instance_count") or 0),
|
||||
"candidate": cand,
|
||||
})
|
||||
break
|
||||
@@ -1206,7 +1289,8 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
except (TypeError, ValueError):
|
||||
raise HTTPException(status_code=400, detail="prefix_length must be an integer")
|
||||
|
||||
from services.keepalived_parser import remaining_blockers
|
||||
from services.keepalived_parser import (
|
||||
MULTI_INSTANCE_MARKER, multi_instance_blocker, remaining_blockers)
|
||||
|
||||
vrid = vip_fields.get("virtual_router_id")
|
||||
virtual_ip = vip_fields.get("virtual_ip")
|
||||
@@ -1236,6 +1320,9 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
prefix_supplied=vip_fields.get("prefix_length") is not None,
|
||||
accept_data_loss=bool(payload.get("accept_data_loss")),
|
||||
)
|
||||
# Issue #73: also for discoveries analysed before the parser reported this itself.
|
||||
if p["instance_count"] > 1 and not any(MULTI_INSTANCE_MARKER in b for b in rem):
|
||||
rem.append(multi_instance_blocker(p["instance_count"]))
|
||||
if rem:
|
||||
raise HTTPException(status_code=422, detail={
|
||||
"message": f"This keepalived config cannot be adopted as-is ({p['agent_name']})",
|
||||
@@ -1263,18 +1350,21 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
# this VRID in the pool (the clash check above would have refused first);
|
||||
# * a node whose file carries our ownership marker (is_managed) is ours already.
|
||||
adopted_ids = [p["agent_id"] for p in participants]
|
||||
# Issue #73: every address of the instance, not only the first.
|
||||
extras = _extra_ips(vip_fields.get("additional_ips") or [])
|
||||
all_addrs = [virtual_ip] + [e["address"] for e in extras]
|
||||
stranded = await conn.fetch("""
|
||||
SELECT a.name AS agent_name, a.pool_id, COALESCE(a.enabled, TRUE) AS enabled,
|
||||
d.parse_error
|
||||
FROM vip_discoveries d
|
||||
JOIN agents a ON a.id = d.agent_id
|
||||
LEFT JOIN vip_instances av ON av.id = d.adopted_vip_id
|
||||
WHERE d.raw_config_masked LIKE '%' || $1 || '%'
|
||||
WHERE d.raw_config_masked LIKE ANY($1::text[])
|
||||
AND NOT (a.id = ANY($2::int[]))
|
||||
AND COALESCE(av.is_active, FALSE) = FALSE
|
||||
AND d.is_managed = FALSE
|
||||
ORDER BY a.name
|
||||
""", virtual_ip, adopted_ids)
|
||||
""", [f"%{a}%" for a in all_addrs], adopted_ids)
|
||||
if stranded:
|
||||
def _why(r):
|
||||
if r["parse_error"]:
|
||||
@@ -1286,7 +1376,7 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
return "it does not report this vrrp_instance"
|
||||
named = "; ".join(f"{r['agent_name']} — {_why(r)}" for r in stranded)
|
||||
raise HTTPException(status_code=422, detail=(
|
||||
f"These node(s) also reference {virtual_ip} but cannot be taken over with the rest "
|
||||
f"These node(s) also reference {', '.join(all_addrs)} but cannot be taken over with the rest "
|
||||
f"of the instance, so adopting now would rewrite the others and leave them running "
|
||||
f"an unmanaged config on the same address: {named}. Resolve that first, then adopt."))
|
||||
|
||||
@@ -1296,11 +1386,17 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
# the sharpest: the design refuses to GUESS a netmask for a live VIP, and quietly copying
|
||||
# one node's netmask onto another is the same change by another name.
|
||||
def _vfield(p, key):
|
||||
return (p["candidate"].get("vip") or {}).get(key)
|
||||
value = (p["candidate"].get("vip") or {}).get(key)
|
||||
if key == "additional_ips":
|
||||
# Hashable and order-free; an analysis from before v1.12.0 has no key → none.
|
||||
return tuple(sorted((e["address"], e["prefix_length"])
|
||||
for e in _extra_ips(value or [])))
|
||||
return value
|
||||
|
||||
for key, label in (("prefix_length", "prefix length"),
|
||||
("use_unicast", "unicast/multicast mode"),
|
||||
("track_haproxy", "HAProxy tracking")):
|
||||
("track_haproxy", "HAProxy tracking"),
|
||||
("additional_ips", "additional addresses")):
|
||||
seen = {}
|
||||
for p in participants:
|
||||
seen.setdefault(_vfield(p, key), []).append(p["agent_name"])
|
||||
@@ -1370,9 +1466,9 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
f"holding the VIP. Register those nodes as agents so they report their config, "
|
||||
f"then adopt again."))
|
||||
|
||||
# One active VIP per agent — the delivery endpoint serves a single keepalived.conf per
|
||||
# node, so a second active membership never converges. create/update enforce this via
|
||||
# _validate_members_against_pool; adoption did not call it at all.
|
||||
# Adoption hands a node's whole keepalived.conf over to OpenManager. A node that is
|
||||
# already a member of a managed VIP is not taken over a second time (issue #73 allows
|
||||
# several VIPs per node for VIPs created here, not for a second adoption).
|
||||
busy = await conn.fetchrow("""
|
||||
SELECT a.name AS agent_name, v.name AS vip_name
|
||||
FROM vip_members vm JOIN vip_instances v ON v.id = vm.vip_id
|
||||
@@ -1396,19 +1492,20 @@ async def adopt_vip(payload: dict, request: Request, authorization: str = Header
|
||||
f"config must keep its VRID, so resolve the collision first"))
|
||||
|
||||
async with conn.transaction():
|
||||
await _lock_and_check_addresses(conn, virtual_ip, extras, None)
|
||||
vip_id = await conn.fetchval("""
|
||||
INSERT INTO vip_instances
|
||||
(name, description, pool_id, virtual_ip, prefix_length, virtual_router_id,
|
||||
advert_int, auth_pass_encrypted, use_unicast, track_haproxy,
|
||||
is_active, last_config_status, adopted_at, created_by)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',CURRENT_TIMESTAMP,$11)
|
||||
is_active, last_config_status, adopted_at, created_by, additional_ips)
|
||||
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,TRUE,'PENDING',CURRENT_TIMESTAMP,$11,$12::jsonb)
|
||||
RETURNING id
|
||||
""", name, payload.get("description") or f"Adopted from {disc['agent_name']}",
|
||||
disc["pool_id"], virtual_ip, int(vip_fields["prefix_length"]), int(vrid),
|
||||
int(vip_fields.get("advert_int") or 1),
|
||||
disc["auth_pass_encrypted"], # already Fernet-encrypted at ingest
|
||||
bool(vip_fields.get("use_unicast")), bool(vip_fields.get("track_haproxy")),
|
||||
current_user["id"])
|
||||
current_user["id"], json.dumps(extras))
|
||||
# Every node of the instance becomes a member with the role/priority/interface ITS
|
||||
# OWN file declares, and each carries its own one-shot takeover authorisation pinned
|
||||
# to the hash of the file we analysed on THAT node. The guard stays per-node: a file
|
||||
|
||||
@@ -183,6 +183,8 @@ def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
|
||||
lines.append(" }")
|
||||
lines.append(" virtual_ipaddress {")
|
||||
lines.append(f' {vip["virtual_ip"]}/{int(vip.get("prefix_length", 24))} dev {iface}')
|
||||
for extra in vip.get("additional_ips") or []:
|
||||
lines.append(f' {extra["address"]}/{int(extra["prefix_length"])} dev {iface}')
|
||||
lines.append(" }")
|
||||
if track:
|
||||
lines.append(" track_script {")
|
||||
@@ -190,3 +192,101 @@ def render_keepalived_conf(*, vip: dict, members: List[dict], this_agent: dict,
|
||||
lines.append(" }")
|
||||
lines.append("}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Issue #73 — several VIPs on one node
|
||||
# ---------------------------------------------------------------------------
|
||||
# A node deploys ONE keepalived.conf, and each member row stores the full single-VIP file
|
||||
# render_keepalived_conf produced at apply time. A node that belongs to several VIPs gets
|
||||
# those applied files merged into one: global_defs once, then every VIP's vrrp_script and
|
||||
# vrrp_instance. The health-check script is renamed per VIP (chk_haproxy_<id>) because each
|
||||
# VIP's failover weight is derived from its own priorities, and keepalived needs unique
|
||||
# script names. A node with a single VIP is served its applied file unchanged.
|
||||
class KeepalivedComposeError(ValueError):
|
||||
"""An applied snapshot is not in the shape render_keepalived_conf produces."""
|
||||
|
||||
|
||||
def _top_level_blocks(conf: str) -> List[tuple]:
|
||||
"""Split our own render into (name, lines) top-level blocks; comment lines are skipped."""
|
||||
blocks: List[tuple] = []
|
||||
current: Optional[List[str]] = None
|
||||
depth = 0
|
||||
for line in conf.split("\n"):
|
||||
stripped = line.strip()
|
||||
if current is None:
|
||||
if not stripped or stripped.startswith("#"):
|
||||
continue
|
||||
if not stripped.endswith("{"):
|
||||
raise KeepalivedComposeError(f"unexpected top-level line: {stripped[:60]}")
|
||||
current = [line]
|
||||
depth = 1
|
||||
continue
|
||||
current.append(line)
|
||||
if stripped.startswith("auth_pass "):
|
||||
continue # the secret may contain braces; it is a single token, never a block
|
||||
depth += stripped.count("{") - stripped.count("}")
|
||||
if depth == 0:
|
||||
blocks.append((" ".join(current[0].strip().rstrip("{").split()), current))
|
||||
current = None
|
||||
elif depth < 0:
|
||||
raise KeepalivedComposeError("unbalanced braces")
|
||||
if current is not None:
|
||||
raise KeepalivedComposeError("unterminated block")
|
||||
return blocks
|
||||
|
||||
|
||||
def compose_node_keepalived_conf(applied: List[tuple]) -> str:
|
||||
"""Merge the applied single-VIP files of one node into the file the node deploys.
|
||||
|
||||
`applied` is [(vip_id, applied_config_content), ...]. One entry is returned unchanged,
|
||||
so nodes with a single VIP keep the exact bytes (and hash) they already run. Raises
|
||||
KeepalivedComposeError when a snapshot does not have the expected shape; the caller
|
||||
must then leave the node alone rather than deploy a partial file.
|
||||
"""
|
||||
if not applied:
|
||||
raise KeepalivedComposeError("nothing to compose")
|
||||
if len(applied) == 1:
|
||||
return applied[0][1]
|
||||
|
||||
header: List[str] = [f"{OWNERSHIP_MARKER} — DO NOT EDIT"]
|
||||
global_defs: Optional[List[str]] = None
|
||||
scripts: List[str] = []
|
||||
instances: List[str] = []
|
||||
for vip_id, conf in sorted(applied, key=lambda item: int(item[0])):
|
||||
vip_id = int(vip_id)
|
||||
for line in (conf or "").split("\n"):
|
||||
if line.startswith("# VIP "):
|
||||
header.append(line)
|
||||
blocks = _top_level_blocks(conf or "")
|
||||
names = [name for name, _ in blocks]
|
||||
if names.count("global_defs") != 1 or names.count(f"vrrp_instance {_vrrp_instance_name(vip_id)}") != 1:
|
||||
raise KeepalivedComposeError(f"VIP {vip_id}: applied config has an unexpected layout")
|
||||
if any(n not in ("global_defs", "vrrp_script chk_haproxy", f"vrrp_instance {_vrrp_instance_name(vip_id)}")
|
||||
for n in names):
|
||||
raise KeepalivedComposeError(f"VIP {vip_id}: applied config has an unexpected block")
|
||||
script_name = f"chk_haproxy_{vip_id}"
|
||||
for name, body in blocks:
|
||||
if name == "global_defs":
|
||||
if global_defs is None:
|
||||
global_defs = body
|
||||
elif name == "vrrp_script chk_haproxy":
|
||||
scripts.append("\n".join([f"vrrp_script {script_name} {{"] + body[1:]))
|
||||
else:
|
||||
renamed, in_track = [], False
|
||||
for line in body:
|
||||
s = line.strip()
|
||||
if s == "track_script {":
|
||||
in_track = True
|
||||
elif in_track and s == "}":
|
||||
in_track = False
|
||||
elif in_track and s == "chk_haproxy":
|
||||
line = line.replace("chk_haproxy", script_name)
|
||||
renamed.append(line)
|
||||
instances.append("\n".join(renamed))
|
||||
|
||||
out: List[str] = header + global_defs + [""]
|
||||
for block in scripts:
|
||||
out.extend([block, ""])
|
||||
out.append("\n\n".join(instances))
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
@@ -31,6 +31,12 @@ import ipaddress
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from models.vip import MAX_ADDITIONAL_IPS
|
||||
|
||||
# Stored with every discovery (vip_discoveries.analysis_version). Bump it whenever the
|
||||
# analysis or its blockers change: rows from an older parser make the agent re-report once.
|
||||
ANALYSIS_VERSION = 1
|
||||
|
||||
# Directives `render_keepalived_conf` emits, and therefore the only ones a takeover can
|
||||
# reproduce. Anything else found in a vrrp_instance is reported in `unsupported`.
|
||||
_SUPPORTED_INSTANCE_KEYS = {
|
||||
@@ -379,6 +385,17 @@ _DEFAULT_STATE = "BACKUP"
|
||||
OUR_CHECK_SCRIPT_NAME = "chk_haproxy"
|
||||
|
||||
|
||||
MULTI_INSTANCE_MARKER = "vrrp_instance blocks"
|
||||
|
||||
|
||||
def multi_instance_blocker(count: int) -> str:
|
||||
"""Adoption replaces the node's whole keepalived.conf with ONE instance. Not resolvable:
|
||||
the text must contain neither _LOSS_MARKER nor _PREFIX_MARKER."""
|
||||
return (f"the file declares {count} {MULTI_INSTANCE_MARKER}; adoption takes over the whole "
|
||||
f"file, so the other instances would be removed from the node — adopting from a "
|
||||
f"keepalived.conf with several instances is not supported")
|
||||
|
||||
|
||||
def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Map one parsed `vrrp_instance` onto vip_instances / vip_members fields.
|
||||
|
||||
@@ -399,17 +416,54 @@ def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -
|
||||
f"line {d['line']}: `{d['directive']}` — OpenManager's renderer cannot reproduce "
|
||||
f"this, so adopting would delete it")
|
||||
|
||||
# Adoption replaces the node's whole keepalived.conf with the render of ONE instance, so
|
||||
# every other instance in the file would disappear from the node. Not resolvable: none of
|
||||
# the wording below may contain _LOSS_MARKER or _PREFIX_MARKER.
|
||||
instance_total = len(parsed.get("instances") or [])
|
||||
if instance_total > 1:
|
||||
blockers.append(multi_instance_blocker(instance_total))
|
||||
|
||||
vips = instance.get("virtual_ips") or []
|
||||
if len(vips) == 0:
|
||||
blockers.append("the instance declares no virtual_ipaddress — nothing to adopt")
|
||||
elif len(vips) > 1:
|
||||
addrs = ", ".join(v["address"] for v in vips)
|
||||
blockers.append(
|
||||
f"the instance carries {len(vips)} addresses ({addrs}); a managed VIP holds exactly "
|
||||
f"one, so adopting would drop all but the first")
|
||||
|
||||
vip_entry = vips[0] if vips else None
|
||||
|
||||
# Issue #73: the addresses after the first become the VIP's additional addresses. Each one
|
||||
# must render back exactly, so nothing about it may be guessed.
|
||||
additional_ips: List[Dict[str, Any]] = []
|
||||
extra_entries = vips[1:]
|
||||
if len(extra_entries) > MAX_ADDITIONAL_IPS:
|
||||
blockers.append(
|
||||
f"the instance carries {len(vips)} addresses; a managed VIP holds at most "
|
||||
f"{MAX_ADDITIONAL_IPS + 1}")
|
||||
seen_addrs = {vip_entry["address"]} if vip_entry else set()
|
||||
for extra in extra_entries:
|
||||
addr = extra["address"]
|
||||
if ipaddress.ip_address(addr).version != 4:
|
||||
blockers.append(f"additional address `{addr}` is IPv6; only IPv4 addresses are managed")
|
||||
continue
|
||||
if addr in seen_addrs:
|
||||
blockers.append(f"additional address `{addr}` is listed more than once")
|
||||
continue
|
||||
seen_addrs.add(addr)
|
||||
prefix = extra.get("prefix_length")
|
||||
if prefix is None:
|
||||
blockers.append(
|
||||
f"additional address `{addr}` has no /prefix in the file; add it there so the "
|
||||
f"netmask cannot change on takeover, then let the agent report again")
|
||||
continue
|
||||
if not (1 <= int(prefix) <= 32):
|
||||
blockers.append(f"additional address `{addr}/{prefix}` has an invalid prefix length")
|
||||
continue
|
||||
if extra.get("dev") and instance.get("interface") and extra["dev"] != instance["interface"]:
|
||||
blockers.append(
|
||||
f"additional address `{addr}` is bound to `dev {extra['dev']}` but the instance "
|
||||
f"uses `interface {instance['interface']}`; the render always uses the instance "
|
||||
f"interface")
|
||||
continue
|
||||
additional_ips.append({"address": addr, "prefix_length": int(prefix)})
|
||||
|
||||
if instance.get("virtual_router_id") is None:
|
||||
blockers.append("no virtual_router_id — it cannot be guessed: a wrong VRID puts the "
|
||||
"nodes in separate VRRP domains and both would claim the VIP")
|
||||
@@ -468,6 +522,7 @@ def build_adoption_candidate(parsed: Dict[str, Any], instance: Dict[str, Any]) -
|
||||
"use_unicast": bool(peers),
|
||||
"track_haproxy": OUR_CHECK_SCRIPT_NAME in tracked,
|
||||
"auth_pass": instance.get("auth_pass"),
|
||||
"additional_ips": additional_ips,
|
||||
},
|
||||
"member": {
|
||||
"network_interface": instance.get("interface"),
|
||||
@@ -526,6 +581,7 @@ def analyse_keepalived_conf(text: str) -> Dict[str, Any]:
|
||||
"""Parse + map in one call: the shape the discovery endpoint stores and the UI renders."""
|
||||
parsed = parse_keepalived_conf(text)
|
||||
return {
|
||||
"analysis_version": ANALYSIS_VERSION,
|
||||
"instance_count": len(parsed["instances"]),
|
||||
"sync_groups": parsed["sync_groups"],
|
||||
"global_defs": parsed["global_defs"],
|
||||
|
||||
@@ -18,7 +18,7 @@ os.environ.setdefault("SECRET_KEY", "test-secret-key-for-keepalived-parser-tests
|
||||
from services import keepalived_config as kc # noqa: E402
|
||||
from services.keepalived_parser import ( # noqa: E402
|
||||
KeepalivedParseError, analyse_keepalived_conf, build_adoption_candidate,
|
||||
parse_keepalived_conf,
|
||||
parse_keepalived_conf, remaining_blockers,
|
||||
)
|
||||
|
||||
|
||||
@@ -78,6 +78,7 @@ def test_parses_a_handwritten_config_into_model_fields():
|
||||
"use_unicast": True,
|
||||
"track_haproxy": True,
|
||||
"auth_pass": "s3cr3t",
|
||||
"additional_ips": [],
|
||||
}
|
||||
assert cand["member"] == {"network_interface": "eth0", "role": "MASTER", "priority": 150}
|
||||
assert cand["peers"] == ["10.0.0.12"] and cand["unicast_src_ip"] == "10.0.0.11"
|
||||
@@ -141,11 +142,46 @@ def test_directives_we_cannot_render_block_adoption():
|
||||
assert _only_candidate(text)["adoptable"] is False
|
||||
|
||||
|
||||
def test_multiple_addresses_in_one_instance_block_adoption():
|
||||
text = HANDWRITTEN.replace(" 10.0.0.100/24 dev eth0",
|
||||
" 10.0.0.100/24 dev eth0\n 10.0.0.101/24 dev eth0")
|
||||
blockers = _blockers_for(text)
|
||||
assert "2 addresses" in blockers and "10.0.0.101" in blockers
|
||||
def _with_extra(line):
|
||||
return HANDWRITTEN.replace(" 10.0.0.100/24 dev eth0",
|
||||
" 10.0.0.100/24 dev eth0\n" + line)
|
||||
|
||||
|
||||
def test_multiple_addresses_in_one_instance_are_adopted_as_additional_ips():
|
||||
"""Issue #73: the addresses after the first become the VIP's additional addresses."""
|
||||
cand = _only_candidate(_with_extra(" 10.0.0.101/25 dev eth0\n 10.0.0.102/24"))
|
||||
assert cand["adoptable"] is True, cand["blockers"]
|
||||
assert cand["vip"]["virtual_ip"] == "10.0.0.100"
|
||||
assert cand["vip"]["additional_ips"] == [
|
||||
{"address": "10.0.0.101", "prefix_length": 25},
|
||||
{"address": "10.0.0.102", "prefix_length": 24},
|
||||
]
|
||||
|
||||
|
||||
def test_an_additional_address_that_cannot_render_back_exactly_blocks_for_good():
|
||||
"""No prefix, another dev, IPv6 or a repeat cannot be reproduced. None of them may be
|
||||
waivable: their wording must carry neither the loss nor the prefix marker."""
|
||||
cases = {
|
||||
" 10.0.0.101": "no /prefix",
|
||||
" 10.0.0.101/24 dev eth1": "dev eth1",
|
||||
" fd00::1/64 dev eth0": "IPv6",
|
||||
" 10.0.0.100/24 dev eth0": "more than once",
|
||||
" 10.0.0.101/40 dev eth0": "invalid prefix",
|
||||
}
|
||||
for line, reason in cases.items():
|
||||
cand = _only_candidate(_with_extra(line))
|
||||
assert cand["adoptable"] is False, line
|
||||
assert reason in " | ".join(cand["blockers"]), (line, cand["blockers"])
|
||||
assert remaining_blockers(cand["blockers"], prefix_supplied=True,
|
||||
accept_data_loss=True), line
|
||||
|
||||
|
||||
def test_too_many_addresses_block_adoption():
|
||||
from models.vip import MAX_ADDITIONAL_IPS
|
||||
lines = "\n".join(f" 10.0.1.{i}/24 dev eth0" for i in range(1, MAX_ADDITIONAL_IPS + 2))
|
||||
cand = _only_candidate(_with_extra(lines))
|
||||
assert cand["adoptable"] is False
|
||||
assert f"at most {MAX_ADDITIONAL_IPS + 1}" in " | ".join(cand["blockers"])
|
||||
|
||||
|
||||
def test_missing_vrid_blocks_adoption_with_the_split_brain_reason():
|
||||
@@ -237,7 +273,12 @@ vrrp_instance VI_2 {
|
||||
names = [c["instance_name"] for c in analysed["candidates"]]
|
||||
assert names == ["VI_1", "VI_2"]
|
||||
assert [c["vip"]["virtual_ip"] for c in analysed["candidates"]] == ["10.0.0.100", "10.0.0.200"]
|
||||
assert all(c["adoptable"] for c in analysed["candidates"])
|
||||
# Issue #73: adoption takes over the whole file, so adopting either instance would remove
|
||||
# the other from the node. Neither may be adoptable, and no flag may waive it.
|
||||
for c in analysed["candidates"]:
|
||||
assert c["adoptable"] is False
|
||||
assert remaining_blockers(c["blockers"], prefix_supplied=True, accept_data_loss=True)
|
||||
assert any("2 vrrp_instance blocks" in b for b in c["blockers"])
|
||||
|
||||
|
||||
def test_unbalanced_braces_raise():
|
||||
|
||||
@@ -460,7 +460,8 @@ def test_discovery_cache_defers_to_the_server():
|
||||
assert agent_router.count('AS discovery_known') == 1, (
|
||||
"the keepalived-config query must report whether a discovery row exists"
|
||||
)
|
||||
assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 5, (
|
||||
# 7 = the five original paths + the v1.12.0 merged-delivery and compose_error paths.
|
||||
assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 7, (
|
||||
"every response path that knows the agent must carry the flag — the adoptable nodes are "
|
||||
"precisely the not_configured ones"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
"""Issue #73 (v1.12.0) — several VIPs on one node, several addresses on one VIP.
|
||||
|
||||
Pure tests for the merge of a node's applied single-VIP files, the extra-address rendering
|
||||
and validation, plus source-level guards for the delivery / ack / adoption / migration
|
||||
invariants that need a live database to exercise end to end.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from models.vip import MAX_ADDITIONAL_IPS, VIPCreate, VIPUpdate # noqa: E402
|
||||
from services import keepalived_config as kc # noqa: E402
|
||||
from services.keepalived_parser import ( # noqa: E402
|
||||
ANALYSIS_VERSION, analyse_keepalived_conf, parse_keepalived_conf,
|
||||
)
|
||||
|
||||
BACKEND = pathlib.Path(__file__).resolve().parents[1]
|
||||
AGENT_ROUTER = (BACKEND / "routers" / "agent.py").read_text()
|
||||
VIP_ROUTER = (BACKEND / "routers" / "vip.py").read_text()
|
||||
MIGRATIONS = (BACKEND / "database" / "migrations.py").read_text()
|
||||
|
||||
NODE = {"role": "MASTER", "priority": 150, "network_interface": "eth0",
|
||||
"agent_id": 1, "ip_address": "10.0.0.11"}
|
||||
PEER = {"role": "BACKUP", "priority": 100, "network_interface": "eth0",
|
||||
"agent_id": 2, "ip_address": "10.0.0.12"}
|
||||
|
||||
|
||||
def _vip(vid, ip, vrid, **over):
|
||||
base = {"id": vid, "name": f"vip-{vid}", "virtual_ip": ip, "prefix_length": 24,
|
||||
"virtual_router_id": vrid, "advert_int": 1, "use_unicast": True,
|
||||
"track_haproxy": True}
|
||||
base.update(over)
|
||||
return base
|
||||
|
||||
|
||||
def _applied(vip, auth="s3cr3t", this=NODE, members=(NODE, PEER)):
|
||||
peers = [m["ip_address"] for m in members if m["agent_id"] != this["agent_id"]]
|
||||
return kc.render_keepalived_conf(vip=vip, members=list(members), this_agent=this,
|
||||
peer_ips=peers, auth_pass_plain=auth)
|
||||
|
||||
|
||||
A = _vip(3, "10.0.0.100", 51)
|
||||
B = _vip(7, "10.0.0.200", 52, track_haproxy=False)
|
||||
|
||||
|
||||
class TestCompose:
|
||||
def test_single_vip_is_served_byte_identical(self):
|
||||
conf = _applied(A)
|
||||
assert kc.compose_node_keepalived_conf([(3, conf)]) == conf
|
||||
|
||||
def test_two_vips_merge_into_one_valid_file(self):
|
||||
out = kc.compose_node_keepalived_conf([(3, _applied(A)), (7, _applied(B))])
|
||||
assert out.count("global_defs {") == 1
|
||||
assert out.startswith(kc.OWNERSHIP_MARKER)
|
||||
assert "vrrp_script chk_haproxy_3 {" in out
|
||||
assert "vrrp_script chk_haproxy {" not in out
|
||||
assert "vrrp_instance VI_3 {" in out and "vrrp_instance VI_7 {" in out
|
||||
assert out.index("VI_3") < out.index("VI_7")
|
||||
assert " chk_haproxy_3\n" in out
|
||||
parsed = parse_keepalived_conf(out)
|
||||
assert [i["instance_name"] for i in parsed["instances"]] == ["VI_3", "VI_7"]
|
||||
assert [i["track_scripts"] for i in parsed["instances"]] == [["chk_haproxy_3"], []]
|
||||
|
||||
def test_order_follows_vip_id_not_input_order(self):
|
||||
a, b = (3, _applied(A)), (7, _applied(B))
|
||||
assert kc.compose_node_keepalived_conf([b, a]) == kc.compose_node_keepalived_conf([a, b])
|
||||
|
||||
def test_each_instance_keeps_its_own_failover_weight(self):
|
||||
a2 = _vip(9, "10.0.0.150", 53)
|
||||
low_peer = {**PEER, "priority": 50}
|
||||
out = kc.compose_node_keepalived_conf([
|
||||
(3, _applied(A)), (9, _applied(a2, members=(NODE, low_peer)))])
|
||||
assert "weight -51" in out and "weight -101" in out
|
||||
|
||||
def test_braces_in_the_secret_do_not_break_the_merge(self):
|
||||
out = kc.compose_node_keepalived_conf([(3, _applied(A, auth="a}b{")),
|
||||
(7, _applied(B, auth="}x"))])
|
||||
assert "auth_pass a}b{" in out and "auth_pass }x" in out
|
||||
assert out.count("vrrp_instance") == 2
|
||||
|
||||
@pytest.mark.parametrize("bad", [
|
||||
"",
|
||||
"global_defs {\n}\n",
|
||||
"vrrp_instance VI_3 {\n state MASTER\n",
|
||||
"something else\n",
|
||||
])
|
||||
def test_unexpected_snapshot_fails_closed(self, bad):
|
||||
with pytest.raises(kc.KeepalivedComposeError):
|
||||
kc.compose_node_keepalived_conf([(3, _applied(A)), (7, bad)])
|
||||
|
||||
def test_foreign_block_fails_closed(self):
|
||||
tampered = _applied(B) + "virtual_server 10.0.0.1 80 {\n}\n"
|
||||
with pytest.raises(kc.KeepalivedComposeError):
|
||||
kc.compose_node_keepalived_conf([(3, _applied(A)), (7, tampered)])
|
||||
|
||||
def test_snapshot_of_another_vip_id_fails_closed(self):
|
||||
with pytest.raises(kc.KeepalivedComposeError):
|
||||
kc.compose_node_keepalived_conf([(3, _applied(A)), (8, _applied(B))])
|
||||
|
||||
def test_nothing_to_compose_raises(self):
|
||||
with pytest.raises(kc.KeepalivedComposeError):
|
||||
kc.compose_node_keepalived_conf([])
|
||||
|
||||
|
||||
class TestAdditionalAddresses:
|
||||
def test_empty_list_renders_byte_identical(self):
|
||||
assert _applied({**A, "additional_ips": []}) == _applied(A)
|
||||
|
||||
def test_extras_follow_the_primary_on_the_same_interface(self):
|
||||
conf = _applied({**A, "additional_ips": [
|
||||
{"address": "10.0.0.101", "prefix_length": 24},
|
||||
{"address": "192.168.5.9", "prefix_length": 32}]})
|
||||
lines = [l.strip() for l in conf.splitlines()]
|
||||
i = lines.index("10.0.0.100/24 dev eth0")
|
||||
assert lines[i + 1:i + 3] == ["10.0.0.101/24 dev eth0", "192.168.5.9/32 dev eth0"]
|
||||
|
||||
def test_rendered_extras_are_adopted_back_unchanged(self):
|
||||
extras = [{"address": "10.0.0.101", "prefix_length": 25}]
|
||||
conf = _applied({**A, "additional_ips": extras})
|
||||
cand = analyse_keepalived_conf(conf)["candidates"][0]
|
||||
assert cand["adoptable"] is True, cand["blockers"]
|
||||
assert cand["vip"]["additional_ips"] == extras
|
||||
|
||||
|
||||
def _members():
|
||||
return [{"agent_id": 1, "network_interface": "eth0", "role": "MASTER", "priority": 150}]
|
||||
|
||||
|
||||
class TestModels:
|
||||
def test_default_is_empty(self):
|
||||
v = VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members())
|
||||
assert v.additional_ips == []
|
||||
|
||||
def test_address_is_normalised_and_prefix_defaults(self):
|
||||
v = VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members(),
|
||||
additional_ips=[{"address": " 10.0.0.101 "}])
|
||||
assert v.additional_ips[0].model_dump() == {"address": "10.0.0.101", "prefix_length": 24}
|
||||
|
||||
@pytest.mark.parametrize("extras", [
|
||||
[{"address": "10.0.0.100"}],
|
||||
[{"address": "10.0.0.101"}, {"address": "10.0.0.101"}],
|
||||
[{"address": "fd00::1"}],
|
||||
[{"address": "nope"}],
|
||||
[{"address": "10.0.0.101", "prefix_length": 33}],
|
||||
[{"address": f"10.0.1.{i}"} for i in range(1, MAX_ADDITIONAL_IPS + 2)],
|
||||
])
|
||||
def test_invalid_extras_are_rejected(self, extras):
|
||||
with pytest.raises(ValueError):
|
||||
VIPCreate(name="v", pool_id=1, virtual_ip="10.0.0.100", members=_members(),
|
||||
additional_ips=extras)
|
||||
|
||||
def test_update_omitted_keeps_and_empty_clears(self):
|
||||
assert VIPUpdate().additional_ips is None
|
||||
assert VIPUpdate(additional_ips=[]).additional_ips == []
|
||||
|
||||
|
||||
class TestDeliveryAndAckGuards:
|
||||
def test_failures_answer_with_statuses_the_agent_ignores(self):
|
||||
assert '"status": "compose_error"' in AGENT_ROUTER
|
||||
generic = AGENT_ROUTER.split("keepalived-config delivery failed", 1)[1][:600]
|
||||
assert '"status": "unavailable"' in generic
|
||||
assert "not_configured" not in generic.split("return", 1)[1].split("}", 1)[0]
|
||||
|
||||
def test_merged_delivery_hash_is_md5_of_the_composed_text(self):
|
||||
assert 'hashlib.md5(composed.encode("utf-8")).hexdigest()' in AGENT_ROUTER
|
||||
out = kc.compose_node_keepalived_conf([(3, _applied(A)), (7, _applied(B))])
|
||||
assert len(hashlib.md5(out.encode("utf-8")).hexdigest()) == 32
|
||||
|
||||
def test_only_applied_snapshots_are_merged(self):
|
||||
q = AGENT_ROUTER.split("async def _node_applied_vips", 1)[1].split("async def ", 1)[0]
|
||||
assert "v.is_active = TRUE" in q
|
||||
assert "m.applied_config_content IS NOT NULL" in q
|
||||
|
||||
def test_ack_failure_to_merge_is_not_a_500(self):
|
||||
ack = AGENT_ROUTER.split("acks the merged file once", 1)[1][:1200]
|
||||
assert "except Exception" in ack and "composed_hash = None" in ack
|
||||
|
||||
def test_discovery_known_asks_old_analyses_to_be_reported_again(self):
|
||||
assert "vd.analysis_version >= $2" in AGENT_ROUTER
|
||||
assert ANALYSIS_VERSION >= 1
|
||||
assert analyse_keepalived_conf(_applied(A))["analysis_version"] == ANALYSIS_VERSION
|
||||
|
||||
|
||||
class TestAdoptionGuards:
|
||||
def test_apply_refuses_an_inflight_adoption_of_a_multi_instance_file(self):
|
||||
apply = VIP_ROUTER.split("async def apply_vip", 1)[1].split("\nasync def ", 1)[0]
|
||||
assert "d.config_hash = m.takeover_expected_hash" in apply
|
||||
assert "(d.analysis->>'instance_count')::int, 0) > 1" in apply
|
||||
assert "status_code=409" in apply
|
||||
|
||||
def test_old_analyses_get_the_blocker_in_the_list_and_on_adopt(self):
|
||||
assert VIP_ROUTER.count("multi_instance_blocker(") >= 2
|
||||
row_api = VIP_ROUTER.split("def _discovery_row_to_api", 1)[1].split("\ndef ", 1)[0]
|
||||
assert 'cand["adoptable"] = False' in row_api
|
||||
|
||||
def test_address_writes_take_the_address_lock(self):
|
||||
for fn in ("create_vip", "update_vip", "adopt_vip"):
|
||||
body = VIP_ROUTER.split(f"async def {fn}", 1)[1].split("\nasync def ", 1)[0]
|
||||
assert "_lock_and_check_addresses(" in body, fn
|
||||
|
||||
|
||||
class TestDeltaMigration:
|
||||
def test_versions(self):
|
||||
assert "SCHEMA_VERSION = 13" in MIGRATIONS
|
||||
assert "_DELTA_BASE_VERSION = 12" in MIGRATIONS
|
||||
assert "return [(13, ensure_vip_multi_address_columns)]" in MIGRATIONS
|
||||
|
||||
def test_full_sequence_also_creates_the_columns(self):
|
||||
inner = MIGRATIONS.split("async def _run_all_migrations_inner", 1)[1].split("\nasync def ", 1)[0]
|
||||
assert inner.index("await ensure_vip_tables()") < inner.index(
|
||||
"await ensure_vip_multi_address_columns()")
|
||||
|
||||
def test_delta_path_does_not_reseed_roles(self):
|
||||
run = MIGRATIONS.split("async def run_all_migrations", 1)[1].split("\nasync def ", 1)[0]
|
||||
delta = run.split("applied_version >= _DELTA_BASE_VERSION", 1)[1].split("else:", 1)[0]
|
||||
assert "_delta_steps()" in delta
|
||||
assert "_run_all_migrations_inner" not in delta
|
||||
assert "update_system_roles" not in delta
|
||||
|
||||
def test_columns_are_additive_with_safe_defaults(self):
|
||||
body = MIGRATIONS.split("async def ensure_vip_multi_address_columns", 1)[1].split(
|
||||
"\nasync def ", 1)[0].replace('"\n "', "")
|
||||
assert "ADD COLUMN IF NOT EXISTS additional_ips JSONB NOT NULL DEFAULT '[]'::jsonb" in body
|
||||
assert "ADD COLUMN IF NOT EXISTS analysis_version SMALLINT NOT NULL DEFAULT 0" in body
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": "1.11.3",
|
||||
"releaseName": "Config Import keeps stick-table and stick rules instead of dropping them",
|
||||
"version": "1.12.0",
|
||||
"releaseName": "Several keepalived VIPs on one node, and several addresses on one VIP",
|
||||
"releaseDate": "2026-10-16"
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "haproxy-openmanager-frontend",
|
||||
"version": "1.11.3",
|
||||
"version": "1.12.0",
|
||||
"description": "HAProxy Load Balancer Management UI",
|
||||
"license": "AGPL-3.0-or-later",
|
||||
"dependencies": {
|
||||
|
||||
@@ -6,7 +6,7 @@ import {
|
||||
import {
|
||||
PlusOutlined, EditOutlined, DeleteOutlined, ReloadOutlined, WarningOutlined,
|
||||
PlayCircleOutlined, SyncOutlined, CrownOutlined, ClockCircleOutlined, FileSearchOutlined,
|
||||
InfoCircleOutlined
|
||||
InfoCircleOutlined, MinusCircleOutlined
|
||||
} from '@ant-design/icons';
|
||||
import { useCluster } from '../contexts/ClusterContext';
|
||||
import { extractApiError } from '../utils/apiError';
|
||||
@@ -76,9 +76,13 @@ const mergeBlockers = (lists) => {
|
||||
return Array.from(seen.values());
|
||||
};
|
||||
|
||||
// Mirrors backend/models/vip.py MAX_ADDITIONAL_IPS.
|
||||
const MAX_ADDITIONAL_IPS = 19;
|
||||
const IPV4_RE = /^(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}$/;
|
||||
|
||||
// This component uses raw fetch(), but extractApiError expects an axios-shaped error
|
||||
// (err.response.data). Read the fetch Response body and reuse the envelope-aware extractor
|
||||
// so backend messages — e.g. the 409 "node already in VIP X" — actually reach the user.
|
||||
// so backend messages — e.g. the 409 "address already used by VIP X" — actually reach the user.
|
||||
const fetchApiError = async (res, fallback) => {
|
||||
try { const data = await res.json(); return extractApiError({ response: { data } }, fallback); }
|
||||
catch { return fallback; }
|
||||
@@ -351,7 +355,8 @@ const VIPManagement = () => {
|
||||
setSelectedPoolId(null);
|
||||
setMemberRows([]);
|
||||
form.resetFields();
|
||||
form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true });
|
||||
form.setFieldsValue({ prefix_length: 24, advert_int: 1, use_unicast: true, track_haproxy: true,
|
||||
additional_ips: [] });
|
||||
setModalVisible(true);
|
||||
};
|
||||
|
||||
@@ -365,6 +370,7 @@ const VIPManagement = () => {
|
||||
virtual_ip: vip.virtual_ip, prefix_length: vip.prefix_length,
|
||||
virtual_router_id: vip.virtual_router_id, advert_int: vip.advert_int,
|
||||
use_unicast: vip.use_unicast, track_haproxy: vip.track_haproxy,
|
||||
additional_ips: (vip.additional_ips || []).map((a) => ({ ...a })),
|
||||
});
|
||||
setModalVisible(true);
|
||||
};
|
||||
@@ -478,7 +484,19 @@ const VIPManagement = () => {
|
||||
|
||||
const columns = [
|
||||
{ title: 'Name', dataIndex: 'name', key: 'name' },
|
||||
{ title: 'Virtual IP', key: 'vip', render: (_, v) => <Text code>{v.virtual_ip}/{v.prefix_length}</Text> },
|
||||
{ title: 'Virtual IP', key: 'vip', render: (_, v) => {
|
||||
const extra = v.additional_ips || [];
|
||||
return (
|
||||
<Space size={4}>
|
||||
<Text code>{v.virtual_ip}/{v.prefix_length}</Text>
|
||||
{extra.length > 0 && (
|
||||
<Tooltip title={extra.map((a) => `${a.address}/${a.prefix_length}`).join(', ')}>
|
||||
<Tag>+{extra.length}</Tag>
|
||||
</Tooltip>
|
||||
)}
|
||||
</Space>
|
||||
);
|
||||
} },
|
||||
{ title: 'Pool', dataIndex: 'pool_name', key: 'pool' },
|
||||
{ title: 'VRID', dataIndex: 'virtual_router_id', key: 'vrid' },
|
||||
{ title: 'Members / Live state', key: 'members', render: renderMembers },
|
||||
@@ -843,6 +861,37 @@ const VIPManagement = () => {
|
||||
</Form.Item>
|
||||
</Space>
|
||||
|
||||
<Form.List name="additional_ips">
|
||||
{(fields, { add, remove }) => (
|
||||
<div style={{ marginBottom: 16 }}>
|
||||
<Text strong>Additional addresses</Text>
|
||||
<div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
|
||||
Optional. Extra IPv4 addresses carried by the same VRRP instance — they move between
|
||||
nodes together with the virtual IP, on the same interface (up to {MAX_ADDITIONAL_IPS}).
|
||||
</div>
|
||||
{fields.map(({ key, name, ...rest }) => (
|
||||
<Space key={key} align="baseline">
|
||||
<Form.Item {...rest} name={[name, 'address']}
|
||||
rules={[{ required: true, message: 'Address required' },
|
||||
{ pattern: IPV4_RE, message: 'Enter an IPv4 address' }]}>
|
||||
<Input placeholder="10.0.0.101" style={{ width: 180 }} />
|
||||
</Form.Item>
|
||||
<Form.Item {...rest} name={[name, 'prefix_length']}
|
||||
rules={[{ required: true, message: 'Prefix required' }]}>
|
||||
<InputNumber min={1} max={32} addonBefore="/" />
|
||||
</Form.Item>
|
||||
<MinusCircleOutlined onClick={() => remove(name)} />
|
||||
</Space>
|
||||
))}
|
||||
<Button type="dashed" size="small" icon={<PlusOutlined />}
|
||||
disabled={fields.length >= MAX_ADDITIONAL_IPS}
|
||||
onClick={() => add({ prefix_length: form.getFieldValue('prefix_length') || 24 })}>
|
||||
Add address
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
</Form.List>
|
||||
|
||||
<Text strong>Participating nodes</Text>
|
||||
<div style={{ color: '#888', fontSize: 12, marginBottom: 8 }}>
|
||||
These are the HAProxy servers (agents) already enrolled in this pool — toggle which join the VIP.
|
||||
|
||||
Reference in New Issue
Block a user