Compare commits

...

14 Commits

Author SHA1 Message Date
briquet f9370e85bc 🔧(helm) create livekit rooms only from the backend in dev
Run the LiveKit server of both Helm development environments, dev-keycloak
and dev-dinum, with room auto-creation off
2026-11-16 22:24:07 +01:00
briquet 24ae0edded 🔧(docker) create livekit rooms only from the backend
Nothing in Meet relies on auto-creation any more
2026-11-16 22:23:20 +01:00
briquet 5643e7ce61 ✨(frontend) connect to the livekit url returned by the backend
Handle multi LiveKit clusters in the frontend
2026-11-16 22:22:43 +01:00
briquet fbed98b5c4 ✨(backend) relocate livekit rooms by reconnecting participants
Nothing could move a live room off its LiveKit cluster, to drain a cluster
for maintenance or to rebalance rooms. PlacementService.relocate now moves
a room's placement to another cluster, then closes its LiveKit room on the
cluster it left, which makes participants reconnect where the placement
now points.
2026-11-16 22:19:33 +01:00
briquet 5b526d48d4 ✨(backend) record rooms on the livekit cluster hosting them
Recordings still went to the default LiveKit cluster: the egress was
started, stopped and looked up there, and the metadata collector agent
dispatched and deleted there. Once rooms are placed on several clusters,
a room placed elsewhere would be recorded nowhere.

A recording now keeps, by name in livekit_cluster, the LiveKit cluster
recording its room: the one the room is placed on when the recording is
created, which PlacementService.cluster_for tells inside the transaction
creating it. Everything done about the recording afterwards goes to that
cluster, wherever the room was placed since.
2026-11-16 22:19:30 +01:00
briquet 36f543ccae ✨(backend) place livekit rooms when handing out credentials
Django now places a LiveKit room each time it hands out credentials for
it, and creates the LiveKit room itself on the cluster it is placed on:
joins, accepted lobby entries and roomkit joins go through
PlacementService.place, and credentials are minted for the cluster it
returns. When the LiveKit room cannot be created, the request answers 503.
The connection test creates its room on the cluster the placement strategy
picks, without placement, as no one else ever joins it.

Unregistered rooms are not placed, since Django does not follow their
lifecycle: LIVEKIT_WEBHOOK_EVENTS_FILTER_REGEX usually filters their
webhooks out. Their LiveKit room is created on the default cluster, where
they all live, so they are neither balanced across clusters nor relocated,
and cannot be joined while the default cluster is down.
2026-11-16 22:19:27 +01:00
briquet 52195305ba ✨(backend) add livekit placement strategies
Spreading rooms over several LiveKit clusters needs a rule picking the
cluster of each new placement.

We introduce some basic load balancing strategies:

- WeightedRandomStrategy, the default, picks each cluster with a
probability proportional to its weight.

RoundRobinStrategy takes the clusters in turn, whatever their weights.

- LeastPlacementsStrategy picks the cluster holding the fewest placements
for its weight or the one registered first on a tie.
2026-11-16 22:19:22 +01:00
briquet 2f18d247d8 ✨(backend) follow the livekit room lifecycle on placements
A placement must end with the LiveKit room it records, or its room would
stay bound to a cluster once the meeting is over. room_finished now deletes
it, but only when the cluster that sent the webhook and the SID of the
finished LiveKit room are the placement's. Otherwise the placement points
at a LiveKit room created since, which participants admitted from the
lobby may be joining: the room is not cleaned up, so they are not queued
again.

A deleted room revived by a stale token is closed on the cluster that sent
room_started. Webhook handlers are now keyed by LiveKitWebhookEventType.
2026-11-16 22:19:07 +01:00
briquet e107d6ad9c ♻️(backend) manage livekit rooms on their placed cluster
`PlacementService.cluster_for` now tells on which LiveKit cluster a
LiveKit room is managed. It uses the default cluster for rooms without
placement.
2026-11-16 21:59:39 +01:00
briquet 4e65b99aab ✨(backend) add placements of livekit rooms
To spread rooms over several LiveKit clusters, Django chooses the cluster
each LiveKit room runs on, and has to remember that choice so that everyone
joining the room later is sent to the same cluster. A placement records for
every room the assigned cluster.
2026-11-16 21:59:25 +01:00
briquet 9bae778a45 ✨(backend) identify livekit clusters by their api key
Django checked LiveKit webhooks and X-LiveKit-Token headers with the API key
and secret of LIVEKIT_CONFIGURATION alone. A webhook or participant token
signed by any other LiveKit cluster was refused, and nothing told which
cluster had sent one. We use the issuer to distinguish between the
cluster and verify the token.
2026-11-16 20:38:22 +01:00
briquet 926e8e0935 ♻️(backend) build livekit clients and tokens for a given cluster
LIVEKIT_CONFIGURATION is now read only by the registry, which falls back to
it, and by the checks of LiveKit tokens and webhooks.
2026-11-16 20:38:18 +01:00
briquet 7e5884f979 🔒️(backend) refuse telephony with several livekit clusters
Phone and Roomkit callers reach the SIP server of one LiveKit cluster before
their PIN tells which room they join, so Django cannot route them to the
cluster the room is placed on: once rooms are spread over several clusters,
a caller could end up apart from the other participants. Telephony and
roomkit therefore need a single cluster at the moment.
2026-11-16 20:38:16 +01:00
briquet 65b84cfab3 ✨(backend) add a livekit cluster registry
Placing rooms on several independent LiveKit clusters requires
Django to know them all, and for each of them to register their
API key and secrets.

A new LIVEKIT_CLUSTERS settings is introduced to keep a registry that leverage
the existing LIVEKIT_CONFIGURATION settings.
2026-11-16 20:38:06 +01:00
69 changed files with 6257 additions and 199 deletions
@@ -9,6 +9,9 @@ webhook:
urls:
- http://app-dev:8000/api/v1.0/rooms/webhooks-livekit/
room:
auto_create: false
turn:
enabled: true
domain: turn.127.0.0.1.nip.io
+1
View File
@@ -444,6 +444,7 @@ class RecordingAdmin(admin.ModelAdmin):
"status",
"updated_at",
"worker_id",
"livekit_cluster",
)
actions = [resend_notification, mark_as_failed_to_stop]
+10 -2
View File
@@ -8,11 +8,16 @@ from rest_framework import views as drf_views
from rest_framework.decorators import api_view
from rest_framework.response import Response
from core.livekit.clusters import get_default_livekit_cluster
from core.livekit.exceptions import LiveKitRoomUnavailableError
from core.utils import build_telephony_config
from .exceptions import LiveKitRoomUnavailable
def exception_handler(exc, context):
"""Handle Django ValidationError as an accepted exception.
"""Handle Django ValidationError as an accepted exception, and answer a
LiveKit room that cannot be joined for now with a 503.
For the parameters, see ``exception_handler``
This code comes from twidi's gist:
@@ -30,6 +35,9 @@ def exception_handler(exc, context):
exc = drf_exceptions.ValidationError(detail=detail)
elif isinstance(exc, LiveKitRoomUnavailableError):
exc = LiveKitRoomUnavailable()
return drf_views.exception_handler(exc, context)
@@ -81,7 +89,7 @@ def get_frontend_configuration(request):
"subtitle": {"enabled": settings.ROOM_SUBTITLE_ENABLED},
"diagnostics": {"connection_test_enabled": settings.CONNECTION_TEST_ENABLED},
"livekit": {
"url": settings.LIVEKIT_CONFIGURATION["url"],
"url": get_default_livekit_cluster().url,
"force_wss_protocol": settings.LIVEKIT_FORCE_WSS_PROTOCOL,
"enable_firefox_proxy_workaround": settings.LIVEKIT_ENABLE_FIREFOX_PROXY_WORKAROUND,
"default_sources": settings.LIVEKIT_DEFAULT_SOURCES,
+9
View File
@@ -15,6 +15,15 @@ class RoomSoftDeleted(exceptions.APIException):
default_code = "room_deleted"
class LiveKitRoomUnavailable(exceptions.APIException):
"""Raised when the LiveKit room to hand out credentials for cannot be
joined for now: it cannot be placed, or created on its LiveKit cluster."""
status_code = status.HTTP_503_SERVICE_UNAVAILABLE
default_detail = _("This room cannot be joined for now, please try again.")
default_code = "livekit_room_unavailable"
def ensure_room_not_deleted(resource):
"""Raise a 410 Gone if the resource is a soft-deleted room.
+46 -10
View File
@@ -46,6 +46,8 @@ from core.api.exceptions import ensure_room_not_deleted
from core.api.filters import ListFileFilter
from core.api.locks import room_locked_for_update
from core.enums import MEDIA_STORAGE_URL_PATTERN
from core.livekit.clusters import get_default_livekit_cluster
from core.livekit.placements import PlacementService
from core.recording.enums import FileExtension
from core.recording.event.authentication import RecordingProcessWebhookAuthentication
from core.recording.services.metadata_collector import (
@@ -88,7 +90,6 @@ from core.services.subtitle import SubtitleException, SubtitleService
from core.services.transit_code import TransitCodeService
from core.tasks.connection_test import delete_connection_test_room
from core.tasks.file import process_file_deletion
from core.utils import generate_token
from ..authentication.livekit import LiveKitTokenAuthentication
from ..models import RoomAccessLevel
@@ -310,12 +311,21 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
return obj
def _get_object_or_unregistered_slug(self):
"""Return the requested room, or the slug of an unregistered room."""
"""Return the requested room, or the slug of an unregistered room.
A slug no room could have, empty, a UUID or longer than a room's slug
may be, is no unregistered room either.
"""
try:
return self.get_object(), None
except Http404:
slug = slugify(self.kwargs["pk"])
if not settings.ALLOW_UNREGISTERED_ROOMS or not slug or utils.is_uuid(slug):
if (
not settings.ALLOW_UNREGISTERED_ROOMS
or not slug
or utils.is_uuid(slug)
or len(slug) > models.Room._meta.get_field("slug").max_length # noqa: SLF001
):
raise
return None, slug
@@ -486,6 +496,7 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
room=room,
mode=mode,
options=options_data,
livekit_cluster=PlacementService.cluster_for(str(room.id)).name,
)
recording.grant_initial_accesses(starter=self.request.user)
@@ -503,7 +514,9 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
status=drf_status.HTTP_409_CONFLICT,
)
worker_service = get_worker_service(mode=recording.mode)
worker_service = get_worker_service(
mode=recording.mode, cluster=recording.get_livekit_cluster()
)
worker_manager = WorkerServiceMediator(worker_service=worker_service)
try:
@@ -557,7 +570,9 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
self.audit_target = recording
worker_service = get_worker_service(mode=recording.mode)
worker_service = get_worker_service(
mode=recording.mode, cluster=recording.get_livekit_cluster()
)
worker_manager = WorkerServiceMediator(worker_service=worker_service)
try:
@@ -587,7 +602,14 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
going through the lobby. Otherwise, it has no livekit object and the
client should request entry through the lobby.
Unregistered rooms, when allowed, are always joined directly.
Unregistered rooms, when allowed, are always joined directly. They are
never placed, since Django does not follow their lifecycle: they live on
the default LiveKit cluster.
The credentials are for the LiveKit cluster the LiveKit room is placed
on, or the default one for an unregistered room, where it is created
first. When it cannot be, the response is a 503, and asking again later
may succeed.
"""
serializer = serializers.JoinRoomSerializer(data=request.data)
@@ -596,11 +618,16 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
room, slug = self._get_object_or_unregistered_slug()
if room is None:
cluster = get_default_livekit_cluster()
PlacementService.create_livekit_room(slug, cluster)
return drf_response.Response(
{
**self._get_unregistered_room_data(slug),
"livekit": utils.generate_livekit_config(
room_id=slug, user=request.user, username=username
room_id=slug,
user=request.user,
username=username,
cluster=cluster,
),
}
)
@@ -611,12 +638,14 @@ class RoomViewSet( # pylint: disable=too-many-public-methods
).data
if LobbyService.can_bypass_lobby(room=room, user=request.user, role=role):
cluster = PlacementService.place(str(room.id), room)
data["livekit"] = utils.generate_livekit_config(
room_id=str(room.id),
user=request.user,
username=username,
configuration=data["configuration"],
role=role,
cluster=cluster,
)
return drf_response.Response(data)
@@ -1727,9 +1756,15 @@ class DiagnosticsViewSet(viewsets.ViewSet):
Going through the room API is not an option here: it is tied to
registered meetings, lobby rules and longer-lived tokens. Each call
gets its own room so two people testing at the same time never meet.
The room is created on the LiveKit cluster the placement strategy
picks, but has no placement: no one else ever joins it. When it cannot
be created, the response is a 503.
"""
room = f"{settings.CONNECTION_TEST_ROOM_PREFIX}-{uuid4()}"
expires_in = settings.CONNECTION_TEST_TOKEN_TTL_SECONDS
cluster = PlacementService.choose_cluster(room)
PlacementService.create_livekit_room(room, cluster)
# LiveKit refreshes tokens for connected clients, so JWT TTL alone does not
# eject someone who stays connected. Schedule a hard DeleteRoom when Celery
@@ -1740,20 +1775,21 @@ class DiagnosticsViewSet(viewsets.ViewSet):
+ settings.CONNECTION_TEST_ROOM_EXTRA_AGE_SECONDS
)
delete_connection_test_room.apply_async(
args=[room],
args=[room, cluster.name],
countdown=max_age,
)
return drf_response.Response(
{
"livekit": {
"url": settings.LIVEKIT_CONFIGURATION["url"],
"url": cluster.url,
"room": room,
"token": generate_token(
"token": utils.generate_token(
room=room,
user=request.user,
username="Connection Test",
ttl=timedelta(seconds=expires_in),
cluster=cluster,
),
"expires_in": expires_in,
},
+13
View File
@@ -45,6 +45,10 @@ RECORDING_TRANSCRIPT_REPORT = audit.Action(
RECORDING_SUMMARY_REPORT = audit.Action(
"recording.summary.report", types=(EventType.END,)
)
LIVEKIT_PLACEMENT_RELOCATE = audit.Action(
"livekit.placement.relocate", types=(EventType.CHANGE,)
)
LIVEKIT_ROOM_CLOSE = audit.Action("livekit.room.close", types=(EventType.END,))
# Targets that are not models
@@ -52,3 +56,12 @@ RECORDING_SUMMARY_REPORT = audit.Action(
def transit_code_target(code: str) -> dict[str, str]:
"""Describe a transit code as a target, by its digest: never the code itself."""
return {"id": TransitCodeService.digest(code), "sub_type": "transit_code"}
def livekit_room_target(room_name: str) -> dict[str, str]:
"""Describe a LiveKit room as a target, by its name.
That is the id of the room it is named after, or the slug of an
unregistered room.
"""
return {"id": room_name, "sub_type": "livekit_room"}
+8 -8
View File
@@ -1,19 +1,23 @@
"""Authentication using LiveKit token for the Meet core app."""
from django.conf import settings
from django.contrib.auth import get_user_model
from django.contrib.auth.models import AnonymousUser
from livekit.api import TokenVerifier
from rest_framework import authentication, exceptions
from core.livekit.clusters import LiveKitClusterTokenVerifier
UserModel = get_user_model()
LIVEKIT_AUTH_SCHEME = "X-LiveKit-Token"
class LiveKitTokenAuthentication(authentication.BaseAuthentication):
"""Authenticate using LiveKit token and load the associated Django user."""
"""Authenticate using LiveKit token and load the associated Django user.
The token may be signed by any registered LiveKit cluster. The request's
auth is the token's claims, which carry that cluster as their `cluster`.
"""
def authenticate(self, request):
auth_header = request.headers.get("Authorization")
@@ -35,11 +39,7 @@ class LiveKitTokenAuthentication(authentication.BaseAuthentication):
token = parts[1]
try:
verifier = TokenVerifier(
api_key=settings.LIVEKIT_CONFIGURATION["api_key"],
api_secret=settings.LIVEKIT_CONFIGURATION["api_secret"],
)
claims = verifier.verify(token)
claims = LiveKitClusterTokenVerifier().verify(token)
user_id = claims.identity
if not user_id:
+1
View File
@@ -0,0 +1 @@
"""The LiveKit clusters on which Django places rooms."""
+150
View File
@@ -0,0 +1,150 @@
"""Registry of the LiveKit clusters on which Django places rooms.
The first registered cluster is the default one. Each cluster has an API key of
its own, so the issuer of a token or webhook tells which cluster signed it.
"""
from dataclasses import dataclass, field
from functools import lru_cache
from django.conf import settings
import jwt
from livekit.api import TokenVerifier
from livekit.api.access_token import Claims
from .exceptions import UnknownLiveKitClusterError
from .specs import parse_livekit_clusters
FALLBACK_LIVEKIT_CLUSTER_NAME = "default"
@dataclass(frozen=True, kw_only=True)
class LiveKitClusterConfig:
"""How browsers and Django reach a LiveKit cluster.
Browsers connect to `url` and Django calls `api_url`. The API secret is
left out of the representation, which may end up in logs.
"""
name: str
url: str
api_url: str
api_key: str
api_secret: str = field(repr=False)
weight: int = 1
@lru_cache(maxsize=8)
def _clusters_from_json(raw: str) -> tuple[LiveKitClusterConfig, ...]:
"""Build the LiveKit clusters listed in a value of LIVEKIT_CLUSTERS."""
return tuple(
LiveKitClusterConfig(
name=spec.name,
url=spec.url,
api_url=spec.api_url or spec.url,
api_key=spec.api_key,
api_secret=spec.api_secret.get_secret_value(),
weight=spec.weight,
)
for spec in parse_livekit_clusters(raw)
)
@lru_cache(maxsize=8)
def _clusters_from_configuration(
url: str, api_key: str, api_secret: str
) -> tuple[LiveKitClusterConfig, ...]:
"""Build the single LiveKit cluster that LIVEKIT_CONFIGURATION describes."""
return (
LiveKitClusterConfig(
name=FALLBACK_LIVEKIT_CLUSTER_NAME,
url=url,
api_url=url,
api_key=api_key,
api_secret=api_secret,
),
)
def get_livekit_clusters() -> tuple[LiveKitClusterConfig, ...]:
"""Return the registered LiveKit clusters, the default one first.
LIVEKIT_CLUSTERS settings lists them. When it is unset, a single cluster named
"default" is built from LIVEKIT_CONFIGURATION.
"""
if settings.LIVEKIT_CLUSTERS is not None:
return _clusters_from_json(settings.LIVEKIT_CLUSTERS)
configuration = settings.LIVEKIT_CONFIGURATION
return _clusters_from_configuration(
configuration["url"],
configuration["api_key"],
configuration["api_secret"],
)
def get_default_livekit_cluster() -> LiveKitClusterConfig:
"""Return the default LiveKit cluster: the first registered one."""
return get_livekit_clusters()[0]
def get_livekit_cluster(name: str) -> LiveKitClusterConfig:
"""Return the registered LiveKit cluster with this name."""
for cluster in get_livekit_clusters():
if cluster.name == name:
return cluster
raise UnknownLiveKitClusterError(f"No LiveKit cluster is named {name!r}.")
def get_livekit_cluster_by_api_key(api_key: str | None) -> LiveKitClusterConfig | None:
"""Return the registered LiveKit cluster using this API key, if any."""
if not api_key:
return None
for cluster in get_livekit_clusters():
if cluster.api_key == api_key:
return cluster
return None
def _get_livekit_cluster_by_issuer(token: str) -> LiveKitClusterConfig:
"""Return the registered LiveKit cluster whose API key is a token's issuer.
The issuer is read without verifying the token. A missing issuer, or one
that is no registered cluster's API key, raises the error PyJWT raises when
an issuer is missing or is not the expected one.
"""
payload = jwt.decode(token, options={"verify_signature": False})
if "iss" not in payload:
raise jwt.MissingRequiredClaimError("iss")
cluster = get_livekit_cluster_by_api_key(payload["iss"])
if cluster is None:
raise jwt.InvalidIssuerError("Invalid issuer")
return cluster
class LiveKitClusterTokenVerifier(TokenVerifier):
"""Verify tokens signed by any registered LiveKit cluster.
A token's issuer is the API key it was signed with, and each cluster has an
API key of its own, so the issuer names the cluster whose API key and secret
verify the token. A token without issuer, or issued by no registered
cluster, is refused as a TokenVerifier refuses a token without issuer or
issued with another API key than its own.
The claims returned carry that cluster as their `cluster`. The verifier
also keeps the cluster of the last token it accepted as its own `cluster`,
for callers that never see the claims, such as those of a webhook receiver,
which returns the event alone.
"""
cluster: LiveKitClusterConfig | None = None
def verify(self, token: str, *, verify_signature: bool = True) -> Claims:
"""Return the claims of a token, verified for the cluster that signed it."""
cluster = _get_livekit_cluster_by_issuer(token)
claims = TokenVerifier(
cluster.api_key, cluster.api_secret, leeway=self._leeway
).verify(token, verify_signature=verify_signature)
claims.cluster = cluster
self.cluster = cluster
return claims
+34
View File
@@ -0,0 +1,34 @@
"""LiveKit clusters specific exceptions."""
class UnknownLiveKitClusterError(LookupError):
"""Raised when no registered LiveKit cluster has the requested name."""
class LiveKitRoomUnavailableError(Exception):
"""Raised when a LiveKit room cannot be joined for now.
Either it cannot be placed on any LiveKit cluster, or it cannot be created
on the one it is placed on. Asking again later may succeed.
"""
class NoCandidateLiveKitClusterError(LiveKitRoomUnavailableError):
"""Raised when no LiveKit cluster may receive new placements."""
class LiveKitRoomCreationError(LiveKitRoomUnavailableError):
"""Raised when a LiveKit cluster fails to create a LiveKit room."""
class PlacementChangedError(LiveKitRoomUnavailableError):
"""Raised when the placement of a LiveKit room keeps changing while
credentials are handed out for it."""
class RelocationRefusedError(Exception):
"""Raised when a room cannot be relocated, nothing having changed.
Its LiveKit room is not placed, or no other LiveKit cluster may receive it,
or its placement changed meanwhile.
"""
+515
View File
@@ -0,0 +1,515 @@
"""Placements of LiveKit rooms on the LiveKit clusters.
A placement records the LiveKit cluster a LiveKit room lives on, from the first
credentials handed out for it until the LiveKit room finishes. Only rooms whose
lifecycle Django follows are placed: unregistered rooms never are, and live on
the default cluster.
"""
from logging import getLogger
from typing import Any
from uuid import uuid4
from django.conf import settings
from django.db.models import F
from django.utils import timezone
import aiohttp
from asgiref.sync import async_to_sync
from livekit.api import CreateRoomRequest, TwirpError # pylint: disable=E0611
from core import audit, auditing, utils
from core.models import Placement, Room
from .clusters import LiveKitClusterConfig, get_livekit_clusters
from .exceptions import (
LiveKitRoomCreationError,
NoCandidateLiveKitClusterError,
PlacementChangedError,
RelocationRefusedError,
)
from .strategies import PlacementContext, get_placement_strategy
logger = getLogger(__name__)
PLACEMENT_CREATION_ATTEMPTS = 3
PLACEMENT_UPDATE_ATTEMPTS = 2
def _audit(action: audit.Action, actor: Any, **fields: Any) -> None:
"""Emit the audit event of an action taken by an actor, the system when None."""
audit.log(
action,
actor=actor,
actor_type=audit.ActorType.SYSTEM if actor is None else None,
**fields,
)
def _insert_placement(new_placement: Placement) -> Placement | None:
"""Insert a placement, unless its LiveKit room is placed already, and
return the LiveKit room's placement read back.
bulk_create(ignore_conflicts=True) does not tell whether it inserted the
placement: the one read back is the new placement only if it has its id.
When another request placed the LiveKit room meanwhile and its placement is
gone by the time it is read back, the insertion is tried again, a bounded
number of times, after which None is returned.
"""
for _attempt in range(PLACEMENT_CREATION_ATTEMPTS):
Placement.objects.bulk_create([new_placement], ignore_conflicts=True)
placement = Placement.objects.filter(room_name=new_placement.room_name).first()
if placement is not None:
return placement
return None
class PlacementService:
"""Service for the placements of LiveKit rooms on LiveKit clusters."""
@classmethod
def cluster_for(cls, room_name: str) -> LiveKitClusterConfig:
"""Return the LiveKit cluster on which a LiveKit room is managed.
With a single registered cluster, every room lives there, and the
database is not queried. Otherwise it is the cluster of the room's
placement, as long as that cluster is registered. Rooms without
placement, either unregistered, live since before placements existed or
not placed yet, and rooms placed on a cluster since removed from the
registry, are managed on the default cluster.
"""
clusters = get_livekit_clusters()
default_cluster = clusters[0]
if len(clusters) == 1:
return default_cluster
placed_on = (
Placement.objects.filter(room_name=room_name)
.values_list("cluster", flat=True)
.first()
)
if placed_on is None:
return default_cluster
for cluster in clusters:
if cluster.name == placed_on:
return cluster
logger.warning(
"LiveKit room %s is placed on LiveKit cluster %s, which is not "
"registered: managing it on the default cluster %s",
room_name,
placed_on,
default_cluster.name,
)
return default_cluster
@classmethod
def candidates(cls) -> tuple[LiveKitClusterConfig, ...]:
"""Return the LiveKit clusters that may receive new placements.
They are the registered clusters, in registry order, but for those
weighing 0, which keep their live rooms and receive no new placement.
"""
return tuple(
cluster for cluster in get_livekit_clusters() if cluster.weight > 0
)
@classmethod
def choose_cluster(
cls, room_name: str, room: Room | None = None
) -> LiveKitClusterConfig:
"""Return the LiveKit cluster on which to place a LiveKit room.
The placement strategy named by LIVEKIT_PLACEMENT_STRATEGY picks it
among the candidates, knowing the LiveKit room's name and the room it is
named after, if any. Without any candidate, no LiveKit room can be
placed anywhere: this is logged as critical and raises
NoCandidateLiveKitClusterError.
"""
candidates = cls.candidates()
if not candidates:
logger.critical(
"No LiveKit cluster may receive new placements: LiveKit room %s "
"cannot be placed",
room_name,
)
raise NoCandidateLiveKitClusterError(
"No LiveKit cluster may receive new placements."
)
context = PlacementContext(room_name=room_name, room=room)
return get_placement_strategy().choose(candidates, context)
@classmethod
def place(cls, room_name: str, room: Room | None = None) -> LiveKitClusterConfig:
"""Make a LiveKit room ready to be joined and return its LiveKit cluster.
It is called each time credentials are handed out for the LiveKit room,
which must be for the cluster returned. A LiveKit room without placement
is placed on the cluster the placement strategy picks, knowing the room
it is named after, if any. A placement on a cluster no longer registered
is deleted, and the LiveKit room placed again.
LiveKit clusters need not create rooms on their own, so the LiveKit room
is then created on its cluster, unless it is live there already, and
its SID recorded on its placement, which is no longer absent from its
cluster. That is done only as long as the placement is still the one
read, with the same epoch, and with the SID read or the very SID the
cluster just returned, which another request creating the LiveKit room
at the same time records too. Otherwise, as when it was relocated or
deleted meanwhile, or got another SID, the placement is read again and
the LiveKit room created once more.
Raise LiveKitRoomUnavailableError, which means trying again later:
NoCandidateLiveKitClusterError when no cluster may receive the LiveKit
room, LiveKitRoomCreationError when its cluster fails to create it, and
PlacementChangedError when its placement keeps changing. A placement
created by this call is deleted when its LiveKit room cannot be created,
since no one was handed out credentials for it, so that the next call
may place the LiveKit room elsewhere. An older placement stays.
"""
for _attempt in range(PLACEMENT_UPDATE_ATTEMPTS):
placement, cluster, created = cls._get_or_create_placement(room_name, room)
try:
sid = cls.create_livekit_room(room_name, cluster)
except LiveKitRoomCreationError:
if created:
cls._delete_created_placement(placement, cluster)
raise
if Placement.objects.filter(
pk=placement.pk, epoch=placement.epoch, sid__in=(placement.sid, sid)
).update(sid=sid, absent_since=None, updated_at=timezone.now()):
return cluster
logger.info(
"Placement of LiveKit room %s changed while LiveKit cluster %s "
"created it: reading it again",
room_name,
cluster.name,
)
logger.warning(
"Placement of LiveKit room %s kept changing while its LiveKit room "
"was created",
room_name,
)
raise PlacementChangedError(
f"The placement of LiveKit room {room_name} kept changing."
)
@classmethod
def _get_or_create_placement(
cls, room_name: str, room: Room | None
) -> tuple[Placement, LiveKitClusterConfig, bool]:
"""Return the placement of a LiveKit room, placing it if need be.
Return the placement, its registered LiveKit cluster and whether this
call created the placement. Another request may place the LiveKit room
at the same time: both then get the one placement read back. A
placement on a cluster no longer registered is deleted, provided it did
not change meanwhile, and the LiveKit room placed again, a bounded
number of times.
"""
clusters = {cluster.name: cluster for cluster in get_livekit_clusters()}
for _attempt in range(PLACEMENT_CREATION_ATTEMPTS):
created = False
placement = Placement.objects.filter(room_name=room_name).first()
if placement is None:
new_placement = Placement(
room_name=room_name,
room=room,
cluster=cls.choose_cluster(room_name, room).name,
)
placement = _insert_placement(new_placement)
if placement is None:
break
created = placement.pk == new_placement.pk
if created:
logger.info(
"Placed LiveKit room %s on LiveKit cluster %s",
room_name,
placement.cluster,
)
cluster = clusters.get(placement.cluster)
if cluster is not None:
return placement, cluster, created
logger.warning(
"LiveKit room %s is placed on LiveKit cluster %s, which is not "
"registered: placing it again",
room_name,
placement.cluster,
)
Placement.objects.filter(
pk=placement.pk, epoch=placement.epoch, cluster=placement.cluster
).delete()
raise PlacementChangedError(
f"The placement of LiveKit room {room_name} kept changing."
)
@staticmethod
def _delete_created_placement(placement: Placement, cluster: LiveKitClusterConfig):
"""Delete a placement whose LiveKit room its cluster failed to create.
It is deleted only while it still has the epoch read and no SID: no one
was handed out credentials for it then.
"""
deleted, _ = Placement.objects.filter(
pk=placement.pk, epoch=placement.epoch, sid=""
).delete()
if deleted:
logger.info(
"Deleted the placement of LiveKit room %s on LiveKit cluster %s, "
"which failed to create it",
placement.room_name,
cluster.name,
)
@classmethod
def create_livekit_room(cls, room_name: str, cluster: LiveKitClusterConfig) -> str:
"""Create a LiveKit room on a LiveKit cluster and return its SID.
A LiveKit room live there already is kept as it is, and its SID
returned. No metadata is sent: the cluster would store it in place of
a live LiveKit room's, but leave the live room with its own. The
cluster is given LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS to answer.
Raise LiveKitRoomCreationError when the cluster cannot be reached, does
not answer in time or answers with an error.
"""
return cls._create_livekit_room(room_name, cluster)
@staticmethod
@async_to_sync
async def _create_livekit_room(
room_name: str, cluster: LiveKitClusterConfig
) -> str:
"""Create a LiveKit room on a LiveKit cluster and return its SID."""
lkapi = utils.create_livekit_client(
cluster=cluster, timeout=settings.LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS
)
try:
livekit_room = await lkapi.room.create_room(
CreateRoomRequest(name=room_name)
)
except (TwirpError, TimeoutError, aiohttp.ClientError) as error:
logger.exception(
"LiveKit cluster %s failed to create LiveKit room %s",
cluster.name,
room_name,
)
raise LiveKitRoomCreationError(
f"LiveKit cluster {cluster.name} failed to create LiveKit room "
f"{room_name}."
) from error
finally:
await lkapi.aclose()
return livekit_room.sid
@classmethod
def relocate(
cls, room_name: str, to: str | None = None, *, actor: Any
) -> LiveKitClusterConfig:
"""Relocate a room to another LiveKit cluster and return that cluster.
The placement of its LiveKit room moves to the candidate named `to`, or
else to the one the placement strategy picks among the candidates other
than its current cluster. Credentials are then handed out for the new
cluster, where the next join creates the LiveKit room. The LiveKit room
is closed on the cluster it leaves, which tells its participants to ask
Django again where to connect. A cluster failing to close it is logged:
the relocation stands.
The placement is read once, and moves only as long as it is still the
one read, with the same epoch, on the same cluster: the relocation of a
placement that changed meanwhile is refused, for the caller to ask
again.
`actor` is who relocates the room, as recorded in the audit log: a user,
such as one acting through the admin, or None for the system.
Raise RelocationRefusedError, having changed nothing, when the LiveKit
room is not placed, or is placed on a cluster that is not registered,
when `to` names no other candidate, when there is no other candidate,
and when the placement changed since it was read: it is never moved
twice.
"""
source, target = cls._move_placement(room_name, to)
relocation_id = str(uuid4())
logger.info(
"Relocated LiveKit room %s from LiveKit cluster %s to LiveKit cluster %s",
room_name,
source.name,
target.name,
)
_audit(
auditing.LIVEKIT_PLACEMENT_RELOCATE,
actor,
target=auditing.livekit_room_target(room_name),
source_cluster=source.name,
target_cluster=target.name,
relocation_id=relocation_id,
)
cls._close_livekit_room(source, room_name, relocation_id, actor=actor)
return target
@classmethod
def _move_placement(
cls, room_name: str, to: str | None
) -> tuple[LiveKitClusterConfig, LiveKitClusterConfig]:
"""Move the placement of a LiveKit room to another LiveKit cluster.
Return the cluster it left and the one it moved to. The move bumps its
epoch and records the cluster it left, and when. It clears the SID and
the absence, which were those of the LiveKit room on the cluster left:
the LiveKit room gets a SID once created on the new cluster.
"""
placement = (
Placement.objects.select_related("room").filter(room_name=room_name).first()
)
if placement is None:
raise RelocationRefusedError(f"LiveKit room {room_name} is not placed.")
source, target = cls._relocation_clusters(placement, to)
now = timezone.now()
if not Placement.objects.filter(
pk=placement.pk, epoch=placement.epoch, cluster=source.name
).update(
cluster=target.name,
previous_cluster=source.name,
relocated_at=now,
sid="",
absent_since=None,
epoch=F("epoch") + 1,
updated_at=now,
):
raise RelocationRefusedError(
f"The placement of LiveKit room {room_name} changed meanwhile."
)
return source, target
@classmethod
def _relocation_clusters(
cls, placement: Placement, to: str | None
) -> tuple[LiveKitClusterConfig, LiveKitClusterConfig]:
"""Return the LiveKit cluster a placement leaves and the one it moves to.
That is the candidate named `to`, or else the one the placement strategy
picks among the candidates other than the cluster the placement leaves.
"""
room_name = placement.room_name
clusters = {cluster.name: cluster for cluster in get_livekit_clusters()}
source = clusters.get(placement.cluster)
if source is None:
raise RelocationRefusedError(
f"LiveKit room {room_name} is placed on LiveKit cluster "
f"{placement.cluster}, which is not registered."
)
candidates = tuple(
cluster for cluster in cls.candidates() if cluster.name != source.name
)
if to is None:
if not candidates:
raise RelocationRefusedError(
f"No LiveKit cluster other than {source.name} may receive "
f"LiveKit room {room_name}."
)
context = PlacementContext(room_name=room_name, room=placement.room)
return source, get_placement_strategy().choose(candidates, context)
if to == source.name:
raise RelocationRefusedError(
f"LiveKit room {room_name} is placed on LiveKit cluster {to} already."
)
for candidate in candidates:
if candidate.name == to:
return source, candidate
if to not in clusters:
raise RelocationRefusedError(f"No LiveKit cluster is named {to!r}.")
raise RelocationRefusedError(
f"LiveKit cluster {to} may not receive new placements."
)
@classmethod
def close_stray_room(
cls, cluster: LiveKitClusterConfig, room_name: str, *, actor: Any
) -> bool:
"""Close a LiveKit room live on a LiveKit cluster it is not placed on.
Its participants are told of a relocation, which makes them ask Django
again where to connect, and the LiveKit room is deleted from that
cluster. Its placement, if any, is left as it is.
`actor` is who closes the LiveKit room, as recorded in the audit log: a
user, such as one acting through the admin, or None for the system.
Return whether the LiveKit room was closed. Failures are logged and
audited, never raised.
"""
return cls._close_livekit_room(cluster, room_name, str(uuid4()), actor=actor)
@staticmethod
def _close_livekit_room(
cluster: LiveKitClusterConfig,
room_name: str,
relocation_id: str,
*,
actor: Any,
) -> bool:
"""Announce a relocation in a LiveKit room, then delete it, on a cluster.
The relocation id is merged into the LiveKit room's metadata, which
reaches participants ahead of the disconnection its deletion causes, and
which participants joining it read as they join: they ask Django again
where to connect, once per relocation id. A LiveKit room that is not
live there is left alone. When the cluster fails to announce the
relocation, the LiveKit room is not deleted, as that would send its
participants away as if the meeting were over.
Return whether the LiveKit room was closed. Failures are logged and
audited, never raised.
"""
# pylint: disable-next=import-outside-toplevel,cyclic-import
from core.services import room_management # noqa: PLC0415
error = None
try:
room_management.RoomManagement.update_metadata(
room_name, {"relocation": {"id": relocation_id}}, cluster=cluster
)
room_management.RoomManagement.delete_room(room_name, cluster=cluster)
except room_management.RoomNotFoundException:
logger.info(
"LiveKit room %s is not live on LiveKit cluster %s: nothing to close",
room_name,
cluster.name,
)
return False
except Exception as failure: # pylint: disable=broad-exception-caught
logger.exception(
"LiveKit cluster %s failed to close LiveKit room %s",
cluster.name,
room_name,
)
error = failure
else:
logger.info(
"Closed LiveKit room %s on LiveKit cluster %s", room_name, cluster.name
)
_audit(
auditing.LIVEKIT_ROOM_CLOSE,
actor,
target=auditing.livekit_room_target(room_name),
target_service="livekit",
outcome=audit.Outcome.SUCCESS if error is None else audit.Outcome.FAILURE,
reason=None if error is None else audit.Reason.INTERNAL_ERROR,
error_type=None if error is None else audit.exception_type(error),
cluster=cluster.name,
relocation_id=relocation_id,
)
return error is None
+117
View File
@@ -0,0 +1,117 @@
"""Specification of the LiveKit clusters settings (LIVEKIT_CLUSTERS)."""
import re
from typing import Annotated
from pydantic import (
AfterValidator,
AnyUrl,
BaseModel,
Field,
NonNegativeInt,
SecretStr,
StringConstraints,
TypeAdapter,
UrlConstraints,
ValidationError,
)
from pydantic_core import PydanticCustomError
LIVEKIT_CLUSTER_NAME_MAX_LENGTH = 63
LIVEKIT_CLUSTER_NAME_PATTERN = rf"^[a-z0-9-]{{1,{LIVEKIT_CLUSTER_NAME_MAX_LENGTH}}}$"
_LIVEKIT_URL_ADAPTER = TypeAdapter(
Annotated[AnyUrl, UrlConstraints(allowed_schemes=["http", "https", "ws", "wss"])]
)
_LIVEKIT_URL_AUTHORITY = re.compile(r"(?i)(https?|wss?)://[^/\\]")
def _check_livekit_url(url: str) -> str:
"""Ensure a URL reaches LiveKit over HTTP/WebSocket, and stay unchanged."""
if " " in url or not url.isprintable():
raise PydanticCustomError(
"url_characters",
"URL should not contain whitespace or non-printable characters",
)
_LIVEKIT_URL_ADAPTER.validate_python(url)
if not _LIVEKIT_URL_AUTHORITY.match(url):
raise PydanticCustomError(
"url_authority", "URL should be written scheme://host"
)
return url
LiveKitUrl = Annotated[str, AfterValidator(_check_livekit_url)]
class LiveKitClusterSpec(BaseModel):
"""A LiveKit cluster, as listed in LIVEKIT_CLUSTERS.
Browsers connect to `url` and Django calls `api_url`, which defaults to
`url`. Each cluster has its own API key, which tells it apart.
"""
name: Annotated[str, StringConstraints(pattern=LIVEKIT_CLUSTER_NAME_PATTERN)]
url: LiveKitUrl
api_url: LiveKitUrl | None = None
api_key: Annotated[str, StringConstraints(min_length=1)]
api_secret: Annotated[SecretStr, Field(min_length=1)]
weight: NonNegativeInt = 1
model_config = {"extra": "forbid"}
_LIVEKIT_CLUSTERS_ADAPTER = TypeAdapter(
Annotated[list[LiveKitClusterSpec], Field(min_length=1)]
)
def _locate(loc) -> str:
"""Name a place in LIVEKIT_CLUSTERS, such as LIVEKIT_CLUSTERS[0].url."""
return "LIVEKIT_CLUSTERS" + "".join(
f"[{part}]" if isinstance(part, int) else f".{part}" for part in loc
)
def _find_duplicates(specs: list[LiveKitClusterSpec]) -> list[str]:
"""Describe each cluster reusing the name or the API key of an earlier one."""
problems = []
first_by_name = {}
first_by_api_key = {}
for index, spec in enumerate(specs):
first = first_by_name.setdefault(spec.name, index)
if first != index:
problems.append(
f"{_locate((index, 'name'))}: {spec.name!r} is already the name "
f"of {_locate((first,))}"
)
first = first_by_api_key.setdefault(spec.api_key, index)
if first != index:
problems.append(
f"{_locate((index, 'api_key'))}: {_locate((first,))} already uses "
"this API key, each LiveKit cluster needs its own"
)
return problems
def parse_livekit_clusters(raw: str) -> list[LiveKitClusterSpec]:
"""Parse LIVEKIT_CLUSTERS, a non-empty JSON list of LiveKit clusters.
The first cluster is the default one. Names and API keys must be unique.
Every problem found is reported in a single ValueError.
"""
specs = []
problems = []
try:
specs = _LIVEKIT_CLUSTERS_ADAPTER.validate_json(raw, strict=True)
except ValidationError as error:
problems = [
f"{_locate(problem['loc'])}: {problem['msg']}"
for problem in error.errors(include_input=False, include_url=False)
]
problems += _find_duplicates(specs)
if problems:
raise ValueError("; ".join(problems))
return specs
+145
View File
@@ -0,0 +1,145 @@
"""Placement strategies, the rules picking the LiveKit cluster of a new placement.
LIVEKIT_PLACEMENT_STRATEGY names the strategy in use. It picks among candidates:
the LiveKit clusters that may receive new placements.
"""
from abc import ABC, abstractmethod
from bisect import bisect_right
from dataclasses import dataclass
from fractions import Fraction
from itertools import accumulate
from secrets import randbelow
from django.conf import settings
from django.core.cache import cache
from django.db.models import Count
from django.utils.module_loading import import_string
from core.models import Placement, Room
from .clusters import LiveKitClusterConfig
@dataclass(frozen=True, kw_only=True)
class PlacementContext:
"""What a placement strategy is told about the LiveKit room to place.
`room` is the room the LiveKit room is named after, or None for a
connection test room.
"""
room_name: str
room: Room | None = None
class PlacementStrategy(ABC):
"""A placement strategy, which picks the LiveKit cluster of a new placement.
A strategy is instantiated each time a cluster is chosen, so whatever it
has to remember goes to the cache or the database.
"""
@abstractmethod
def choose(
self,
candidates: tuple[LiveKitClusterConfig, ...],
context: PlacementContext,
) -> LiveKitClusterConfig:
"""Return the candidate on which to place a LiveKit room.
The candidates are the LiveKit clusters that may receive new placements,
in registry order. There is at least one, and each weighs more than 0.
"""
class WeightedRandomStrategy(PlacementStrategy):
"""Pick LiveKit clusters at random, in proportion to their weights.
Clusters of equal weights receive as many new placements on average, and
with weights of 1 and 3, the second cluster receives three quarters of them.
"""
def choose(
self,
candidates: tuple[LiveKitClusterConfig, ...],
context: PlacementContext,
) -> LiveKitClusterConfig:
"""Return a candidate drawn with a probability proportional to its weight."""
cumulative_weights = list(accumulate(cluster.weight for cluster in candidates))
draw = randbelow(cumulative_weights[-1])
return candidates[bisect_right(cumulative_weights, draw)]
class RoundRobinStrategy(PlacementStrategy):
"""Pick the LiveKit clusters in turn, whatever their weights.
The first new placement among candidates goes to the first of them, the
next one to the second, and so on. Turns are counted in the Django cache,
which all processes share, with a counter for each set of candidates.
"""
@staticmethod
def counter_key(candidates: tuple[LiveKitClusterConfig, ...]) -> str:
"""Return the cache key counting the placements made among candidates."""
names = ",".join(cluster.name for cluster in candidates)
return f"livekit-placement-round-robin_{names}"
def choose(
self,
candidates: tuple[LiveKitClusterConfig, ...],
context: PlacementContext,
) -> LiveKitClusterConfig:
"""Return the candidate whose turn it is.
Should the counter vanish between its creation and its increment, the
turn goes to the first candidate.
"""
key = self.counter_key(candidates)
cache.add(key, 0, timeout=None)
try:
count = cache.incr(key)
except ValueError:
count = 1
return candidates[(count - 1) % len(candidates)]
class LeastPlacementsStrategy(PlacementStrategy):
"""Pick the LiveKit cluster holding the fewest placements for its weight.
A cluster weighing twice as much as another is given twice as many
placements. On a tie, the cluster registered first is picked. Rooms placed
at the same moment may all go to the same cluster, as each of them is
placed before the placements of the others are recorded.
"""
def choose(
self,
candidates: tuple[LiveKitClusterConfig, ...],
context: PlacementContext,
) -> LiveKitClusterConfig:
"""Return the candidate holding the fewest placements for its weight."""
placement_counts = dict(
Placement.objects.filter(
cluster__in=[cluster.name for cluster in candidates]
)
.values_list("cluster")
.annotate(Count("pk"))
.order_by()
)
return min(
candidates,
key=lambda cluster: Fraction(
placement_counts.get(cluster.name, 0), cluster.weight
),
)
def get_placement_strategy() -> PlacementStrategy:
"""Return the placement strategy named by LIVEKIT_PLACEMENT_STRATEGY.
It is looked up on every call and never while the settings load, so that a
strategy may need applications that are not ready then, and a change of
settings applies at once.
"""
return import_string(settings.LIVEKIT_PLACEMENT_STRATEGY)()
@@ -0,0 +1,70 @@
# Generated by Django 5.2.17 on 2026-11-06 22:46
import django.db.models.deletion
import uuid
from django.db import migrations, models
SET_NULL_ON_ROOM_DELETE = """
DO $$
DECLARE
fk_name text;
BEGIN
SELECT conname INTO fk_name
FROM pg_constraint
WHERE contype = 'f'
AND conrelid = 'meet_placement'::regclass
AND confrelid = 'meet_room'::regclass;
EXECUTE format(
'ALTER TABLE meet_placement DROP CONSTRAINT %I', fk_name
);
EXECUTE format(
'ALTER TABLE meet_placement ADD CONSTRAINT %I '
'FOREIGN KEY (room_id) REFERENCES meet_room (resource_id) '
'ON DELETE SET NULL DEFERRABLE INITIALLY DEFERRED',
fk_name
);
END $$;
"""
def set_null_on_room_delete(apps, schema_editor):
"""Make PostgreSQL itself unset the room of a placement when the room is deleted."""
for statement in schema_editor.deferred_sql:
schema_editor.execute(statement)
schema_editor.deferred_sql.clear()
schema_editor.execute(SET_NULL_ON_ROOM_DELETE, params=None)
class Migration(migrations.Migration):
dependencies = [
('core', '0029_alter_application_client_secret_sha256'),
]
operations = [
migrations.CreateModel(
name='Placement',
fields=[
('id', models.UUIDField(default=uuid.uuid4, editable=False, help_text='primary key for the record as UUID', primary_key=True, serialize=False, verbose_name='id')),
('created_at', models.DateTimeField(auto_now_add=True, help_text='date and time at which a record was created', verbose_name='created on')),
('updated_at', models.DateTimeField(auto_now=True, help_text='date and time at which a record was last updated', verbose_name='updated on')),
('room_name', models.CharField(help_text="The room's id.", max_length=255, unique=True, verbose_name='LiveKit room name')),
('cluster', models.CharField(db_index=True, help_text='Name of the LiveKit cluster the LiveKit room lives on.', max_length=63, verbose_name='LiveKit cluster')),
('sid', models.CharField(blank=True, default='', help_text='SID of the LiveKit room on its cluster, empty until known.', max_length=63, verbose_name='LiveKit room SID')),
('epoch', models.PositiveIntegerField(default=0, help_text='Incremented on every relocation.', verbose_name='Epoch')),
('previous_cluster', models.CharField(blank=True, default='', help_text='Name of the LiveKit cluster the room was last relocated from.', max_length=63, verbose_name='Previous LiveKit cluster')),
('relocated_at', models.DateTimeField(blank=True, help_text='Date and time at which the room was last relocated.', null=True, verbose_name='Relocated at')),
('absent_since', models.DateTimeField(blank=True, help_text='Date and time since which the LiveKit room is missing from its cluster.', null=True, verbose_name='Absent since')),
('room', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='placements', to='core.room', verbose_name='Room')),
],
options={
'verbose_name': 'Placement',
'verbose_name_plural': 'Placements',
'db_table': 'meet_placement',
},
),
migrations.RunPython(
set_null_on_room_delete, reverse_code=migrations.RunPython.noop
),
]
@@ -0,0 +1,18 @@
# Generated by Django 5.2.17 on 2026-11-07 15:03
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('core', '0030_placement'),
]
operations = [
migrations.AddField(
model_name='recording',
name='livekit_cluster',
field=models.CharField(blank=True, db_default='', default='', help_text='Name of the LiveKit cluster recording the room, the default cluster when empty.', max_length=63, verbose_name='LiveKit cluster'),
),
]
+112
View File
@@ -27,6 +27,13 @@ from lasuite.tools.email import get_domain_from_email
from timezone_field import TimeZoneField
from . import fields, hashers, utils
from .livekit.clusters import (
LiveKitClusterConfig,
get_default_livekit_cluster,
get_livekit_cluster,
)
from .livekit.exceptions import UnknownLiveKitClusterError
from .livekit.specs import LIVEKIT_CLUSTER_NAME_MAX_LENGTH
from .recording.enums import FileExtension
from .validators import sub_validator
@@ -707,6 +714,17 @@ class Recording(BaseModel):
verbose_name=_("External Process ID"),
help_text=_("ID of the external process associated with the recording."),
)
livekit_cluster = models.CharField(
max_length=LIVEKIT_CLUSTER_NAME_MAX_LENGTH,
blank=True,
default="",
db_default="",
verbose_name=_("LiveKit cluster"),
help_text=_(
"Name of the LiveKit cluster recording the room, the default "
"cluster when empty."
),
)
class Meta:
db_table = "meet_recording"
@@ -775,6 +793,30 @@ class Recording(BaseModel):
for user_id in room_privileged_user_ids
)
def get_livekit_cluster(self) -> LiveKitClusterConfig:
"""Return the LiveKit cluster recording the room.
It is the cluster the room was placed on when the recording started,
wherever the room was placed since. A recording naming no cluster, as
one started before recordings kept theirs, is on the default cluster.
So is a recording on a cluster no longer registered, which is logged.
"""
if not self.livekit_cluster:
return get_default_livekit_cluster()
try:
return get_livekit_cluster(self.livekit_cluster)
except UnknownLiveKitClusterError:
cluster = get_default_livekit_cluster()
logger.warning(
"Recording %s is on LiveKit cluster %s, which is not registered: "
"managing it on the default cluster %s",
self.id,
self.livekit_cluster,
cluster.name,
)
return cluster
def is_savable(self) -> bool:
"""Determine if the recording can be saved based on its current status."""
@@ -1262,3 +1304,73 @@ class File(BaseModel):
self.hard_deleted_at = timezone.now()
self.save(update_fields=["hard_deleted_at"])
class Placement(BaseModel):
"""The record that a LiveKit room lives on a given LiveKit cluster.
It lasts from the first credentials handed out for the LiveKit room until
the LiveKit room finishes.
"""
room_name = models.CharField(
max_length=255,
unique=True,
verbose_name=_("LiveKit room name"),
help_text=_("The room's id."),
)
room = models.ForeignKey(
Room,
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name="placements",
verbose_name=_("Room"),
)
cluster = models.CharField(
max_length=LIVEKIT_CLUSTER_NAME_MAX_LENGTH,
db_index=True,
verbose_name=_("LiveKit cluster"),
help_text=_("Name of the LiveKit cluster the LiveKit room lives on."),
)
sid = models.CharField(
max_length=63,
blank=True,
default="",
verbose_name=_("LiveKit room SID"),
help_text=_("SID of the LiveKit room on its cluster, empty until known."),
)
epoch = models.PositiveIntegerField(
default=0,
verbose_name=_("Epoch"),
help_text=_("Incremented on every relocation."),
)
previous_cluster = models.CharField(
max_length=LIVEKIT_CLUSTER_NAME_MAX_LENGTH,
blank=True,
default="",
verbose_name=_("Previous LiveKit cluster"),
help_text=_("Name of the LiveKit cluster the room was last relocated from."),
)
relocated_at = models.DateTimeField(
null=True,
blank=True,
verbose_name=_("Relocated at"),
help_text=_("Date and time at which the room was last relocated."),
)
absent_since = models.DateTimeField(
null=True,
blank=True,
verbose_name=_("Absent since"),
help_text=_(
"Date and time since which the LiveKit room is missing from its cluster."
),
)
class Meta:
db_table = "meet_placement"
verbose_name = _("Placement")
verbose_name_plural = _("Placements")
def __str__(self):
return f"{self.room_name} on {self.cluster}"
@@ -121,7 +121,7 @@ class NotificationService:
return not has_failures
@staticmethod
async def _get_recording_timestamps(worker_id):
async def _get_recording_timestamps(worker_id, cluster):
"""Fetch FileInfo.started_at and ended_at from LiveKit's egress API.
FileInfo.started_at is more accurate than EgressInfo.started_at because
@@ -130,6 +130,9 @@ class NotificationService:
corresponds to when the egress service received the request, not the moment
the egress worker effectively joined the room.
The egress is looked up on `cluster`, the LiveKit cluster that recorded
the room.
Returns:
Tuple of (started_at, ended_at) datetimes, either may be None.
"""
@@ -137,7 +140,7 @@ class NotificationService:
if not worker_id:
return None, None
lkapi = utils.create_livekit_client()
lkapi = utils.create_livekit_client(cluster=cluster)
try:
egress_list = await lkapi.egress.list_egress(
livekit_api.ListEgressRequest(egress_id=worker_id) # pylint: disable=no-member
@@ -266,7 +269,7 @@ class NotificationService:
started_at, ended_at = async_to_sync(
NotificationService._get_recording_timestamps
)(recording.worker_id)
)(recording.worker_id, recording.get_livekit_cluster())
payload = {
"owner_id": str(recipient.id),
@@ -335,7 +338,7 @@ class NotificationService:
started_at, ended_at = async_to_sync(
NotificationService._get_recording_timestamps
)(recording.worker_id)
)(recording.worker_id, recording.get_livekit_cluster())
form_base_url = settings.TRANSCRIPTION_SATISFACTION_FORM_BASE_URL
form_link = (
@@ -10,6 +10,7 @@ from livekit.protocol.agent_dispatch import (
)
from core import utils
from core.livekit.clusters import LiveKitClusterConfig
from core.models import Recording
logger = getLogger(__name__)
@@ -20,13 +21,22 @@ class MetadataCollectorException(Exception):
class MetadataCollectorService:
"""Service for dispatching and managing the metadata collector agent."""
"""Service for dispatching and managing the metadata collector agent.
@async_to_sync
async def start(self, recording: Recording):
The agent is dispatched to the LiveKit room on the LiveKit cluster
recording it, where its dispatch is deleted too.
"""
def start(self, recording: Recording):
"""Explicitly dispatch the metadata collector agent to a room."""
lkapi = utils.create_livekit_client()
return self._start(recording, recording.get_livekit_cluster())
@async_to_sync
async def _start(self, recording: Recording, cluster: LiveKitClusterConfig):
"""Dispatch the metadata collector agent to a room on a LiveKit cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
room_id = str(recording.room.id)
try:
@@ -60,13 +70,18 @@ class MetadataCollectorService:
return dispatch_id
@async_to_sync
async def stop(self, recording: Recording):
def stop(self, recording: Recording):
"""Stop and delete the agent dispatch associated to the room."""
return self._stop(recording, recording.get_livekit_cluster())
@async_to_sync
async def _stop(self, recording: Recording, cluster: LiveKitClusterConfig):
"""Delete the agent dispatch associated to the room on a LiveKit cluster."""
room_id = str(recording.room.id)
dispatch_id = recording.options.get("metadata_collector_dispatch_id")
lkapi = utils.create_livekit_client()
lkapi = utils.create_livekit_client(cluster=cluster)
try:
if not dispatch_id:
@@ -108,6 +108,7 @@ class RecordingEventsService:
utils.notify_participants(
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=recording.get_livekit_cluster(),
)
except utils.NotificationError as e:
raise RecordingEventsError(
@@ -139,7 +140,9 @@ class RecordingEventsService:
if recording_status:
try:
RoomManagement.update_metadata(
room_name, {"recording_status": recording_status}
room_name,
{"recording_status": recording_status},
cluster=recording.get_livekit_cluster(),
)
except RoomNotFoundException:
logger.info(
+18 -5
View File
@@ -3,7 +3,7 @@
# pylint: disable=no-member
import logging
from dataclasses import dataclass
from dataclasses import dataclass, replace
from functools import lru_cache
from typing import Any, ClassVar, Dict, Optional, Protocol, Type
@@ -12,6 +12,9 @@ from django.utils.module_loading import import_string
from livekit import api as livekit_api
from ... import utils
from ...livekit.clusters import LiveKitClusterConfig
logger = logging.getLogger(__name__)
# Codec / frequency constants matching LiveKit's H264_720P_30 preset.
@@ -91,7 +94,7 @@ class WorkerServiceConfig:
return cls(
output_folder=settings.RECORDING_OUTPUT_FOLDER,
server_configurations=settings.LIVEKIT_CONFIGURATION,
server_configurations=utils.livekit_api_configuration(),
bucket_args={
"endpoint": settings.AWS_S3_ENDPOINT_URL,
"access_key": settings.AWS_S3_ACCESS_KEY_ID,
@@ -124,8 +127,15 @@ class WorkerService(Protocol):
"""Stop recording for a specified worker."""
def get_worker_service(mode: str) -> WorkerService:
"""Instantiate a worker service by its mode."""
def get_worker_service(
mode: str, *, cluster: Optional[LiveKitClusterConfig] = None
) -> WorkerService:
"""Instantiate a worker service by its mode, recording on a LiveKit cluster.
The worker service calls `cluster`, by default the default cluster as
registered now, with the URL, API key and secret Django calls it with. The
rest of its configuration comes from the settings, read once.
"""
worker_registry: Dict[str, str] = settings.RECORDING_WORKER_CLASSES
@@ -139,5 +149,8 @@ def get_worker_service(mode: str) -> WorkerService:
worker_class: Type[WorkerService] = import_string(worker_class_path)
config = WorkerServiceConfig.from_settings()
config = replace(
WorkerServiceConfig.from_settings(),
server_configurations=utils.livekit_api_configuration(cluster),
)
return worker_class(config=config)
@@ -72,7 +72,9 @@ class WorkerServiceMediator:
try:
RoomManagement.update_metadata(
room_name, {"recording_mode": mode, "recording_status": "starting"}
room_name,
{"recording_mode": mode, "recording_status": "starting"},
cluster=recording.get_livekit_cluster(),
)
except RoomNotFoundException:
logger.info(
+7
View File
@@ -16,6 +16,7 @@ from rest_framework import (
from core import analytics, models
from core.api import permissions, throttling
from core.api.feature_flag import FeatureFlag
from core.livekit.placements import PlacementService
from core.services.sip_management import SIPException, SIPManagement
from . import authentication, serializers
@@ -51,6 +52,10 @@ class RoomKitViewSet(viewsets.ViewSet):
The webhook-based creation path is kept: both converge on the same rule
through the shared SIPManagement.
The device may be the room's first participant, so the LiveKit room is
placed and created first, since LiveKit clusters need not create rooms on
their own. When it cannot be, the response is a 503.
"""
serializer = serializers.RoomKitJoinSerializer(data=request.data)
@@ -63,6 +68,8 @@ class RoomKitViewSet(viewsets.ViewSet):
except models.Room.DoesNotExist as e:
raise drf_exceptions.NotFound("No room found for this PIN code.") from e
PlacementService.place(str(room.id), room)
try:
created = SIPManagement().ensure_dispatch_rule(room)
except SIPException as e:
+91 -17
View File
@@ -13,6 +13,11 @@ from django.utils import timezone
from livekit import api
from core import audit, auditing, models
from core.livekit.clusters import (
LiveKitClusterConfig,
LiveKitClusterTokenVerifier,
get_default_livekit_cluster,
)
from core.recording.enums import UNSUCCESSFUL_EVENTS, RecordingWorkerEvent
from core.recording.event.authentication import MachineUser
from core.recording.services.metadata_collector import (
@@ -147,18 +152,14 @@ class LiveKitEventsService:
"""Initialize with required services."""
self._webhook_handlers = {
"egress_updated": self._handle_egress_updated,
"egress_ended": self._handle_egress_ended,
"room_started": self._handle_room_started,
"room_finished": self._handle_room_finished,
"participant_left": self._handle_participant_left,
LiveKitWebhookEventType.EGRESS_UPDATED: self._handle_egress_updated,
LiveKitWebhookEventType.EGRESS_ENDED: self._handle_egress_ended,
LiveKitWebhookEventType.ROOM_STARTED: self._handle_room_started,
LiveKitWebhookEventType.ROOM_FINISHED: self._handle_room_finished,
LiveKitWebhookEventType.PARTICIPANT_LEFT: self._handle_participant_left,
}
token_verifier = api.TokenVerifier(
settings.LIVEKIT_CONFIGURATION["api_key"],
settings.LIVEKIT_CONFIGURATION["api_secret"],
)
self.webhook_receiver = api.WebhookReceiver(token_verifier)
self.token_verifier = LiveKitClusterTokenVerifier()
self.lobby_service = LobbyService()
self.presence_cache = PresenceCache()
self.sip_management = SIPManagement()
@@ -175,6 +176,15 @@ class LiveKitEventsService:
"Invalid LIVEKIT_WEBHOOK_EVENTS_FILTER_REGEX. Webhook filtering disabled."
)
@property
def cluster(self) -> LiveKitClusterConfig:
"""Return the LiveKit cluster that signed the webhook received.
It is the default cluster until a webhook is received, as when a
handler is called directly.
"""
return self.token_verifier.cluster or get_default_livekit_cluster()
def receive(self, request):
"""Process webhook and route to appropriate handler."""
@@ -183,7 +193,7 @@ class LiveKitEventsService:
raise AuthenticationError("Authorization header missing")
try:
data = self.webhook_receiver.receive(
data = api.WebhookReceiver(self.token_verifier).receive(
request.body.decode("utf-8"), auth_token
)
except Exception as e:
@@ -213,7 +223,7 @@ class LiveKitEventsService:
return
# Handle according to received webhook type
handler = self._webhook_handlers.get(webhook_type.value)
handler = self._webhook_handlers.get(webhook_type)
if handler is not None:
handler(data)
@@ -240,6 +250,10 @@ class LiveKitEventsService:
Egress ended is sent with one of these statuses:
EGRESS_COMPLETE, EGRESS_FAILED, EGRESS_ABORTED, EGRESS_LIMIT_REACHED
The room's metadata is cleaned up, the metadata collector's dispatch
deleted and participants notified on the LiveKit cluster recording the
room, even once the room was placed elsewhere.
"""
# Fetch recording
@@ -268,7 +282,9 @@ class LiveKitEventsService:
try:
room_name = str(recording.room.id)
RoomManagement.update_metadata(
room_name, remove_keys=["recording_mode", "recording_status"]
room_name,
remove_keys=["recording_mode", "recording_status"],
cluster=recording.get_livekit_cluster(),
)
except RoomNotFoundException:
logger.info(
@@ -354,12 +370,12 @@ class LiveKitEventsService:
f"Failed to create sip dispatch rule for room {room_id}"
) from e
@staticmethod
def _close_deleted_room(room_id):
def _close_deleted_room(self, room_id):
"""Close a LiveKit room recreated after its room was soft deleted.
LiveKit auto-creates a room on join, so a participant still holding a
valid token can bring a deleted room back to life until the token expires.
The LiveKit room is closed on the LiveKit cluster it started on.
"""
logger.warning(
@@ -367,14 +383,18 @@ class LiveKitEventsService:
)
try:
RoomManagement.delete_room(str(room_id))
RoomManagement.delete_room(str(room_id), cluster=self.cluster)
except RoomNotFoundException:
logger.info("LiveKit room %s is already closed", room_id)
except RoomManagementException as e:
raise ActionFailedError(f"Failed to close deleted room {room_id}") from e
def _handle_room_finished(self, data):
"""Handle 'room_finished' event."""
"""Handle 'room_finished' event.
The room is cleaned up only when the placement of its LiveKit room
ended, or when it has none.
"""
try:
room_id = uuid.UUID(data.room.name)
@@ -385,6 +405,9 @@ class LiveKitEventsService:
)
raise ActionFailedError("Failed to process room finished event") from e
if not self._end_placement(data.room):
return
if settings.ROOM_TELEPHONY_ENABLED or settings.ROOMKIT_ENABLED:
try:
self.sip_management.delete_dispatch_rule(room_id)
@@ -402,6 +425,57 @@ class LiveKitEventsService:
f"Failed to clear room cache for room {room_id}"
) from e
def _end_placement(self, livekit_room) -> bool:
"""Delete the placement of a LiveKit room that finished.
A room_finished event is about one LiveKit room, one SID on one cluster,
while the cleanup of its room (SIP dispatch rule, presence and lobby
caches) is keyed by the room's id: it wipes state the room keeps from one
LiveKit room to the next, whether created again on the same cluster with
a new SID or relocated to another cluster. The placement tells which
LiveKit room is the current one.
The placement is deleted only when it is on the cluster that sent the
webhook, with the SID of the finished LiveKit room.
- When nothing is deleted, the placement is read again to tell why.
- Without placement, as for a LiveKit room live since before placements
existed or a webhook delivered twice, the meeting is over and the room
is cleaned up, or lobby admissions would outlive it.
- With a placement on another cluster or with another SID, the room has
moved on to another LiveKit room, relocated or created again since:
cleaning up would erase the admissions of participants admitted from
the lobby, who would be queued again, and delete the SIP dispatch rule
the live LiveKit room uses.
Return whether the room is to be cleaned up.
"""
if not livekit_room.sid:
return True
cluster_name = self.cluster.name
placements = models.Placement.objects.filter(room_name=livekit_room.name)
deleted, _ = placements.filter(
cluster=cluster_name, sid=livekit_room.sid
).delete()
if deleted:
return True
placement = placements.first()
if placement is None:
return True
logger.info(
"LiveKit room %s finished on LiveKit cluster %s with SID %s, but is "
"placed on LiveKit cluster %s with SID %r: ignoring the event",
livekit_room.name,
cluster_name,
livekit_room.sid,
placement.cluster,
placement.sid,
)
return False
def _handle_participant_left(self, data):
"""Handle 'participant_left': invalidate the presence cache.
+9
View File
@@ -12,6 +12,7 @@ from django.core.cache import cache
from django.utils import timezone
from core import models, utils
from core.livekit.placements import PlacementService
logger = logging.getLogger(__name__)
@@ -176,6 +177,10 @@ class LobbyService:
3. If unknown, add to waiting list
4. If accepted, generate LiveKit config
5. If denied, do nothing.
The LiveKit config is for the LiveKit cluster the LiveKit room is
placed on, where it is created first. When it cannot be,
LiveKitRoomUnavailableError is raised, and the lobby left as it was.
"""
participant = None
@@ -190,6 +195,7 @@ class LobbyService:
user_role = room.get_role(user)
if self.can_bypass_lobby(room=room, user=user, role=user_role):
cluster = PlacementService.place(room_id, room)
if not is_new_participant:
self.clear_participant_cache(room.id, participant.id)
participant.status = LobbyParticipantStatus.ACCEPTED
@@ -202,6 +208,7 @@ class LobbyService:
configuration=room.configuration,
participant_id=participant.id,
role=user_role,
cluster=cluster,
)
return participant, livekit_config
@@ -215,6 +222,7 @@ class LobbyService:
self.refresh_waiting_status(room.id, participant.id)
elif participant.status == LobbyParticipantStatus.ACCEPTED:
cluster = PlacementService.place(room_id, room)
# wrongly named, contains access token to join a room
livekit_config = utils.generate_livekit_config(
room_id=room_id,
@@ -224,6 +232,7 @@ class LobbyService:
configuration=room.configuration,
participant_id=participant.id,
role=user_role,
cluster=cluster,
)
return participant, livekit_config
@@ -19,6 +19,8 @@ from livekit.api import (
from livekit.protocol.models import ParticipantInfo
from core import utils
from core.livekit.clusters import LiveKitClusterConfig
from core.livekit.placements import PlacementService
from .lobby import LobbyService
from .presence import PresenceCache
@@ -37,11 +39,31 @@ class ParticipantNotFoundException(ParticipantsManagementException):
class ParticipantsManagement:
"""Service for managing participants."""
@async_to_sync
async def mute(self, room_name: str, identity: str, track_sid: str):
def mute(
self,
room_name: str,
identity: str,
track_sid: str,
*,
cluster: Optional[LiveKitClusterConfig] = None,
):
"""Mute a specific audio or video track for a participant in a room."""
lkapi = utils.create_livekit_client()
cluster = cluster or PlacementService.cluster_for(room_name)
self._mute(room_name, identity, track_sid, cluster=cluster)
@async_to_sync
async def _mute(
self,
room_name: str,
identity: str,
track_sid: str,
*,
cluster: LiveKitClusterConfig,
):
"""Mute a participant's track in a LiveKit room on a LiveKit cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
await lkapi.room.mute_published_track(
@@ -72,10 +94,24 @@ class ParticipantsManagement:
finally:
await lkapi.aclose()
@async_to_sync
async def remove(self, room_name: str, identity: str):
def remove(
self,
room_name: str,
identity: str,
*,
cluster: Optional[LiveKitClusterConfig] = None,
):
"""Remove a participant from a room and clear their lobby/presence cache."""
cluster = cluster or PlacementService.cluster_for(room_name)
self._remove(room_name, identity, cluster=cluster)
@async_to_sync
async def _remove(
self, room_name: str, identity: str, *, cluster: LiveKitClusterConfig
):
"""Remove a participant from a LiveKit room on a LiveKit cluster."""
PresenceCache().clear(room_name, identity)
try:
@@ -90,7 +126,7 @@ class ParticipantsManagement:
exc_info=exc,
)
lkapi = utils.create_livekit_client()
lkapi = utils.create_livekit_client(cluster=cluster)
try:
await lkapi.room.remove_participant(
@@ -115,8 +151,7 @@ class ParticipantsManagement:
finally:
await lkapi.aclose()
@async_to_sync
async def update( # noqa: PLR0917
def update( # noqa: PLR0917
self,
room_name: str,
identity: str,
@@ -124,10 +159,37 @@ class ParticipantsManagement:
attributes: Optional[Dict] = None,
permission: Optional[Dict] = None,
name: Optional[str] = None,
*,
cluster: Optional[LiveKitClusterConfig] = None,
):
"""Update participant properties such as metadata, attributes, permissions, or name."""
lkapi = utils.create_livekit_client()
cluster = cluster or PlacementService.cluster_for(room_name)
self._update(
room_name,
identity,
cluster=cluster,
metadata=metadata,
attributes=attributes,
permission=permission,
name=name,
)
@async_to_sync
async def _update(
self,
room_name: str,
identity: str,
*,
cluster: LiveKitClusterConfig,
metadata: Optional[Dict],
attributes: Optional[Dict],
permission: Optional[Dict],
name: Optional[str],
):
"""Update a participant of a LiveKit room on a LiveKit cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
await lkapi.room.update_participant(
@@ -184,8 +246,13 @@ class ParticipantsManagement:
return present
@async_to_sync
async def check_if_in_meeting(self, room_name: str, identity: str) -> bool:
def check_if_in_meeting(
self,
room_name: str,
identity: str,
*,
cluster: Optional[LiveKitClusterConfig] = None,
) -> bool:
"""Check whether `identity` is currently a participant in `room_name`.
Raises ParticipantsManagementException for unexpected LiveKit errors
@@ -195,7 +262,16 @@ class ParticipantsManagement:
if not room_name or not identity:
return False
lkapi = utils.create_livekit_client()
cluster = cluster or PlacementService.cluster_for(room_name)
return self._check_if_in_meeting(room_name, identity, cluster=cluster)
@async_to_sync
async def _check_if_in_meeting(
self, room_name: str, identity: str, *, cluster: LiveKitClusterConfig
) -> bool:
"""Check whether `identity` is a participant of a LiveKit room on a cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
participant = await lkapi.room.get_participant(
+38 -7
View File
@@ -19,6 +19,8 @@ from livekit.api import (
)
from core import utils
from core.livekit.clusters import LiveKitClusterConfig
from core.livekit.placements import PlacementService
from core.models import Room
from .sip_management import SIPException, SIPManagement
@@ -38,24 +40,41 @@ class RoomManagement:
"""Service for managing LiveKit rooms."""
@classmethod
@async_to_sync
async def update_metadata(
def update_metadata(
cls,
room_name: str,
metadata: Optional[Dict] = None,
remove_keys: Optional[list[str]] = None,
*,
cluster: Optional[LiveKitClusterConfig] = None,
):
"""Merge values into a LiveKit room's metadata.
The `room_name` corresponds to the LiveKit room identifier
(i.e. the Room model's UUID as a string).
(i.e. the Room model's UUID as a string). The LiveKit room is
updated on `cluster`, by default the LiveKit cluster it is placed on.
Raises:
RoomNotFoundException: the room does not exist in LiveKit.
RoomManagementException: the metadata update otherwise fails.
"""
lkapi = utils.create_livekit_client()
cluster = cluster or PlacementService.cluster_for(room_name)
cls._update_metadata(room_name, metadata, remove_keys, cluster=cluster)
@classmethod
@async_to_sync
async def _update_metadata(
cls,
room_name: str,
metadata: Optional[Dict],
remove_keys: Optional[list[str]],
*,
cluster: LiveKitClusterConfig,
):
"""Merge values into a LiveKit room's metadata on a LiveKit cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
response = await lkapi.room.list_rooms(ListRoomsRequest(names=[room_name]))
@@ -95,16 +114,28 @@ class RoomManagement:
await lkapi.aclose()
@classmethod
@async_to_sync
async def delete_room(cls, room_name: str):
def delete_room(
cls, room_name: str, *, cluster: Optional[LiveKitClusterConfig] = None
):
"""Delete a LiveKit room and disconnect all participants.
The LiveKit room is deleted on `cluster`, by default the LiveKit
cluster it is placed on.
Raises:
RoomNotFoundException: the room does not exist in LiveKit.
RoomManagementException: the deletion otherwise fails.
"""
lkapi = utils.create_livekit_client()
cluster = cluster or PlacementService.cluster_for(room_name)
cls._delete_room(room_name, cluster=cluster)
@classmethod
@async_to_sync
async def _delete_room(cls, room_name: str, *, cluster: LiveKitClusterConfig):
"""Delete a LiveKit room on a LiveKit cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
await lkapi.room.delete_room(DeleteRoomRequest(room=room_name))
+17 -4
View File
@@ -1,6 +1,7 @@
"""Service for managing subtitle agents in LiveKit rooms."""
from logging import getLogger
from typing import Optional
from django.conf import settings
@@ -8,6 +9,8 @@ from asgiref.sync import async_to_sync
from livekit.protocol.agent_dispatch import CreateAgentDispatchRequest
from core import utils
from core.livekit.clusters import LiveKitClusterConfig
from core.livekit.placements import PlacementService
logger = getLogger(__name__)
@@ -19,11 +22,21 @@ class SubtitleException(Exception):
class SubtitleService:
"""Service for managing subtitle agents in LiveKit rooms."""
@async_to_sync
async def start_subtitle(self, room):
"""Start subtitle agent for the specified room."""
def start_subtitle(self, room, *, cluster: Optional[LiveKitClusterConfig] = None):
"""Start subtitle agent for the specified room.
lkapi = utils.create_livekit_client()
The agent is dispatched on `cluster`, by default the LiveKit cluster the
room's LiveKit room is placed on.
"""
cluster = cluster or PlacementService.cluster_for(str(room.id))
self._start_subtitle(room, cluster=cluster)
@async_to_sync
async def _start_subtitle(self, room, *, cluster: LiveKitClusterConfig):
"""Dispatch the subtitle agent to a room's LiveKit room on a cluster."""
lkapi = utils.create_livekit_client(cluster=cluster)
try:
# Transcriber agent prevents duplicate subtitle agents per room
+25 -2
View File
@@ -1,9 +1,12 @@
"""Tasks related to connection test rooms."""
import logging
from typing import Optional
from django.conf import settings
from core.livekit.clusters import get_default_livekit_cluster, get_livekit_cluster
from core.livekit.exceptions import UnknownLiveKitClusterError
from core.services.room_management import (
RoomManagement,
RoomManagementException,
@@ -15,11 +18,16 @@ logger = logging.getLogger(__name__)
@task
def delete_connection_test_room(room_name: str):
def delete_connection_test_room(room_name: str, cluster_name: Optional[str] = None):
"""Force-delete an ephemeral connection-test room.
Used as a hard cap so a participant cannot keep an auto-refreshed
LiveKit session open indefinitely after requesting a test token.
The room is deleted on the LiveKit cluster named `cluster_name`, where it
was created. Without a cluster name, as in tasks queued before connection
test rooms were created on a chosen cluster, and when the cluster is no
longer registered, it is deleted on the default cluster.
"""
prefix = settings.CONNECTION_TEST_ROOM_PREFIX
if not room_name.startswith(prefix):
@@ -30,8 +38,23 @@ def delete_connection_test_room(room_name: str):
)
return
if cluster_name is None:
cluster = get_default_livekit_cluster()
else:
try:
cluster = get_livekit_cluster(cluster_name)
except UnknownLiveKitClusterError:
cluster = get_default_livekit_cluster()
logger.warning(
"Connection test room '%s' was created on LiveKit cluster %s, "
"which is not registered: deleting it on the default cluster %s.",
room_name,
cluster_name,
cluster.name,
)
try:
RoomManagement.delete_room(room_name)
RoomManagement.delete_room(room_name, cluster=cluster)
except RoomNotFoundException:
# Room may already be gone after empty/departure timeout.
logger.info("Connection test room '%s' already gone.", room_name)
@@ -0,0 +1,32 @@
"""Unit tests for the LiveKit token authentication."""
import pytest
from livekit.api.access_token import Claims
from rest_framework.test import APIRequestFactory
from core.authentication.livekit import LiveKitTokenAuthentication
from core.factories import UserFactory
from core.utils import generate_token
pytestmark = pytest.mark.django_db
def test_authenticate_token_from_another_cluster(two_livekit_clusters):
"""A token signed by any registered LiveKit cluster authenticates its user.
The request's auth stays the token's claims, which carry that cluster.
"""
_paris, lyon = two_livekit_clusters
user = UserFactory()
token = generate_token(room="my-room", user=user, cluster=lyon)
request = APIRequestFactory().post(
"/", HTTP_AUTHORIZATION=f"X-LiveKit-Token {token}"
)
authenticated_user, claims = LiveKitTokenAuthentication().authenticate(request)
assert authenticated_user == user
assert isinstance(claims, Claims)
assert claims.identity == str(user.sub)
assert claims.video.room == "my-room"
assert claims.cluster == lyon
+46
View File
@@ -1,17 +1,30 @@
"""Fixtures for tests in the Meet core application"""
import json
from unittest import mock
import pytest
from dockerflow.logging import request_id_context
from core.audit.testing import capture_audit
from core.livekit.clusters import get_livekit_clusters
from core.livekit.placements import PlacementService
from core.tests.utils import CREATED_LIVEKIT_ROOM_SID, livekit_cluster_entry
USER = "user"
TEAM = "team"
VIA = [USER, TEAM]
def pytest_configure(config):
"""Register the markers of the tests in the Meet core application."""
config.addinivalue_line(
"markers",
"real_livekit_room_creation: create LiveKit rooms through the LiveKit "
"API rather than the fake_livekit_room_creation fixture",
)
@pytest.fixture
def mock_user_get_teams():
"""Mock for the "get_teams" method on the User model."""
@@ -26,6 +39,27 @@ def audit_events():
yield events
@pytest.fixture(autouse=True)
def fake_livekit_room_creation(request):
"""Create LiveKit rooms without calling LiveKit.
Credentials are handed out only once the LiveKit room is created on its
cluster, so every join would otherwise call LiveKit. Each LiveKit room
created gets CREATED_LIVEKIT_ROOM_SID as SID. Return the mock standing for
PlacementService.create_livekit_room, or None in tests marked
real_livekit_room_creation, which go through the real one.
"""
if request.node.get_closest_marker("real_livekit_room_creation"):
yield None
return
with mock.patch.object(
PlacementService,
"create_livekit_room",
return_value=CREATED_LIVEKIT_ROOM_SID,
) as create_livekit_room:
yield create_livekit_room
@pytest.fixture(autouse=True)
def isolated_request_id():
"""Keep dockerflow's request id from leaking from one test to the next.
@@ -37,3 +71,15 @@ def isolated_request_id():
token = request_id_context.set(None)
yield
request_id_context.reset(token)
@pytest.fixture
def two_livekit_clusters(settings):
"""Register the Paris and Lyon LiveKit clusters, Paris being the default.
Each has its own API key and secret. Return both clusters, Paris first.
"""
settings.LIVEKIT_CLUSTERS = json.dumps(
[livekit_cluster_entry("paris"), livekit_cluster_entry("lyon")]
)
return get_livekit_clusters()
@@ -0,0 +1,260 @@
"""Unit tests for the registry of LiveKit clusters."""
import dataclasses
import json
import time
from datetime import timedelta
import jwt
import pytest
from livekit.api import AccessToken, VideoGrants
from livekit.api.access_token import Claims
from core.livekit.clusters import (
LiveKitClusterConfig,
LiveKitClusterTokenVerifier,
get_default_livekit_cluster,
get_livekit_cluster,
get_livekit_cluster_by_api_key,
get_livekit_clusters,
)
from core.livekit.exceptions import UnknownLiveKitClusterError
from core.tests.utils import livekit_cluster_entry
PARIS = LiveKitClusterConfig(
name="paris",
url="wss://paris.livekit.test",
api_url="wss://paris.livekit.test",
api_key="key-paris",
api_secret="secret-paris-padded-for-minimum-len!-livekit",
)
LYON = LiveKitClusterConfig(
name="lyon",
url="wss://lyon.livekit.test",
api_url="http://livekit.lyon.svc:7880",
api_key="key-lyon",
api_secret="secret-lyon-padded-for-minimum-len!-livekit",
weight=0,
)
@pytest.fixture
def two_clusters(settings):
"""Register the Paris and Lyon LiveKit clusters, Paris being the default."""
settings.LIVEKIT_CLUSTERS = json.dumps(
[
livekit_cluster_entry("paris"),
livekit_cluster_entry(
"lyon",
api_url="http://livekit.lyon.svc:7880",
weight=0,
),
]
)
def test_get_livekit_clusters_from_livekit_configuration(settings):
"""Without LIVEKIT_CLUSTERS, LIVEKIT_CONFIGURATION describes a single cluster.
It is named "default" and its URL serves both browsers and Django.
"""
configuration = settings.LIVEKIT_CONFIGURATION
assert settings.LIVEKIT_CLUSTERS is None
assert get_livekit_clusters() == (
LiveKitClusterConfig(
name="default",
url=configuration["url"],
api_url=configuration["url"],
api_key=configuration["api_key"],
api_secret=configuration["api_secret"],
weight=1,
),
)
@pytest.mark.usefixtures("two_clusters")
def test_get_livekit_clusters_from_livekit_clusters():
"""LIVEKIT_CLUSTERS lists the clusters in order, api_url defaulting to url."""
assert get_livekit_clusters() == (PARIS, LYON)
def test_get_livekit_clusters_follows_settings_changes(settings):
"""The clusters are cached on the raw settings, so a change applies at once."""
clusters = get_livekit_clusters()
assert get_livekit_clusters() is clusters
settings.LIVEKIT_CONFIGURATION = {
**settings.LIVEKIT_CONFIGURATION,
"url": "wss://elsewhere.livekit.test",
}
assert get_livekit_clusters()[0].url == "wss://elsewhere.livekit.test"
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("paris")])
assert get_livekit_clusters() == (PARIS,)
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("lille")])
assert [cluster.name for cluster in get_livekit_clusters()] == ["lille"]
def test_get_livekit_clusters_invalid_livekit_clusters(settings):
"""An invalid LIVEKIT_CLUSTERS is refused at runtime as it is at startup."""
settings.LIVEKIT_CLUSTERS = "[]"
with pytest.raises(ValueError, match="LIVEKIT_CLUSTERS"):
get_livekit_clusters()
@pytest.mark.usefixtures("two_clusters")
def test_get_default_livekit_cluster():
"""The default cluster is the first one listed."""
assert get_default_livekit_cluster() == PARIS
def test_get_default_livekit_cluster_from_livekit_configuration():
"""Without LIVEKIT_CLUSTERS, the default cluster is the one named "default"."""
assert get_default_livekit_cluster() == get_livekit_clusters()[0]
assert get_default_livekit_cluster().name == "default"
@pytest.mark.usefixtures("two_clusters")
def test_get_livekit_cluster():
"""A cluster is found by its name."""
assert get_livekit_cluster("lyon") == LYON
assert get_livekit_cluster("paris") == PARIS
@pytest.mark.usefixtures("two_clusters")
def test_get_livekit_cluster_unknown():
"""An unknown name raises an error naming it."""
with pytest.raises(
UnknownLiveKitClusterError, match="No LiveKit cluster is named 'default'."
):
get_livekit_cluster("default")
@pytest.mark.usefixtures("two_clusters")
def test_get_livekit_cluster_by_api_key():
"""A cluster is found by its API key."""
assert get_livekit_cluster_by_api_key("key-lyon") == LYON
assert get_livekit_cluster_by_api_key("key-paris") == PARIS
@pytest.mark.usefixtures("two_clusters")
@pytest.mark.parametrize("api_key", ["key-lille", "KEY-LYON", "", None])
def test_get_livekit_cluster_by_api_key_unknown(api_key):
"""An API key no cluster uses finds none."""
assert get_livekit_cluster_by_api_key(api_key) is None
@pytest.mark.parametrize("api_key", ["", None])
def test_get_livekit_cluster_by_api_key_unset(settings, api_key):
"""A missing API key finds no cluster, not even one whose key is unset."""
settings.LIVEKIT_CONFIGURATION = {
"url": None,
"api_key": api_key,
"api_secret": None,
}
assert get_livekit_clusters()[0].api_key == api_key
assert get_livekit_cluster_by_api_key(api_key) is None
def test_livekit_cluster_config_hides_the_api_secret():
"""The API secret stays out of the representation, which may end up in logs."""
assert PARIS.api_secret not in repr(PARIS)
assert PARIS.api_secret not in str(PARIS)
assert "key-paris" in repr(PARIS)
def test_livekit_cluster_config_is_frozen():
"""A cluster is shared between callers, so it cannot be changed."""
with pytest.raises(dataclasses.FrozenInstanceError):
PARIS.url = "wss://elsewhere.livekit.test"
def livekit_token(api_key, api_secret, ttl_minutes=60):
"""Return a participant token signed with this API key and secret."""
return (
AccessToken(api_key=api_key, api_secret=api_secret)
.with_identity("participant-1")
.with_grants(VideoGrants(room="my-room", room_join=True))
.with_ttl(timedelta(minutes=ttl_minutes))
.to_jwt()
)
@pytest.mark.usefixtures("two_clusters")
@pytest.mark.parametrize("cluster", [PARIS, LYON], ids=["paris", "lyon"])
def test_livekit_cluster_token_verifier(cluster):
"""A token signed by any registered cluster is verified with its keys.
The claims carry that cluster, and so does the verifier.
"""
verifier = LiveKitClusterTokenVerifier()
claims = verifier.verify(livekit_token(cluster.api_key, cluster.api_secret))
assert isinstance(claims, Claims)
assert claims.identity == "participant-1"
assert claims.video.room == "my-room"
assert claims.cluster == cluster
assert verifier.cluster == cluster
def test_livekit_cluster_token_verifier_from_livekit_configuration(settings):
"""Without LIVEKIT_CLUSTERS, tokens are verified with LIVEKIT_CONFIGURATION."""
configuration = settings.LIVEKIT_CONFIGURATION
claims = LiveKitClusterTokenVerifier().verify(
livekit_token(configuration["api_key"], configuration["api_secret"])
)
assert claims.cluster == get_default_livekit_cluster()
assert claims.cluster.name == "default"
@pytest.mark.usefixtures("two_clusters")
def test_livekit_cluster_token_verifier_unknown_api_key():
"""A token whose API key no cluster uses is refused as having a wrong issuer."""
verifier = LiveKitClusterTokenVerifier()
token = livekit_token("key-lille", "secret-lille-padded-for-minimum-len!-livekit")
with pytest.raises(jwt.InvalidIssuerError, match="Invalid issuer"):
verifier.verify(token)
assert verifier.cluster is None
@pytest.mark.usefixtures("two_clusters")
def test_livekit_cluster_token_verifier_missing_issuer():
"""A token without issuer names no cluster and is refused."""
token = jwt.encode(
{"sub": "participant-1", "exp": int(time.time()) + 3600},
PARIS.api_secret,
algorithm="HS256",
)
with pytest.raises(jwt.MissingRequiredClaimError, match='"iss"'):
LiveKitClusterTokenVerifier().verify(token)
@pytest.mark.usefixtures("two_clusters")
def test_livekit_cluster_token_verifier_signed_by_another_cluster():
"""A token issued with the API key of a cluster must be signed with its secret.
A cluster cannot sign tokens in the name of another one.
"""
token = livekit_token(LYON.api_key, PARIS.api_secret)
with pytest.raises(jwt.InvalidSignatureError):
LiveKitClusterTokenVerifier().verify(token)
@pytest.mark.usefixtures("two_clusters")
def test_livekit_cluster_token_verifier_expired():
"""A token from a registered cluster is still checked as any token is."""
token = livekit_token(LYON.api_key, LYON.api_secret, ttl_minutes=-5)
with pytest.raises(jwt.ExpiredSignatureError):
LiveKitClusterTokenVerifier().verify(token)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,29 @@
"""Tests that the conditional deletes of placements stay a single statement."""
from django.db import router
from django.db.models import signals
from django.db.models.deletion import Collector
from core.models import Placement
def test_placement_deletes_are_a_single_statement():
"""Deleting placements filtered on their epoch and SID must issue one DELETE
statement, which Django does only while nothing listens for the deletion
of a placement and no relation points at it."""
collector = Collector(using=router.db_for_write(Placement))
assert collector.can_fast_delete(Placement.objects.all()), (
"The conditional deletes of placements (place(), the operations, the "
"reconciliation, the room_finished webhook) must stay a single "
"DELETE ... WHERE on epoch and SID, or a join can delete a placement "
"whose SID another join just recorded: no relation may point at "
"Placement"
)
for signal in (signals.pre_delete, signals.post_delete):
assert not signal.has_listeners(Placement), (
"No receiver may listen for the deletion of a placement: the "
"conditional deletes of placements must stay a single "
"DELETE ... WHERE on epoch and SID, or a join can delete a "
"placement whose SID another join just recorded"
)
@@ -0,0 +1,313 @@
"""Unit tests for the parsing of the LIVEKIT_CLUSTERS setting."""
import json
import re
import pytest
from core.livekit.specs import parse_livekit_clusters
from core.tests.utils import livekit_cluster_entry
SECRET = "s3cr3t-never-shown"
def test_parse_livekit_clusters_reads_every_field():
"""Every field of an entry is kept, the API secret hidden from the repr."""
[spec] = parse_livekit_clusters(
json.dumps(
[
{
"name": "paris-1",
"url": "wss://paris-1.livekit.test",
"api_url": "http://livekit.paris-1.svc:7880",
"api_key": "key-paris-1",
"api_secret": SECRET,
"weight": 0,
}
]
)
)
assert spec.name == "paris-1"
assert spec.url == "wss://paris-1.livekit.test"
assert spec.api_url == "http://livekit.paris-1.svc:7880"
assert spec.api_key == "key-paris-1"
assert spec.api_secret.get_secret_value() == SECRET
assert spec.weight == 0
assert SECRET not in repr(spec)
def test_parse_livekit_clusters_optional_fields():
"""The API URL and the weight may be left out."""
[spec] = parse_livekit_clusters(json.dumps([livekit_cluster_entry("paris")]))
assert spec.api_url is None
assert spec.weight == 1
def test_parse_livekit_clusters_keeps_their_order():
"""The clusters keep the order they are listed in, the default one first."""
specs = parse_livekit_clusters(
json.dumps([livekit_cluster_entry(name) for name in ("lyon", "paris", "lille")])
)
assert [spec.name for spec in specs] == ["lyon", "paris", "lille"]
@pytest.mark.parametrize("scheme", ["http", "https", "ws", "wss"])
def test_parse_livekit_clusters_keeps_urls_as_written(scheme):
"""
URLs of the four schemes LiveKit serves are accepted and left untouched.
"""
[spec] = parse_livekit_clusters(
json.dumps(
[
livekit_cluster_entry(
"paris",
url=f"{scheme}://paris.livekit.test",
api_url=f"{scheme}://10.0.0.1:7880",
)
]
)
)
assert spec.url == f"{scheme}://paris.livekit.test"
assert spec.api_url == f"{scheme}://10.0.0.1:7880"
def test_parse_livekit_clusters_accepts_upper_case_schemes():
"""The scheme is not case sensitive, and the URL is kept as written."""
[spec] = parse_livekit_clusters(
json.dumps([livekit_cluster_entry("paris", url="HTTPS://paris.livekit.test")])
)
assert spec.url == "HTTPS://paris.livekit.test"
def test_parse_livekit_clusters_accepts_international_urls():
"""URLs may hold characters beyond ASCII, such as international domain names."""
[spec] = parse_livekit_clusters(
json.dumps([livekit_cluster_entry("paris", url="wss://visio.exämple.fr/é")])
)
assert spec.url == "wss://visio.exämple.fr/é"
@pytest.mark.parametrize(
"raw,problem",
[
("", "LIVEKIT_CLUSTERS: Invalid JSON"),
('{"name": "paris"}', "LIVEKIT_CLUSTERS: Input should be a valid array"),
("[]", "LIVEKIT_CLUSTERS: List should have at least 1 item"),
('["paris"]', "LIVEKIT_CLUSTERS[0]: Input should be an object"),
],
)
def test_parse_livekit_clusters_rejects_invalid_lists(raw, problem):
"""The value must be a non-empty JSON list of objects."""
with pytest.raises(ValueError, match=re.escape(problem)):
parse_livekit_clusters(raw)
NAME_PATTERN_PROBLEM = (
"LIVEKIT_CLUSTERS[1].name: String should match pattern '^[a-z0-9-]{1,63}$'"
)
URL_SCHEME_PROBLEM = "URL scheme should be 'http', 'https', 'ws' or 'wss'"
URL_AUTHORITY_PROBLEM = "URL should be written scheme://host"
URL_CHARACTERS_PROBLEM = "URL should not contain whitespace or non-printable characters"
@pytest.mark.parametrize(
"changes,problem",
[
({"name": "Paris"}, NAME_PATTERN_PROBLEM),
({"name": "paris_1"}, NAME_PATTERN_PROBLEM),
({"name": ""}, NAME_PATTERN_PROBLEM),
({"name": "p" * 64}, NAME_PATTERN_PROBLEM),
({"name": "paris\n"}, NAME_PATTERN_PROBLEM),
(
{"url": "ftp://paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_SCHEME_PROBLEM}",
),
(
{"url": "paris.livekit.test"},
"LIVEKIT_CLUSTERS[1].url: Input should be a valid URL",
),
(
{"url": "https:/paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_AUTHORITY_PROBLEM}",
),
(
{"url": "wss:\\\\paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_AUTHORITY_PROBLEM}",
),
(
{"url": "https:paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_AUTHORITY_PROBLEM}",
),
(
{"api_url": "https:///paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].api_url: {URL_AUTHORITY_PROBLEM}",
),
(
{"url": "wss://paris.livekit.test "},
f"LIVEKIT_CLUSTERS[1].url: {URL_CHARACTERS_PROBLEM}",
),
(
{"url": "wss://paris.live\tkit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_CHARACTERS_PROBLEM}",
),
(
{"url": "wss://paris.\u200blivekit.test"},
f"LIVEKIT_CLUSTERS[1].url: {URL_CHARACTERS_PROBLEM}",
),
(
{"api_url": "redis://paris.livekit.test"},
f"LIVEKIT_CLUSTERS[1].api_url: {URL_SCHEME_PROBLEM}",
),
(
{"api_url": " http://10.0.0.1:7880"},
f"LIVEKIT_CLUSTERS[1].api_url: {URL_CHARACTERS_PROBLEM}",
),
(
{"api_key": ""},
"LIVEKIT_CLUSTERS[1].api_key: String should have at least 1 character",
),
(
{"api_secret": ""},
"LIVEKIT_CLUSTERS[1].api_secret: Value should have at least 1 item",
),
(
{"region": "eu-west-3"},
"LIVEKIT_CLUSTERS[1].region: Extra inputs are not permitted",
),
(
{"weight": -1},
"LIVEKIT_CLUSTERS[1].weight: Input should be greater than or equal to 0",
),
(
{"weight": "2"},
"LIVEKIT_CLUSTERS[1].weight: Input should be a valid integer",
),
(
{"weight": True},
"LIVEKIT_CLUSTERS[1].weight: Input should be a valid integer",
),
(
{"weight": 1.5},
"LIVEKIT_CLUSTERS[1].weight: Input should be a valid integer",
),
(
{"redis_url": "redis://paris.livekit.test"},
"LIVEKIT_CLUSTERS[1].redis_url: Extra inputs are not permitted",
),
],
)
def test_parse_livekit_clusters_rejects_invalid_entries(changes, problem):
"""Each field is checked, values are not coerced and unknown fields are refused."""
raw = json.dumps(
[livekit_cluster_entry("lyon"), {**livekit_cluster_entry("paris"), **changes}]
)
with pytest.raises(ValueError, match=re.escape(problem)):
parse_livekit_clusters(raw)
@pytest.mark.parametrize("field", ["name", "url", "api_key", "api_secret"])
def test_parse_livekit_clusters_requires_fields(field):
"""The name, the URL and the API key pair are required."""
entry = livekit_cluster_entry("paris")
del entry[field]
with pytest.raises(
ValueError, match=re.escape(f"LIVEKIT_CLUSTERS[0].{field}: Field required")
):
parse_livekit_clusters(json.dumps([entry]))
def test_parse_livekit_clusters_reports_every_problem():
"""All the problems found are reported at once."""
raw = json.dumps(
[
{**livekit_cluster_entry("paris"), "name": "Paris"},
livekit_cluster_entry("lyon", weight=-1),
]
)
with pytest.raises(ValueError) as excinfo:
parse_livekit_clusters(raw)
assert str(excinfo.value) == (
f"{NAME_PATTERN_PROBLEM.replace('[1]', '[0]')}; "
"LIVEKIT_CLUSTERS[1].weight: Input should be greater than or equal to 0"
)
def test_parse_livekit_clusters_rejects_duplicate_names():
"""Two clusters cannot have the same name."""
raw = json.dumps(
[
livekit_cluster_entry("paris"),
livekit_cluster_entry("lyon"),
{**livekit_cluster_entry("lille"), "name": "paris"},
]
)
with pytest.raises(ValueError) as excinfo:
parse_livekit_clusters(raw)
assert str(excinfo.value) == (
"LIVEKIT_CLUSTERS[2].name: 'paris' is already the name of LIVEKIT_CLUSTERS[0]"
)
def test_parse_livekit_clusters_rejects_shared_api_keys():
"""Two clusters cannot share the API key that tells them apart, left unshown."""
raw = json.dumps(
[
livekit_cluster_entry("paris", api_key=SECRET),
livekit_cluster_entry("lyon"),
livekit_cluster_entry("lille", api_key=SECRET),
]
)
with pytest.raises(ValueError) as excinfo:
parse_livekit_clusters(raw)
assert str(excinfo.value) == (
"LIVEKIT_CLUSTERS[2].api_key: LIVEKIT_CLUSTERS[0] already uses this API key, "
"each LiveKit cluster needs its own"
)
@pytest.mark.parametrize(
"raw",
[
f'[{{"api_secret": "{SECRET}"',
json.dumps({"api_secret": SECRET}),
json.dumps([SECRET]),
json.dumps([livekit_cluster_entry("paris", api_secret=[SECRET])]),
json.dumps([livekit_cluster_entry("paris", weight=SECRET)]),
json.dumps([livekit_cluster_entry("paris", api_secret_file=SECRET)]),
json.dumps([livekit_cluster_entry("paris", url=f"wss://{SECRET} ")]),
],
ids=[
"invalid JSON",
"not a list",
"not an object",
"invalid secret",
"secret in another field",
"unknown field",
"secret in a URL with whitespace",
],
)
def test_parse_livekit_clusters_never_shows_the_values_read(raw):
"""Problems name places only, and pydantic's error is not chained.
Each of these values makes pydantic's own message show the secret.
"""
with pytest.raises(ValueError) as excinfo:
parse_livekit_clusters(raw)
assert SECRET not in str(excinfo.value)
assert excinfo.value.__cause__ is None
assert excinfo.value.__context__ is None
@@ -0,0 +1,199 @@
"""Unit tests for the placement strategies picking the LiveKit cluster of new
placements."""
import secrets
from unittest import mock
from django.core.cache import cache
import pytest
from core.livekit.clusters import LiveKitClusterConfig
from core.livekit.strategies import (
LeastPlacementsStrategy,
PlacementContext,
RoundRobinStrategy,
WeightedRandomStrategy,
get_placement_strategy,
)
from core.models import Placement
from meet import settings as meet_settings
CONTEXT = PlacementContext(room_name="my-room")
def livekit_cluster(name, weight=1):
"""Return a LiveKit cluster with this name and weight."""
return LiveKitClusterConfig(
name=name,
url=f"wss://{name}.livekit.test",
api_url=f"wss://{name}.livekit.test",
api_key=f"key-{name}",
api_secret=f"secret-{name}-padded-for-minimum-len!-livekit",
weight=weight,
)
def test_get_placement_strategy_default():
"""By default, LiveKit clusters are picked at random, in proportion to their
weights."""
assert isinstance(get_placement_strategy(), WeightedRandomStrategy)
def test_get_placement_strategy_set(settings):
"""The strategy named by LIVEKIT_PLACEMENT_STRATEGY is looked up on every call."""
settings.LIVEKIT_PLACEMENT_STRATEGY = "core.livekit.strategies.RoundRobinStrategy"
assert isinstance(get_placement_strategy(), RoundRobinStrategy)
settings.LIVEKIT_PLACEMENT_STRATEGY = (
"core.livekit.strategies.LeastPlacementsStrategy"
)
assert isinstance(get_placement_strategy(), LeastPlacementsStrategy)
def test_get_placement_strategy_not_imported_while_settings_load():
"""The settings load without importing the placement strategy.
A strategy may need applications that are not ready while the settings load.
"""
with mock.patch.object(
meet_settings.Test,
"LIVEKIT_PLACEMENT_STRATEGY",
"core.livekit.missing.Strategy",
):
meet_settings.Test.post_setup()
def test_weighted_random_strategy():
"""Each candidate is picked for as many of the equally likely draws as it
weighs."""
candidates = (
livekit_cluster("paris", weight=1),
livekit_cluster("lyon", weight=3),
livekit_cluster("lille", weight=2),
)
with mock.patch(
"core.livekit.strategies.randbelow", side_effect=range(6)
) as randbelow:
picked = [
WeightedRandomStrategy().choose(candidates, CONTEXT).name
for _draw in range(6)
]
assert randbelow.call_args_list == [mock.call(6)] * 6
assert picked == ["paris", "lyon", "lyon", "lyon", "lille", "lille"]
def test_weighted_random_strategy_single_candidate():
"""A single candidate is always picked, whatever its weight."""
lyon = livekit_cluster("lyon", weight=5)
assert WeightedRandomStrategy().choose((lyon,), CONTEXT) == lyon
@pytest.fixture(name="round_robin_clusters")
def round_robin_clusters_fixture():
"""Return three LiveKit clusters of different weights, named for this test
alone.
Turns are counted in the cache, which tests running in parallel share and
which outlives them, so the counters of the first one, two and three of
these clusters are deleted afterwards.
"""
suffix = secrets.token_hex(4)
clusters = tuple(
livekit_cluster(f"{name}-{suffix}", weight=weight)
for name, weight in (("paris", 1), ("lyon", 3), ("lille", 2))
)
yield clusters
cache.delete_many(
[RoundRobinStrategy.counter_key(clusters[:size]) for size in (1, 2, 3)]
)
def test_round_robin_strategy(round_robin_clusters):
"""Candidates are picked in turn, whatever their weights, the turns being
shared by every instance of the strategy."""
paris, lyon, lille = round_robin_clusters
picked = [
RoundRobinStrategy().choose(round_robin_clusters, CONTEXT) for _turn in range(7)
]
assert picked == [paris, lyon, lille, paris, lyon, lille, paris]
def test_round_robin_strategy_turns_per_candidates(round_robin_clusters):
"""Each set of candidates has turns of its own."""
paris, lyon, lille = round_robin_clusters
strategy = RoundRobinStrategy()
assert strategy.choose((paris, lyon, lille), CONTEXT) == paris
assert strategy.choose((paris, lyon), CONTEXT) == paris
assert strategy.choose((paris, lyon, lille), CONTEXT) == lyon
assert strategy.choose((paris, lyon), CONTEXT) == lyon
assert strategy.choose((paris, lyon), CONTEXT) == paris
assert strategy.choose((paris, lyon, lille), CONTEXT) == lille
def test_round_robin_strategy_counter_lost(round_robin_clusters):
"""Should the counter vanish right after its creation, the turn goes to the
first candidate, and turns start over."""
paris, lyon, _lille = round_robin_clusters
strategy = RoundRobinStrategy()
assert strategy.choose(round_robin_clusters, CONTEXT) == paris
assert strategy.choose(round_robin_clusters, CONTEXT) == lyon
key = RoundRobinStrategy.counter_key(round_robin_clusters)
with mock.patch.object(
cache, "add", side_effect=lambda *args, **kwargs: cache.delete(key)
):
assert strategy.choose(round_robin_clusters, CONTEXT) == paris
assert strategy.choose(round_robin_clusters, CONTEXT) == paris
assert strategy.choose(round_robin_clusters, CONTEXT) == lyon
@pytest.mark.django_db
@pytest.mark.parametrize(
("placements", "picked"),
(
({}, "paris"),
({"paris": 1}, "lyon"),
({"paris": 1, "lyon": 2}, "lyon"),
({"paris": 1, "lyon": 3}, "paris"),
({"paris": 1, "lyon": 4}, "paris"),
),
)
def test_least_placements_strategy(placements, picked, django_assert_num_queries):
"""The candidate holding the fewest placements for its weight is picked, the
first one on a tie, with a single query."""
candidates = (
livekit_cluster("paris", weight=1),
livekit_cluster("lyon", weight=3),
)
for cluster, count in placements.items():
for index in range(count):
Placement.objects.create(room_name=f"{cluster}-{index}", cluster=cluster)
with django_assert_num_queries(1):
chosen = LeastPlacementsStrategy().choose(candidates, CONTEXT)
assert chosen.name == picked
@pytest.mark.django_db
def test_least_placements_strategy_tie():
"""On a tie, the candidate registered first is picked."""
paris = livekit_cluster("paris", weight=1)
lyon = livekit_cluster("lyon", weight=2)
Placement.objects.create(room_name="paris-0", cluster="paris")
Placement.objects.create(room_name="lyon-0", cluster="lyon")
Placement.objects.create(room_name="lyon-1", cluster="lyon")
assert LeastPlacementsStrategy().choose((paris, lyon), CONTEXT) == paris
assert LeastPlacementsStrategy().choose((lyon, paris), CONTEXT) == lyon
@@ -12,11 +12,13 @@ from unittest import mock
from django.contrib.sites.models import Site
import pytest
from asgiref.sync import async_to_sync
from celery.exceptions import OperationalError as CeleryOperationalError
from core import factories, models
from core.analytics import UserFeatureFlag
from core.audit.testing import find_events
from core.livekit.clusters import get_default_livekit_cluster
from core.recording.event.notification import NotificationService, notification_service
pytestmark = pytest.mark.django_db
@@ -394,7 +396,9 @@ def test_notify_summary_service_post_args_with_metadata(
mock.call(metadata_filename, expires_in=60 * 60 * 24, override_domain=False),
mock.call(recording.key, expires_in=60 * 60 * 24, override_domain=False),
]
mock_get_recording_timestamps.assert_awaited_once_with("egress-1")
mock_get_recording_timestamps.assert_awaited_once_with(
"egress-1", get_default_livekit_cluster()
)
@mock.patch("core.recording.event.notification.requests.post")
@@ -472,7 +476,9 @@ def test_notify_summary_service_post_args_without_metadata(
mock_generate_download_s3_url.assert_called_once_with(
recording.key, expires_in=60 * 60 * 24, override_domain=False
)
mock_get_recording_timestamps.assert_awaited_once_with(recording.worker_id)
mock_get_recording_timestamps.assert_awaited_once_with(
recording.worker_id, get_default_livekit_cluster()
)
mock_is_feature_flag_enabled.assert_called_once_with(
owner, UserFeatureFlag.TRANSCRIPT_SUMMARY_ENABLED
)
@@ -538,6 +544,70 @@ def test_notify_summary_service_v2_payload_json_serializable_without_timestamps(
json.dumps(payload)
@pytest.mark.parametrize("version", [1, 2])
@mock.patch.object(
NotificationService, "_get_recording_timestamps", new_callable=mock.AsyncMock
)
def test_notify_summary_service_timestamps_on_the_recording_cluster(
mock_get_recording_timestamps, version, settings, two_livekit_clusters
):
"""The recording's egress is looked up on the LiveKit cluster that recorded
the room, even once the room was placed elsewhere."""
settings.SUMMARY_SERVICE_VERSION = version
settings.SUMMARY_SERVICE_ENDPOINT = "https://summary.test/api/v2/tasks"
settings.SUMMARY_SERVICE_API_TOKEN = "summary-token"
_paris, lyon = two_livekit_clusters
recording = factories.RecordingFactory(
mode="transcript", worker_id="egress-1", livekit_cluster="lyon"
)
models.Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
factories.UserRecordingAccessFactory(
recording=recording, role=models.RoleChoices.OWNER
)
mock_get_recording_timestamps.return_value = (None, None)
with (
mock.patch("core.recording.event.notification.requests.post") as mock_post,
mock.patch(
"core.recording.event.notification.generate_download_s3_url",
return_value="https://storage.test/recording.ogg",
),
):
mock_post.return_value.json.return_value = {"job_id": "job-8"}
assert NotificationService._notify_summary_service(recording) is True
mock_get_recording_timestamps.assert_awaited_once_with("egress-1", lyon)
@mock.patch("core.utils.create_livekit_client")
def test_get_recording_timestamps_on_the_given_cluster(
mock_create_livekit_client, two_livekit_clusters
):
"""The egress is looked up on the LiveKit cluster given."""
_paris, lyon = two_livekit_clusters
started_at = datetime.datetime(2026, 1, 2, 10, 30, tzinfo=datetime.timezone.utc)
ended_at = datetime.datetime(2026, 1, 2, 11, 45, tzinfo=datetime.timezone.utc)
file_result = mock.Mock(
started_at=int(started_at.timestamp()) * 10**9,
ended_at=int(ended_at.timestamp()) * 10**9,
)
client = mock.AsyncMock()
client.egress.list_egress.return_value = mock.Mock(
items=[mock.Mock(file_results=[file_result])]
)
mock_create_livekit_client.return_value = client
timestamps = async_to_sync(NotificationService._get_recording_timestamps)(
"egress-1", lyon
)
assert timestamps == (started_at, ended_at)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
client.egress.list_egress.assert_awaited_once()
[request] = client.egress.list_egress.await_args.args
assert request.egress_id == "egress-1"
def test_summary_recipient_is_oldest_access():
"""With several accesses, the summary goes to the oldest one."""
recording = factories.RecordingFactory()
@@ -0,0 +1,94 @@
"""
Test MetadataCollectorService service.
"""
# pylint: disable=redefined-outer-name
from unittest import mock
import pytest
from livekit.protocol.agent_dispatch import CreateAgentDispatchRequest
from core.factories import RecordingFactory
from core.models import Placement
from core.recording.services.metadata_collector import MetadataCollectorService
pytestmark = pytest.mark.django_db
@pytest.fixture
def mock_create_livekit_client():
"""Mock the creation of LiveKit clients: every client created is one mock."""
with mock.patch("core.utils.create_livekit_client") as create_livekit_client:
create_livekit_client.return_value = mock.AsyncMock()
yield create_livekit_client
def test_start_on_the_recording_cluster(
mock_create_livekit_client, two_livekit_clusters, settings
):
"""The metadata collector agent is dispatched to the room on the LiveKit
cluster recording it, even once the room was placed elsewhere."""
_paris, lyon = two_livekit_clusters
recording = RecordingFactory(status="active", livekit_cluster="lyon")
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
client = mock_create_livekit_client.return_value
client.agent_dispatch.create_dispatch.return_value = mock.Mock(id="AD_lyon")
assert MetadataCollectorService().start(recording) == "AD_lyon"
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
client.agent_dispatch.create_dispatch.assert_awaited_once_with(
CreateAgentDispatchRequest(
agent_name=settings.METADATA_COLLECTOR_AGENT_NAME,
room=str(recording.room.id),
metadata=str(recording.id),
)
)
recording.refresh_from_db()
assert recording.options == {"metadata_collector_dispatch_id": "AD_lyon"}
def test_stop_on_the_recording_cluster(
mock_create_livekit_client, two_livekit_clusters
):
"""The metadata collector agent's dispatch is deleted on the LiveKit cluster
recording the room, even once the room was placed elsewhere."""
_paris, lyon = two_livekit_clusters
recording = RecordingFactory(
status="active",
livekit_cluster="lyon",
options={"metadata_collector_dispatch_id": "AD_lyon"},
)
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
MetadataCollectorService().stop(recording)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
client = mock_create_livekit_client.return_value
client.agent_dispatch.delete_dispatch.assert_awaited_once_with(
dispatch_id="AD_lyon", room_name=str(recording.room.id)
)
@pytest.mark.parametrize("livekit_cluster", ["", "lille"])
def test_start_and_stop_on_the_default_cluster(
livekit_cluster, mock_create_livekit_client, two_livekit_clusters
):
"""A recording naming no LiveKit cluster, or one no longer registered, has
the metadata collector agent dispatched and deleted on the default cluster,
wherever the room is placed."""
paris, _lyon = two_livekit_clusters
recording = RecordingFactory(status="active", livekit_cluster=livekit_cluster)
Placement.objects.create(room_name=str(recording.room.id), cluster="lyon")
client = mock_create_livekit_client.return_value
client.agent_dispatch.create_dispatch.return_value = mock.Mock(id="AD_paris")
service = MetadataCollectorService()
service.start(recording)
service.stop(recording)
assert mock_create_livekit_client.call_args_list == [mock.call(cluster=paris)] * 2
client.agent_dispatch.delete_dispatch.assert_awaited_once_with(
dispatch_id="AD_paris", room_name=str(recording.room.id)
)
@@ -10,6 +10,8 @@ from unittest import mock
import pytest
from core.factories import RecordingFactory
from core.livekit.clusters import get_default_livekit_cluster
from core.models import Placement
from core.recording.enums import RecordingWorkerEvent
from core.recording.services.recording_events import (
RecordingEventsError,
@@ -46,7 +48,9 @@ def test_handle_limit_reached_success(mock_notify, mode, notification_type, serv
assert recording.status == "stopped"
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
@@ -66,7 +70,9 @@ def test_handle_interrupted_success(mock_notify, mode, notification_type, servic
assert recording.status == "stopped"
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
@@ -86,7 +92,9 @@ def test_handle_failed_success(mock_notify, mode, notification_type, service):
assert recording.status == "failed"
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
@@ -106,7 +114,9 @@ def test_handle_aborted_success(mock_notify, mode, notification_type, service):
assert recording.status == "aborted"
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
@@ -152,6 +162,7 @@ def test_handle_event_notification_error( # noqa: PLR0913, PLR0917
mock_notify.assert_called_once_with(
room_name=str(recording.room.id),
notification_data={"type": f"{notification_prefix}{notification_suffix}"},
cluster=get_default_livekit_cluster(),
)
@@ -232,7 +243,9 @@ def test_handle_update_syncs_room_metadata(
service.handle_update(recording, event)
mock_update_metadata.assert_called_once_with(
str(recording.room.id), {"recording_status": recording_status}
str(recording.room.id),
{"recording_status": recording_status},
cluster=get_default_livekit_cluster(),
)
@@ -308,6 +321,7 @@ def test_handle_terminal_event_dispatches_on_event_and_status( # noqa: PLR0913,
mock_notify.assert_called_once_with(
room_name=str(recording.room.id),
notification_data={"type": f"screenRecording{notification_type}"},
cluster=get_default_livekit_cluster(),
)
@@ -488,3 +502,46 @@ def test_log_worker_error_stays_quiet_on_anything_else(event, service, caplog):
service.log_worker_error(recording, event, error="some error", error_code=500)
assert "Recording worker reported" not in caplog.text
def live_room_client():
"""Return a LiveKit client mock on which the room is live, without metadata."""
client = mock.AsyncMock()
client.room.list_rooms.return_value = mock.Mock(rooms=[mock.Mock(metadata="")])
return client
@mock.patch("core.utils.create_livekit_client")
def test_handle_update_on_the_recording_cluster(
mock_create_livekit_client, two_livekit_clusters, service
):
"""The room's metadata is updated on the LiveKit cluster recording it, even
once the room was placed elsewhere."""
_paris, lyon = two_livekit_clusters
recording = RecordingFactory(status="active", livekit_cluster="lyon")
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
client = live_room_client()
mock_create_livekit_client.return_value = client
service.handle_update(recording, RecordingWorkerEvent.STARTED)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
client.room.update_room_metadata.assert_awaited_once()
@mock.patch("core.utils.create_livekit_client")
def test_notify_participants_on_the_recording_cluster(
mock_create_livekit_client, two_livekit_clusters, service
):
"""Participants are notified on the LiveKit cluster recording the room, even
once the room was placed elsewhere."""
_paris, lyon = two_livekit_clusters
recording = RecordingFactory(status="active", livekit_cluster="lyon")
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
client = live_room_client()
mock_create_livekit_client.return_value = client
service._handle_limit_reached(recording)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
client.room.send_data.assert_awaited_once()
@@ -4,7 +4,8 @@ Test worker service factories.
# pylint: disable=protected-access,redefined-outer-name,unused-argument,no-member
from dataclasses import FrozenInstanceError
import json
from dataclasses import FrozenInstanceError, replace
from unittest.mock import Mock
from django.test import override_settings
@@ -14,11 +15,13 @@ from livekit import (
api as livekit_api_codec,
)
from core.livekit.clusters import get_livekit_cluster
from core.recording.worker.factories import (
WorkerService,
WorkerServiceConfig,
get_worker_service,
)
from core.tests.utils import livekit_cluster_entry
@pytest.fixture(autouse=True)
@@ -30,11 +33,16 @@ def clear_lru_cache():
@pytest.fixture
def test_settings():
def test_settings(settings):
"""Fixture to provide test Django settings"""
mocked_settings = {
"RECORDING_OUTPUT_FOLDER": "/test/output",
"LIVEKIT_CONFIGURATION": {"server": "test.example.com"},
"LIVEKIT_CLUSTERS": json.dumps(
[
livekit_cluster_entry("paris", api_url="http://livekit.paris.svc:7880"),
livekit_cluster_entry("lyon"),
]
),
"AWS_S3_ENDPOINT_URL": "https://s3.test.com",
"AWS_S3_ACCESS_KEY_ID": "test_key",
"AWS_S3_SECRET_ACCESS_KEY": "test_secret",
@@ -52,9 +60,8 @@ def test_settings():
"RECORDING_ENCODING_KEY_FRAME_INTERVAL_S": 4.0,
}
# Use override_settings to properly patch Django settings
with override_settings(**mocked_settings):
yield test_settings
for name, value in mocked_settings.items():
setattr(settings, name, value)
@pytest.fixture
@@ -67,7 +74,11 @@ def default_config(test_settings):
def test_config_initialization(default_config):
"""Test that WorkerServiceConfig is properly initialized from settings"""
assert default_config.output_folder == "/test/output"
assert default_config.server_configurations == {"server": "test.example.com"}
assert default_config.server_configurations == {
"url": "http://livekit.paris.svc:7880",
"api_key": "key-paris",
"api_secret": "secret-paris-padded-for-minimum-len!-livekit",
}
assert default_config.bucket_args == {
"endpoint": "https://s3.test.com",
"access_key": "test_key",
@@ -90,6 +101,14 @@ def test_config_initialization(default_config):
}
def test_config_server_configurations_from_livekit_configuration(settings):
"""Without LIVEKIT_CLUSTERS, LiveKit is called as LIVEKIT_CONFIGURATION says."""
config = WorkerServiceConfig.from_settings()
assert settings.LIVEKIT_CLUSTERS is None
assert config.server_configurations == settings.LIVEKIT_CONFIGURATION
def test_config_immutability(default_config):
"""Test that config instances are immutable after creation"""
with pytest.raises(FrozenInstanceError):
@@ -99,7 +118,6 @@ def test_config_immutability(default_config):
@pytest.mark.parametrize("custom_encoding_enabled", [True, False])
@override_settings(
RECORDING_OUTPUT_FOLDER="/test/output",
LIVEKIT_CONFIGURATION={"server": "test.example.com"},
AWS_S3_ENDPOINT_URL="https://s3.test.com",
AWS_S3_ACCESS_KEY_ID="test_key",
AWS_S3_SECRET_ACCESS_KEY="test_secret",
@@ -161,7 +179,6 @@ def test_config_encoding_options_none_when_default_missing(
@override_settings(
RECORDING_OUTPUT_FOLDER="/test/output",
LIVEKIT_CONFIGURATION={"server": "test.example.com"},
AWS_S3_ENDPOINT_URL="https://s3.test.com",
AWS_S3_ACCESS_KEY_ID="test_key",
AWS_S3_SECRET_ACCESS_KEY="test_secret",
@@ -208,6 +225,43 @@ def test_factory_valid_mode(mock_import_string, settings, default_config):
assert worker.config == default_config
def test_factory_given_cluster(mock_import_string, settings, default_config):
"""The worker service calls the LiveKit cluster it is given, with the URL,
API key and secret Django calls that cluster with."""
settings.RECORDING_WORKER_CLASSES = {"test_mode": "path.to.MockWorkerService"}
worker = get_worker_service("test_mode", cluster=get_livekit_cluster("lyon"))
assert worker.config == replace(
default_config,
server_configurations={
"url": "wss://lyon.livekit.test",
"api_key": "key-lyon",
"api_secret": "secret-lyon-padded-for-minimum-len!-livekit",
},
)
def test_factory_default_cluster_registered_now(
mock_import_string, settings, default_config
):
"""Without a cluster, the worker service calls the default LiveKit cluster
registered now, though the rest of its configuration is cached."""
settings.RECORDING_WORKER_CLASSES = {"test_mode": "path.to.MockWorkerService"}
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("marseille")])
worker = get_worker_service("test_mode")
assert worker.config == replace(
default_config,
server_configurations={
"url": "wss://marseille.livekit.test",
"api_key": "key-marseille",
"api_secret": "secret-marseille-padded-for-minimum-len!-livekit",
},
)
def test_factory_invalid_mode(settings, mock_import_string, default_config):
"""Test getting worker service with invalid mode raises ValueError."""
@@ -8,7 +8,8 @@ from unittest.mock import Mock
import pytest
from core.factories import RecordingFactory
from core.models import RecordingStatusChoices
from core.livekit.clusters import get_default_livekit_cluster
from core.models import Placement, RecordingStatusChoices
from core.recording.worker.exceptions import (
RecordingStartError,
RecordingStopError,
@@ -61,6 +62,29 @@ def test_start_recording_success(mock_update_metadata, mediator, mock_worker_ser
mock_update_metadata.assert_called_once_with(
str(mock_recording.room.id),
{"recording_mode": mock_recording.mode, "recording_status": "starting"},
cluster=get_default_livekit_cluster(),
)
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_start_recording_on_the_recording_cluster(
mock_update_metadata, mediator, mock_worker_service, two_livekit_clusters
):
"""The room's metadata is updated on the LiveKit cluster recording it, even
once the room was placed elsewhere."""
_paris, lyon = two_livekit_clusters
mock_worker_service.start.return_value = "test-worker-123"
recording = RecordingFactory(
status=RecordingStatusChoices.INITIATED, worker_id=None, livekit_cluster="lyon"
)
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
mediator.start(recording)
mock_update_metadata.assert_called_once_with(
str(recording.room.id),
{"recording_mode": recording.mode, "recording_status": "starting"},
cluster=lyon,
)
@@ -9,7 +9,11 @@ from unittest import mock
import pytest
from ...factories import RoomFactory
from ...livekit.clusters import get_default_livekit_cluster
from ...livekit.exceptions import LiveKitRoomCreationError
from ...models import Placement
from ...services.sip_management import SIPException
from ..utils import CREATED_LIVEKIT_ROOM_SID
pytestmark = pytest.mark.django_db
@@ -303,3 +307,71 @@ def test_join_sip_failure(mock_sip_management, mock_capture, settings, client):
assert response.status_code == 500
mock_sip_instance.ensure_dispatch_rule.assert_called_once_with(room)
mock_capture.assert_not_called()
@mock.patch("core.roomkit.viewsets.SIPManagement")
def test_join_places_the_room_first(
mock_sip_management, settings, client, fake_livekit_room_creation
):
"""The LiveKit room should be placed and created before the dispatch rule, as
the device may be the room's first participant."""
mock_sip_instance = mock_sip_management.return_value
settings.ROOMKIT_ENABLED = True
settings.ROOMKIT_SERVER_TO_SERVER_API_TOKEN = "testAuthToken"
room = RoomFactory(pin_code="1234567890")
placements_at_rule_creation = []
def ensure_dispatch_rule(rule_room):
placements_at_rule_creation.extend(
Placement.objects.filter(room=rule_room).values_list("cluster", "sid")
)
return True
mock_sip_instance.ensure_dispatch_rule.side_effect = ensure_dispatch_rule
response = client.post(
"/api/v1.0/roomkit/join/",
{"pin_code": room.pin_code},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 200
fake_livekit_room_creation.assert_called_once_with(
str(room.id), get_default_livekit_cluster()
)
assert placements_at_rule_creation == [("default", CREATED_LIVEKIT_ROOM_SID)]
mock_sip_instance.ensure_dispatch_rule.assert_called_once_with(room)
@mock.patch("core.roomkit.viewsets.analytics.capture")
@mock.patch("core.roomkit.viewsets.SIPManagement")
def test_join_room_creation_failure(
mock_sip_management, mock_capture, settings, client, fake_livekit_room_creation
):
"""Requests should fail with a 503, without any dispatch rule, when the
LiveKit room cannot be created."""
mock_sip_instance = mock_sip_management.return_value
settings.ROOMKIT_ENABLED = True
settings.ROOMKIT_SERVER_TO_SERVER_API_TOKEN = "testAuthToken"
room = RoomFactory(pin_code="1234567890")
fake_livekit_room_creation.side_effect = LiveKitRoomCreationError("Timed out")
response = client.post(
"/api/v1.0/roomkit/join/",
{"pin_code": room.pin_code},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
mock_sip_instance.ensure_dispatch_rule.assert_not_called()
mock_capture.assert_not_called()
assert not Placement.objects.exists()
@@ -4,10 +4,13 @@ Test rooms API endpoints in the Meet core app: join.
# pylint: disable=redefined-outer-name
import json
import logging
from unittest import mock
from django.conf import settings
from django.contrib.auth.models import AnonymousUser
from django.db import connection
from django.utils.crypto import get_random_string
import jwt
@@ -20,8 +23,14 @@ from ...factories import (
UserFactory,
UserResourceAccessFactory,
)
from ...models import RoleChoices, RoomAccessLevel
from ..utils import generate_user_access_token
from ...livekit.clusters import get_default_livekit_cluster
from ...livekit.exceptions import LiveKitRoomCreationError
from ...models import Placement, RoleChoices, RoomAccessLevel
from ..utils import (
CREATED_LIVEKIT_ROOM_SID,
generate_user_access_token,
livekit_cluster_entry,
)
fake = Faker()
pytestmark = pytest.mark.django_db
@@ -67,11 +76,14 @@ def join_url(room_id):
("access_level", "requester", "can_join_directly"), ACCESS_MATRIX
)
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_access(mock_token, access_level, requester, can_join_directly):
def test_api_rooms_join_access(
mock_token, access_level, requester, can_join_directly, fake_livekit_room_creation
):
"""
Joining a room should return LiveKit credentials and the pin code only to
users who may enter without going through the lobby, and the accesses
only to the room's administrators and owners.
only to the room's administrators and owners. Its LiveKit room should only
be placed and created for those handed out credentials.
"""
room = RoomFactory(
access_level=access_level,
@@ -128,9 +140,18 @@ def test_api_rooms_join_access(mock_token, access_level, requester, can_join_dir
sources=["camera"],
role=role,
participant_id=None,
cluster=get_default_livekit_cluster(),
)
fake_livekit_room_creation.assert_called_once_with(
str(room.id), get_default_livekit_cluster()
)
else:
mock_token.assert_not_called()
fake_livekit_room_creation.assert_not_called()
assert (
Placement.objects.filter(room_name=str(room.id), room=room).exists()
is can_join_directly
)
@mock.patch("core.utils.generate_token", return_value=livekit_token)
@@ -148,14 +169,16 @@ def test_api_rooms_join_by_slug(mock_token):
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_members_num_queries(mock_token, django_assert_num_queries):
"""The role of the requester should only be looked up once."""
"""The role of the requester should only be looked up once. Placing the
LiveKit room takes four more queries: its placement is read, created, read
back and given the SID of the LiveKit room created."""
user = UserFactory()
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
UserResourceAccessFactory(resource=room, user=user, role=RoleChoices.MEMBER)
client = APIClient()
client.force_login(user)
with django_assert_num_queries(3):
with django_assert_num_queries(7):
response = client.post(join_url(room.id), {}, format="json")
assert response.status_code == 200
@@ -163,19 +186,39 @@ def test_api_rooms_join_members_num_queries(mock_token, django_assert_num_querie
assert mock_token.call_args.kwargs["role"] == RoleChoices.MEMBER
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_placed_room_num_queries(mock_token, django_assert_num_queries):
"""Once the LiveKit room is placed, its placement is only read and given the
SID of the LiveKit room created."""
user = UserFactory()
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
UserResourceAccessFactory(resource=room, user=user, role=RoleChoices.MEMBER)
Placement.objects.create(room_name=str(room.id), room=room, cluster="default")
client = APIClient()
client.force_login(user)
with django_assert_num_queries(5):
response = client.post(join_url(room.id), {}, format="json")
assert response.status_code == 200
assert "livekit" in response.json()
mock_token.assert_called_once()
@pytest.mark.parametrize("role", [RoleChoices.ADMIN, RoleChoices.OWNER])
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_administrators_num_queries(
mock_token, role, django_assert_num_queries
):
"""Only the accesses should be queried on top of the role for administrators."""
"""Only the accesses should be queried on top of the role and the placement
of the LiveKit room for administrators."""
user = UserFactory()
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
UserResourceAccessFactory(resource=room, user=user, role=role)
client = APIClient()
client.force_login(user)
with django_assert_num_queries(4):
with django_assert_num_queries(8):
response = client.post(join_url(room.id), {}, format="json")
assert response.status_code == 200
@@ -184,6 +227,165 @@ def test_api_rooms_join_administrators_num_queries(
assert mock_token.call_args.kwargs["role"] == role
def test_api_rooms_join_placed_cluster(
two_livekit_clusters, fake_livekit_room_creation
):
"""
Joining a room should hand out credentials for the LiveKit cluster its
LiveKit room is placed on, where the LiveKit room is created: its URL, and
a token it signed.
"""
_paris, lyon = two_livekit_clusters
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
room_name = str(room.id)
with mock.patch("core.livekit.strategies.randbelow", return_value=1):
response = APIClient().post(join_url(room_name), {}, format="json")
assert response.status_code == 200
config = response.json()["livekit"]
assert config["url"] == lyon.url
assert config["room"] == room_name
claims = jwt.decode(config["token"], lyon.api_secret, algorithms=["HS256"])
assert claims["iss"] == lyon.api_key
assert claims["video"]["room"] == room_name
fake_livekit_room_creation.assert_called_once_with(room_name, lyon)
placement = Placement.objects.get()
assert placement.room_name == room_name
assert placement.room == room
assert placement.cluster == "lyon"
assert placement.sid == CREATED_LIVEKIT_ROOM_SID
def test_api_rooms_join_unregistered_default_cluster(
settings, two_livekit_clusters, fake_livekit_room_creation
):
"""
Joining an unregistered room should hand out credentials for the default
LiveKit cluster, whatever the placement strategy would pick, where its
LiveKit room is created, and place nothing: Django does not follow the
lifecycle of unregistered rooms.
"""
settings.ALLOW_UNREGISTERED_ROOMS = True
paris, _lyon = two_livekit_clusters
with mock.patch("core.livekit.strategies.randbelow", return_value=1):
response = APIClient().post(join_url("unregistered-room"), {}, format="json")
assert response.status_code == 200
config = response.json()["livekit"]
assert config["url"] == paris.url
assert config["room"] == "unregistered-room"
claims = jwt.decode(config["token"], paris.api_secret, algorithms=["HS256"])
assert claims["iss"] == paris.api_key
assert claims["video"]["room"] == "unregistered-room"
fake_livekit_room_creation.assert_called_once_with("unregistered-room", paris)
assert not Placement.objects.exists()
@pytest.mark.django_db(transaction=True)
def test_api_rooms_join_creates_livekit_room_outside_transaction(
fake_livekit_room_creation,
):
"""
The LiveKit room should be created outside any transaction, once its
placement is committed: other requests see it while LiveKit answers.
"""
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
in_atomic_block = []
def create_livekit_room(_room_name, _cluster):
in_atomic_block.append(connection.in_atomic_block)
return CREATED_LIVEKIT_ROOM_SID
fake_livekit_room_creation.side_effect = create_livekit_room
response = APIClient().post(join_url(room.id), {}, format="json")
assert response.status_code == 200
assert in_atomic_block == [False]
@pytest.mark.parametrize("registered", [True, False])
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_room_creation_failure(
mock_token, registered, settings, fake_livekit_room_creation
):
"""
Joining a room whose LiveKit room cannot be created should return a 503
without credentials, and leave the room without placement, so that the
next join may place it on another cluster.
"""
settings.ALLOW_UNREGISTERED_ROOMS = True
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC) if registered else None
fake_livekit_room_creation.side_effect = LiveKitRoomCreationError("Timed out")
response = APIClient().post(
join_url(room.id if registered else "unregistered-room"), {}, format="json"
)
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
mock_token.assert_not_called()
assert not Placement.objects.exists()
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_placed_room_creation_failure(
mock_token, two_livekit_clusters, fake_livekit_room_creation
):
"""
Joining a room placed earlier, whose LiveKit cluster cannot create its
LiveKit room, should return a 503 without credentials, and keep its
placement: participants may be connected there.
"""
_paris, lyon = two_livekit_clusters
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
placement = Placement.objects.create(
room_name=str(room.id), room=room, cluster="lyon", sid="RM_old"
)
fake_livekit_room_creation.side_effect = LiveKitRoomCreationError("Timed out")
response = APIClient().post(join_url(room.id), {}, format="json")
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
mock_token.assert_not_called()
fake_livekit_room_creation.assert_called_once_with(str(room.id), lyon)
kept_placement = Placement.objects.get()
assert kept_placement.pk == placement.pk
assert kept_placement.cluster == "lyon"
assert kept_placement.sid == "RM_old"
assert kept_placement.updated_at == placement.updated_at
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_without_candidate(
mock_token, settings, fake_livekit_room_creation, caplog
):
"""
Joining a room should return a 503 without credentials while no LiveKit
cluster may receive new placements, which is logged as critical.
"""
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("paris", weight=0)])
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
with caplog.at_level(logging.CRITICAL, logger="core.livekit.placements"):
response = APIClient().post(join_url(room.id), {}, format="json")
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
mock_token.assert_not_called()
fake_livekit_room_creation.assert_not_called()
assert "No LiveKit cluster may receive new placements" in caplog.text
def test_api_rooms_join_get_not_allowed():
"""Joining a room should only be possible with a POST request."""
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
@@ -231,6 +433,7 @@ def test_api_rooms_join_unregistered_allowed(mock_token, authenticated, settings
sources=None,
role=None,
participant_id=None,
cluster=get_default_livekit_cluster(),
)
@@ -263,6 +466,7 @@ def test_api_rooms_join_unregistered_allowed_not_normalized(mock_token, settings
sources=None,
role=None,
participant_id=None,
cluster=get_default_livekit_cluster(),
)
@@ -296,6 +500,24 @@ def test_api_rooms_join_unregistered_not_allowed(mock_token, settings):
mock_token.assert_not_called()
@pytest.mark.parametrize(("length", "status_code"), [(100, 200), (101, 404)])
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_unregistered_slug_length(
mock_token, length, status_code, settings, fake_livekit_room_creation
):
"""
Joining an unregistered room whose slug is longer than a room's slug may be
should return a 404 rather than credentials: no room could ever have it.
"""
settings.ALLOW_UNREGISTERED_ROOMS = True
response = APIClient().post(join_url("a" * length), {}, format="json")
assert response.status_code == status_code
assert mock_token.called is (status_code == 200)
assert fake_livekit_room_creation.called is (status_code == 200)
@pytest.mark.parametrize("lookup", ["---", "!!!", "_"])
@mock.patch("core.utils.generate_token", return_value=livekit_token)
def test_api_rooms_join_unregistered_empty_slug(mock_token, lookup, settings):
@@ -9,14 +9,17 @@ from unittest import mock
from django.core import signing
from django.core.cache import cache
import jwt
import pytest
from freezegun import freeze_time
from rest_framework.test import APIClient
from ... import utils
from ...factories import RoomFactory, UserFactory
from ...models import RoomAccessLevel
from ...livekit.exceptions import LiveKitRoomCreationError
from ...models import Placement, RoomAccessLevel
from ...services.lobby import LobbyService
from ..utils import random_remote_addr
pytestmark = pytest.mark.django_db
@@ -347,6 +350,108 @@ def test_request_entry_waiting_participant_public_room(settings):
assert not LobbyService()._index_members(room.id)
def test_request_entry_public_room_placed_cluster(
two_livekit_clusters, fake_livekit_room_creation
):
"""Entry granted at once should come with credentials for the LiveKit cluster
the LiveKit room is placed on, where it is created: its URL, and a token it
signed."""
_paris, lyon = two_livekit_clusters
room = RoomFactory(access_level=RoomAccessLevel.PUBLIC)
with mock.patch("core.livekit.strategies.randbelow", return_value=1):
response = APIClient(REMOTE_ADDR=random_remote_addr()).post(
f"/api/v1.0/rooms/{room.id}/request-entry/", {"username": "Guest"}
)
assert response.status_code == 200
assert response.json()["status"] == "accepted"
config = response.json()["livekit"]
assert config["url"] == lyon.url
claims = jwt.decode(config["token"], lyon.api_secret, algorithms=["HS256"])
assert claims["iss"] == lyon.api_key
assert claims["video"]["room"] == str(room.id)
fake_livekit_room_creation.assert_called_once_with(str(room.id), lyon)
assert Placement.objects.get(room=room).cluster == "lyon"
def test_request_entry_accepted_participant_placed_cluster(
two_livekit_clusters, fake_livekit_room_creation
):
"""A participant admitted from the lobby should be handed out credentials for
the LiveKit cluster the LiveKit room is placed on, where it is created: its
URL, and a token it signed."""
_paris, lyon = two_livekit_clusters
room = RoomFactory(access_level=RoomAccessLevel.RESTRICTED)
Placement.objects.create(room_name=str(room.id), room=room, cluster="lyon")
client = APIClient(REMOTE_ADDR=random_remote_addr())
with mock.patch.object(utils, "notify_participants", return_value=None):
response = client.post(
f"/api/v1.0/rooms/{room.id}/request-entry/", {"username": "Guest"}
)
assert response.json()["livekit"] is None
fake_livekit_room_creation.assert_not_called()
participant_id = response.json()["id"]
LobbyService().handle_participant_entry(
room.id, _lobby_signer().unsign(participant_id), True
)
response = client.post(
f"/api/v1.0/rooms/{room.id}/request-entry/",
{"username": "Guest", "participant_id": participant_id},
)
assert response.status_code == 200
assert response.json()["status"] == "accepted"
config = response.json()["livekit"]
assert config["url"] == lyon.url
claims = jwt.decode(config["token"], lyon.api_secret, algorithms=["HS256"])
assert claims["iss"] == lyon.api_key
assert claims["video"]["room"] == str(room.id)
fake_livekit_room_creation.assert_called_once_with(str(room.id), lyon)
@pytest.mark.parametrize("access_level", ["public", "restricted"])
def test_request_entry_room_creation_failure(
access_level, settings, fake_livekit_room_creation
):
"""Entry granted to a room whose LiveKit room cannot be created should be
answered with a 503, leaving the participant where they were in the lobby."""
settings.LOBBY_KEY_PREFIX = "mocked-cache-prefix"
room = RoomFactory(access_level=access_level)
participant_id = str(uuid.uuid4())
status = "waiting" if access_level == "public" else "accepted"
cache.set(
f"mocked-cache-prefix_{room.id}_{participant_id}",
{
"id": participant_id,
"username": "Guest",
"status": status,
"color": "#123456",
"entered_at": "2025-01-01T10:00:00+00:00",
},
)
fake_livekit_room_creation.side_effect = LiveKitRoomCreationError("Timed out")
response = APIClient(REMOTE_ADDR=random_remote_addr()).post(
f"/api/v1.0/rooms/{room.id}/request-entry/",
{
"username": "Guest",
"participant_id": _lobby_signer().sign(participant_id),
},
)
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
assert (
cache.get(f"mocked-cache-prefix_{room.id}_{participant_id}")["status"] == status
)
assert not Placement.objects.exists()
def test_request_entry_invalid_data():
"""Should return 400 for invalid request data."""
room = RoomFactory()
@@ -2,7 +2,7 @@
Test rooms API endpoints in the Meet core app: start recording.
"""
# pylint: disable=redefined-outer-name,unused-argument,no-member,too-many-lines
# pylint: disable=redefined-outer-name,unused-argument,no-member,too-many-lines,protected-access
from unittest import mock
@@ -10,9 +10,11 @@ import pytest
from livekit import api as livekit_api
from rest_framework.test import APIClient
from ... import utils
from ...audit.testing import capture_audit, find_events
from ...factories import RoomFactory, UserFactory
from ...models import Recording, RoomAccessLevel
from ...livekit.clusters import get_default_livekit_cluster
from ...models import Placement, Recording, RoomAccessLevel
from ...recording.worker.exceptions import RecordingStartError
pytestmark = pytest.mark.django_db
@@ -155,7 +157,9 @@ def test_start_recording_worker_error(
{"mode": "screen_recording"},
)
mock_worker_service_factory.assert_called_once_with(mode="screen_recording")
mock_worker_service_factory.assert_called_once_with(
mode="screen_recording", cluster=get_default_livekit_cluster()
)
assert response.status_code == 502
assert response.json() == {
@@ -265,7 +269,9 @@ def test_start_recording_success(
{"mode": "screen_recording"},
)
mock_worker_service_factory.assert_called_once_with(mode="screen_recording")
mock_worker_service_factory.assert_called_once_with(
mode="screen_recording", cluster=get_default_livekit_cluster()
)
assert response.status_code == 201
assert response.json() == {
@@ -278,6 +284,7 @@ def test_start_recording_success(
assert recording.room == room
assert recording.mode == "screen_recording"
assert recording.livekit_cluster == get_default_livekit_cluster().name
# Verify recording access details
assert recording.accesses.count() == 1
@@ -286,6 +293,54 @@ def test_start_recording_success(
assert access.role == "owner"
@mock.patch("core.utils.create_livekit_client")
def test_start_recording_on_the_placed_cluster(
mock_create_livekit_client, settings, two_livekit_clusters
):
"""A room is recorded on the LiveKit cluster it is placed on, which the
recording keeps: the egress, the room's metadata and the metadata collector
agent are all on that cluster."""
settings.RECORDING_ENABLE = True
settings.METADATA_COLLECTOR_ENABLED = True
_paris, lyon = two_livekit_clusters
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
Placement.objects.create(room_name=str(room.id), room=room, cluster="lyon")
client = mock.AsyncMock()
client._egress.start_room_composite_egress.return_value = mock.Mock(
egress_id="EG_lyon"
)
client.room.list_rooms.return_value = mock.Mock(rooms=[mock.Mock(metadata="")])
client.agent_dispatch.create_dispatch.return_value = mock.Mock(id="AD_lyon")
mock_create_livekit_client.return_value = client
api_client = APIClient()
api_client.force_login(user)
response = api_client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{
"mode": "screen_recording",
"options": {"transcribe": True, "collect_metadata": True},
},
format="json",
)
assert response.status_code == 201
recording = Recording.objects.get(room=room)
assert recording.livekit_cluster == "lyon"
assert recording.worker_id == "EG_lyon"
assert recording.options["metadata_collector_dispatch_id"] == "AD_lyon"
assert mock_create_livekit_client.call_args_list == [
mock.call(utils.livekit_api_configuration(lyon)),
mock.call(cluster=lyon),
mock.call(cluster=lyon),
]
client._egress.start_room_composite_egress.assert_awaited_once()
client.room.update_room_metadata.assert_awaited_once()
client.agent_dispatch.create_dispatch.assert_awaited_once()
@pytest.mark.parametrize("value", ["fr", "en", "nl", "de"])
def test_start_recording_options_language_valid(
settings, mock_worker_service_factory, mock_worker_manager, value
@@ -2,16 +2,19 @@
Test rooms API endpoints in the Meet core app: stop recording.
"""
# pylint: disable=redefined-outer-name,unused-argument
# pylint: disable=redefined-outer-name,unused-argument,no-member,protected-access
from unittest import mock
import pytest
from livekit import api as livekit_api
from rest_framework.test import APIClient
from ... import utils
from ...audit.testing import find_events
from ...factories import RecordingFactory, RoomFactory, UserFactory
from ...models import Recording, RecordingStatusChoices, RoomAccessLevel
from ...livekit.clusters import get_default_livekit_cluster
from ...models import Placement, Recording, RecordingStatusChoices, RoomAccessLevel
from ...recording.worker.exceptions import RecordingStopError
pytestmark = pytest.mark.django_db
@@ -143,7 +146,9 @@ def test_stop_recording_worker_error(
response = client.post(f"/api/v1.0/rooms/{room.id}/stop-recording/")
mock_worker_service_factory.assert_called_once_with(mode="screen_recording")
mock_worker_service_factory.assert_called_once_with(
mode="screen_recording", cluster=get_default_livekit_cluster()
)
mock_stop.assert_called_once_with(recording)
assert response.status_code == 500
@@ -179,7 +184,9 @@ def test_stop_recording_success(
response = client.post(f"/api/v1.0/rooms/{room.id}/stop-recording/")
mock_worker_service_factory.assert_called_once_with(mode="screen_recording")
mock_worker_service_factory.assert_called_once_with(
mode="screen_recording", cluster=get_default_livekit_cluster()
)
mock_stop.assert_called_once_with(recording)
assert response.status_code == 200
@@ -189,6 +196,76 @@ def test_stop_recording_success(
assert Recording.objects.count() == 1
@mock.patch("core.utils.create_livekit_client")
def test_stop_recording_on_the_recording_cluster(
mock_create_livekit_client, settings, two_livekit_clusters
):
"""A recording is stopped on the LiveKit cluster recording the room, even
once the room was placed elsewhere."""
settings.RECORDING_ENABLE = True
_paris, lyon = two_livekit_clusters
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
recording = RecordingFactory(
room=room,
status=RecordingStatusChoices.ACTIVE,
worker_id="EG_lyon",
livekit_cluster="lyon",
)
Placement.objects.create(room_name=str(room.id), room=room, cluster="paris")
client = mock.AsyncMock()
client._egress.stop_egress.return_value = mock.Mock(
status=livekit_api.EgressStatus.EGRESS_ENDING
)
mock_create_livekit_client.return_value = client
api_client = APIClient()
api_client.force_login(user)
response = api_client.post(f"/api/v1.0/rooms/{room.id}/stop-recording/")
assert response.status_code == 200
mock_create_livekit_client.assert_called_once_with(
utils.livekit_api_configuration(lyon)
)
client._egress.stop_egress.assert_awaited_once_with(
livekit_api.StopEgressRequest(egress_id="EG_lyon")
)
recording.refresh_from_db()
assert recording.status == RecordingStatusChoices.STOPPED
@pytest.mark.usefixtures("mock_worker_manager")
@pytest.mark.parametrize("livekit_cluster", ["", "lille"])
def test_stop_recording_on_the_default_cluster(
livekit_cluster, mock_worker_service_factory, settings, two_livekit_clusters
):
"""A recording naming no LiveKit cluster, as one started before recordings
kept theirs, or naming a cluster no longer registered, is stopped on the
default cluster, wherever the room is placed."""
settings.RECORDING_ENABLE = True
paris, _lyon = two_livekit_clusters
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
RecordingFactory(
room=room,
status=RecordingStatusChoices.ACTIVE,
mode="screen_recording",
livekit_cluster=livekit_cluster,
)
Placement.objects.create(room_name=str(room.id), room=room, cluster="lyon")
client = APIClient()
client.force_login(user)
response = client.post(f"/api/v1.0/rooms/{room.id}/stop-recording/")
assert response.status_code == 200
mock_worker_service_factory.assert_called_once_with(
mode="screen_recording", cluster=paris
)
def test_stop_recording_is_audited(
settings, mock_worker_service_factory, mock_worker_manager, audit_events
):
@@ -7,6 +7,7 @@ import uuid
from unittest import mock
from django.conf import settings
from django.contrib.auth.models import AnonymousUser
import pytest
from livekit.api import AccessToken, TwirpError, VideoGrants
@@ -14,6 +15,7 @@ from rest_framework.test import APIClient
from core.factories import RoomFactory, UserFactory
from core.tests.utils import generate_user_access_token
from core.utils import generate_token
pytestmark = pytest.mark.django_db
@@ -214,7 +216,10 @@ def test_start_subtitle_wrong_signature(settings, mock_livekit_token):
"""Test that tokens signed with incorrect signature are rejected."""
settings.ROOM_SUBTITLE_ENABLED = True
settings.LIVEKIT_CONFIGURATION["api_secret"] = "wrong-secret-padded-to-32-bytes!!"
settings.LIVEKIT_CONFIGURATION = {
**settings.LIVEKIT_CONFIGURATION,
"api_secret": "wrong-secret-padded-to-32-bytes!!",
}
room = RoomFactory()
client = APIClient()
@@ -231,6 +236,58 @@ def test_start_subtitle_wrong_signature(settings, mock_livekit_token):
}
def test_start_subtitle_token_from_another_cluster(
settings, mock_livekit_client, two_livekit_clusters
):
"""Test that a token signed by any registered LiveKit cluster is accepted."""
settings.ROOM_SUBTITLE_ENABLED = True
_paris, lyon = two_livekit_clusters
room = RoomFactory()
token = generate_token(room=str(room.id), user=AnonymousUser(), cluster=lyon)
client = APIClient()
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-subtitle/",
{},
HTTP_AUTHORIZATION=f"X-LiveKit-Token {token}",
)
assert response.status_code == 200
assert response.json() == {"status": "success"}
mock_livekit_client.agent_dispatch.create_dispatch.assert_called_once()
@pytest.mark.usefixtures("two_livekit_clusters")
def test_start_subtitle_token_from_an_unknown_cluster(settings, mock_livekit_client):
"""Test that a token signed with an API key no cluster uses is rejected."""
settings.ROOM_SUBTITLE_ENABLED = True
room = RoomFactory()
token = (
AccessToken(
api_key="key-lille",
api_secret="secret-lille-padded-for-minimum-len!-livekit",
)
.with_grants(VideoGrants(room=str(room.id), room_join=True))
.with_identity(str(uuid.uuid4()))
.to_jwt()
)
client = APIClient()
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-subtitle/",
{},
HTTP_AUTHORIZATION=f"X-LiveKit-Token {token}",
)
assert response.status_code == 403
assert response.json() == {"detail": "Invalid LiveKit token: Invalid issuer"}
mock_livekit_client.agent_dispatch.create_dispatch.assert_not_called()
@pytest.fixture
def user_access_token():
"""Generate a valid user access JWT, sent with the "Bearer" scheme."""
@@ -14,7 +14,9 @@ import pytest
from livekit import api
from ...factories import RecordingFactory
from ...livekit.clusters import get_default_livekit_cluster
from ...services.livekit_events import ActionFailedError, LiveKitEventsService
from ..utils import livekit_webhook_token
@pytest.fixture
@@ -140,6 +142,52 @@ def test_handled_event_type(
assert response.json() == {"status": "success"}
@mock.patch.object(LiveKitEventsService, "_handle_room_finished", autospec=True)
def test_handled_event_from_another_cluster(
mock_handler, client, serialized_event_data, two_livekit_clusters
):
"""Should process a webhook signed by any registered LiveKit cluster.
The service handling it knows which cluster sent it.
"""
_paris, lyon = two_livekit_clusters
response = client.post(
"/api/v1.0/rooms/webhooks-livekit/",
data=serialized_event_data,
content_type="application/json",
HTTP_AUTHORIZATION=livekit_webhook_token(
serialized_event_data, lyon.api_key, lyon.api_secret
),
)
assert response.status_code == 200
assert response.json() == {"status": "success"}
mock_handler.assert_called_once()
service = mock_handler.call_args.args[0]
assert service.cluster == lyon
@pytest.mark.usefixtures("two_livekit_clusters")
@mock.patch.object(LiveKitEventsService, "_handle_room_finished")
def test_event_from_an_unknown_cluster(mock_handler, client, serialized_event_data):
"""Should return 400 for a webhook signed with an API key no cluster uses."""
response = client.post(
"/api/v1.0/rooms/webhooks-livekit/",
data=serialized_event_data,
content_type="application/json",
HTTP_AUTHORIZATION=livekit_webhook_token(
serialized_event_data,
"key-lille",
"secret-lille-padded-for-minimum-len!-livekit",
),
)
assert response.status_code == 400
assert response.json() == {"status": "error"}
mock_handler.assert_not_called()
def test_unhandled_event_type(client, mock_livekit_config):
"""Should return 200 for event types that have no handler."""
event_data = json.dumps({"event": "participant_joined"})
@@ -253,6 +301,7 @@ def test_egress_ended_keeps_a_partial_transcript(
mock_notify.assert_called_once_with(
room_name=str(recording.room.id),
notification_data={"type": "transcriptionInterrupted"},
cluster=get_default_livekit_cluster(),
)
mock_notify_external_services.assert_called_once_with(recording)
@@ -3,6 +3,7 @@ Test LiveKitEvents service.
"""
# pylint: disable=W0621,W0613, W0212, E0611, E1101, C0302
import json
import logging
import uuid
from datetime import timedelta
@@ -16,7 +17,12 @@ from livekit.api import EgressStatus
from core.audit.testing import find_events
from core.factories import RecordingFactory, RoomFactory
from core.models import Room
from core.livekit.clusters import (
LiveKitClusterTokenVerifier,
get_default_livekit_cluster,
)
from core.livekit.placements import PlacementService
from core.models import Placement, Room
from core.recording.enums import RecordingWorkerEvent
from core.recording.services.recording_events import RecordingEventsService
from core.services.livekit_events import (
@@ -32,6 +38,7 @@ from core.services.livekit_events import (
to_recording_event,
)
from core.services.lobby import LobbyService
from core.services.presence import PresenceCache
from core.services.room_management import (
RoomManagementException,
RoomNotFoundException,
@@ -40,11 +47,15 @@ from core.services.sip_management import (
SIPException,
SIPManagement,
)
from core.tests.utils import livekit_webhook_token
from core.utils import NotificationError
pytestmark = pytest.mark.django_db
fake = Faker()
SID = "RM_hycBMAjmt6Ub"
OTHER_SID = "RM_dxSBWkfXbe4v"
@pytest.fixture
def mock_livekit_config(settings):
@@ -63,25 +74,93 @@ def service(mock_livekit_config):
return LiveKitEventsService()
@mock.patch("livekit.api.TokenVerifier")
@mock.patch("livekit.api.WebhookReceiver")
def test_initialization(
mock_webhook_receiver, mock_token_verifier, mock_livekit_config
):
"""Should correctly initialize the service with required dependencies."""
def test_initialization():
"""Should correctly initialize the service with required dependencies.
api_key = mock_livekit_config["api_key"]
api_secret = mock_livekit_config["api_secret"]
Webhooks are verified with the keys of whichever LiveKit cluster signed them.
"""
service = LiveKitEventsService()
mock_token_verifier.assert_called_once_with(api_key, api_secret)
mock_webhook_receiver.assert_called_once_with(mock_token_verifier.return_value)
assert isinstance(service.token_verifier, LiveKitClusterTokenVerifier)
assert isinstance(service.lobby_service, LobbyService)
assert isinstance(service.sip_management, SIPManagement)
assert isinstance(service.recording_events, RecordingEventsService)
def test_cluster_defaults_to_the_default_cluster(service):
"""Before any webhook is received, the service's cluster is the default one.
Handlers called directly therefore act on the default cluster.
"""
assert service.cluster == get_default_livekit_cluster()
assert service.cluster.name == "default"
def webhook_request(body, api_key, api_secret):
"""Return a webhook request whose body is signed with this API key and secret."""
request = mock.MagicMock()
request.headers = {
"Authorization": livekit_webhook_token(body, api_key, api_secret)
}
request.body = body.encode()
return request
def room_event_request(cluster, event, room_name, sid=SID):
"""Return the webhook request a LiveKit cluster sends on an event of a room."""
body = json.dumps({"event": event, "room": {"name": room_name, "sid": sid}})
return webhook_request(body, cluster.api_key, cluster.api_secret)
@pytest.fixture
def room_cleanup(settings):
"""Mock the cleanup of a room whose LiveKit room finished.
Provide the mocks clearing its presence and lobby caches. Telephony and
roomkit are disabled, as they are with several LiveKit clusters: the room
has no SIP dispatch rule to delete.
"""
settings.ROOM_TELEPHONY_ENABLED = False
settings.ROOMKIT_ENABLED = False
with (
mock.patch.object(PresenceCache, "clear_room") as clear_presence,
mock.patch.object(LobbyService, "clear_room_cache") as clear_lobby,
):
yield [clear_presence, clear_lobby]
@mock.patch.object(LiveKitEventsService, "_handle_room_finished")
def test_receive_identifies_the_cluster(
mock_handle_room_finished, two_livekit_clusters
):
"""The cluster whose API key signed a webhook becomes the service's cluster."""
_paris, lyon = two_livekit_clusters
body = json.dumps({"event": "room_finished", "room": {"name": str(uuid.uuid4())}})
service = LiveKitEventsService()
service.receive(webhook_request(body, lyon.api_key, lyon.api_secret))
mock_handle_room_finished.assert_called_once()
assert service.cluster == lyon
@pytest.mark.usefixtures("two_livekit_clusters")
@mock.patch.object(LiveKitEventsService, "_handle_room_finished")
def test_receive_unknown_cluster(mock_handle_room_finished):
"""A webhook signed with an API key no cluster uses is an invalid payload."""
body = json.dumps({"event": "room_finished", "room": {"name": str(uuid.uuid4())}})
request = webhook_request(
body, "key-lille", "secret-lille-padded-for-minimum-len!-livekit"
)
service = LiveKitEventsService()
with pytest.raises(InvalidPayloadError, match="Invalid webhook payload"):
service.receive(request)
mock_handle_room_finished.assert_not_called()
@pytest.mark.parametrize(
("mode", "notification_type"),
(
@@ -104,10 +183,14 @@ def test_handle_egress_ended_success( # pylint: disable=too-many-arguments, too
service._handle_egress_ended(mock_data)
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
mock_update_metadata.assert_called_once_with(
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
str(recording.room.id),
remove_keys=["recording_mode", "recording_status"],
cluster=get_default_livekit_cluster(),
)
recording.refresh_from_db()
@@ -137,7 +220,9 @@ def test_handle_egress_updated_success(
service._handle_egress_updated(mock_data)
mock_update_metadata.assert_called_once_with(
str(recording.room.id), {"recording_status": status}
str(recording.room.id),
{"recording_status": status},
cluster=get_default_livekit_cluster(),
)
@@ -165,7 +250,9 @@ def test_handle_egress_ended_metadata_update_fails( # pylint: disable=too-many-
service._handle_egress_ended(mock_data)
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
recording.refresh_from_db()
@@ -197,9 +284,12 @@ def test_handle_egress_ended_notification_fails(
mock_notify.assert_called_once_with(
room_name=str(recording.room.id),
notification_data={"type": "screenRecordingLimitReached"},
cluster=get_default_livekit_cluster(),
)
mock_update_metadata.assert_called_once_with(
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
str(recording.room.id),
remove_keys=["recording_mode", "recording_status"],
cluster=get_default_livekit_cluster(),
)
recording.refresh_from_db()
@@ -254,7 +344,9 @@ def test_handle_egress_ended_recording_should_not_be_saved(
mock_notify.assert_not_called()
mock_update_metadata.assert_called_once_with(
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
str(recording.room.id),
remove_keys=["recording_mode", "recording_status"],
cluster=get_default_livekit_cluster(),
)
recording.refresh_from_db()
@@ -289,7 +381,9 @@ def test_handle_egress_ended_complete_does_not_notify_participants(
mock_notify.assert_not_called()
mock_update_metadata.assert_called_once_with(
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
str(recording.room.id),
remove_keys=["recording_mode", "recording_status"],
cluster=get_default_livekit_cluster(),
)
mock_notify_external_services.assert_called_once_with(recording)
@@ -354,6 +448,50 @@ def test_handle_egress_ended_does_not_call_metadata_collector_stop_when_conditio
mock_collector.stop.assert_not_called()
@mock.patch(
"core.recording.services.recording_events.notification_service."
"notify_external_services",
return_value=True,
)
@mock.patch("core.utils.create_livekit_client")
def test_handle_egress_ended_on_the_recording_cluster(
mock_create_livekit_client,
mock_notify_external_services,
service,
two_livekit_clusters,
):
"""An ended egress is handled on the LiveKit cluster recording the room.
The room's metadata is cleaned up, the metadata collector's dispatch deleted
and participants notified there, even once the room was placed elsewhere.
"""
paris, lyon = two_livekit_clusters
recording = RecordingFactory(
worker_id="worker-1",
status="active",
livekit_cluster="lyon",
options={"metadata_collector_dispatch_id": "dispatch-123"},
)
Placement.objects.create(room_name=str(recording.room.id), cluster="paris")
client = mock.AsyncMock()
client.room.list_rooms.return_value = mock.Mock(rooms=[mock.Mock(metadata="")])
mock_create_livekit_client.return_value = client
mock_data = mock.MagicMock()
mock_data.egress_info.egress_id = recording.worker_id
mock_data.egress_info.status = EgressStatus.EGRESS_LIMIT_REACHED
assert service.cluster == paris
service._handle_egress_ended(mock_data)
assert mock_create_livekit_client.call_args_list == [mock.call(cluster=lyon)] * 3
client.room.update_room_metadata.assert_awaited_once()
client.agent_dispatch.delete_dispatch.assert_awaited_once_with(
dispatch_id="dispatch-123", room_name=str(recording.room.id)
)
client.room.send_data.assert_awaited_once()
mock_notify_external_services.assert_called_once_with(recording)
@pytest.mark.parametrize(
("egress_status", "recording_status", "event", "expected_level"),
(
@@ -638,11 +776,15 @@ def test_handle_egress_ended_interrupted_saves_partial_recording( # noqa: PLR09
service._handle_egress_ended(data)
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
room_name=str(recording.room.id),
notification_data={"type": notification_type},
cluster=get_default_livekit_cluster(),
)
mock_notify_external_services.assert_called_once_with(recording)
mock_update_metadata.assert_called_once_with(
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
str(recording.room.id),
remove_keys=["recording_mode", "recording_status"],
cluster=get_default_livekit_cluster(),
)
assert (
f"Recording worker reported interrupted for recording {recording.id}"
@@ -714,6 +856,7 @@ def test_handle_egress_ended_failed_on_another_error_discards_recording(
mock_notify.assert_called_once_with(
room_name=str(recording.room.id),
notification_data={"type": "transcriptionFailed"},
cluster=get_default_livekit_cluster(),
)
recording.refresh_from_db()
@@ -1041,7 +1184,9 @@ def test_handle_room_started_closes_deleted_room(
service._handle_room_started(mock_data)
mock_delete_room.assert_called_once_with(str(room.id))
mock_delete_room.assert_called_once_with(
str(room.id), cluster=get_default_livekit_cluster()
)
mock_ensure_dispatch_rule.assert_not_called()
@@ -1060,7 +1205,9 @@ def test_handle_room_started_ignores_already_closed_deleted_room(
service._handle_room_started(mock_data)
mock_delete_room.assert_called_once_with(str(room.id))
mock_delete_room.assert_called_once_with(
str(room.id), cluster=get_default_livekit_cluster()
)
@mock.patch(
@@ -1254,6 +1401,146 @@ def test_receive_ignores_connection_test_room(
mock_handle_room_finished.assert_not_called()
@mock.patch("core.services.room_management.RoomManagement.delete_room")
def test_receive_room_started_closes_a_deleted_room_on_its_cluster(
mock_delete_room, two_livekit_clusters
):
"""A LiveKit room started for a deleted room is closed on the cluster that
sent the webhook."""
_paris, lyon = two_livekit_clusters
room = RoomFactory()
room.soft_delete()
LiveKitEventsService().receive(
room_event_request(lyon, "room_started", str(room.id))
)
mock_delete_room.assert_called_once_with(str(room.id), cluster=lyon)
def test_receive_room_finished_deletes_the_placement(
two_livekit_clusters, room_cleanup
):
"""A LiveKit room that finished on the cluster and with the SID of its
placement deletes it, and its room is cleaned up."""
_paris, lyon = two_livekit_clusters
room_id = uuid.uuid4()
Placement.objects.create(room_name=str(room_id), cluster="lyon", sid=SID)
Placement.objects.create(room_name="other-room", cluster="lyon", sid=SID)
LiveKitEventsService().receive(
room_event_request(lyon, "room_finished", str(room_id))
)
assert list(Placement.objects.values_list("room_name", flat=True)) == ["other-room"]
for cleanup in room_cleanup:
cleanup.assert_called_once_with(room_id)
@pytest.mark.parametrize(
("cluster", "sid"),
(("paris", SID), ("lyon", OTHER_SID), ("lyon", "")),
)
def test_receive_room_finished_elsewhere_than_placed(
cluster, sid, two_livekit_clusters, room_cleanup, caplog
):
"""A LiveKit room that finished on another cluster or with another SID than
its placement's leaves the placement and the room as they are.
The placement points at the LiveKit room the room now uses, where
participants admitted from the lobby must not be queued again.
"""
_paris, lyon = two_livekit_clusters
room_id = uuid.uuid4()
placement = Placement.objects.create(
room_name=str(room_id), cluster=cluster, sid=sid
)
with caplog.at_level(logging.INFO, logger="core.services.livekit_events"):
LiveKitEventsService().receive(
room_event_request(lyon, "room_finished", str(room_id))
)
kept_placement = Placement.objects.get()
assert kept_placement.pk == placement.pk
assert kept_placement.cluster == cluster
assert kept_placement.sid == sid
for cleanup in room_cleanup:
cleanup.assert_not_called()
assert (
f"LiveKit room {room_id} finished on LiveKit cluster lyon with SID {SID}, "
f"but is placed on LiveKit cluster {cluster} with SID {sid!r}: ignoring "
"the event"
) in caplog.text
@mock.patch("core.services.room_management.RoomManagement.delete_room")
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_receive_room_finished_after_a_relocation(
mock_update_metadata, mock_delete_room, two_livekit_clusters, room_cleanup, caplog
):
"""The LiveKit room a room was relocated from, finishing on the cluster it
left with the SID it had there, leaves the placement on the new cluster and
the room as they are: participants admitted from the lobby, reconnecting on
the new cluster, must not be queued again."""
paris, _lyon = two_livekit_clusters
room_id = uuid.uuid4()
Placement.objects.create(room_name=str(room_id), cluster="paris", sid=SID)
PlacementService.relocate(str(room_id), "lyon", actor=None)
mock_delete_room.assert_called_once_with(str(room_id), cluster=paris)
relocated = Placement.objects.get()
with caplog.at_level(logging.INFO, logger="core.services.livekit_events"):
LiveKitEventsService().receive(
room_event_request(paris, "room_finished", str(room_id))
)
kept_placement = Placement.objects.get()
assert kept_placement.pk == relocated.pk
assert kept_placement.cluster == "lyon"
assert kept_placement.sid == ""
assert kept_placement.epoch == 1
assert kept_placement.updated_at == relocated.updated_at
for cleanup in room_cleanup:
cleanup.assert_not_called()
assert (
f"LiveKit room {room_id} finished on LiveKit cluster paris with SID {SID}, "
"but is placed on LiveKit cluster lyon with SID '': ignoring the event"
) in caplog.text
def test_receive_room_finished_without_placement(two_livekit_clusters, room_cleanup):
"""A LiveKit room that finished without placement has its room cleaned up,
as before placements existed."""
_paris, lyon = two_livekit_clusters
room_id = uuid.uuid4()
Placement.objects.create(room_name="other-room", cluster="lyon", sid=SID)
LiveKitEventsService().receive(
room_event_request(lyon, "room_finished", str(room_id))
)
assert Placement.objects.filter(room_name="other-room").exists()
for cleanup in room_cleanup:
cleanup.assert_called_once_with(room_id)
def test_receive_room_finished_without_sid(two_livekit_clusters, room_cleanup):
"""A room event without SID leaves placements as they are, and the room is
cleaned up as before placements existed."""
_paris, lyon = two_livekit_clusters
room_id = uuid.uuid4()
Placement.objects.create(room_name=str(room_id), cluster="lyon", sid=SID)
LiveKitEventsService().receive(
room_event_request(lyon, "room_finished", str(room_id), sid="")
)
assert Placement.objects.filter(room_name=str(room_id)).exists()
for cleanup in room_cleanup:
cleanup.assert_called_once_with(room_id)
@mock.patch("core.services.presence.cache.delete")
def test_participant_left_clearing_gated_by_setting(mock_delete, service, settings):
"""PRESENCE_CLEAR_ON_PARTICIPANT_LEFT toggles eager presence invalidation."""
@@ -17,6 +17,7 @@ from freezegun import freeze_time
from core import utils
from core.factories import RoomFactory, UserFactory, UserResourceAccessFactory
from core.livekit.clusters import get_default_livekit_cluster
from core.models import RoleChoices, RoomAccessLevel
from core.services.lobby import (
LobbyParticipant,
@@ -254,6 +255,7 @@ def test_request_entry_public_room(
configuration=room.configuration,
participant_id=participant_id,
role=None,
cluster=get_default_livekit_cluster(),
)
@@ -295,6 +297,7 @@ def test_request_entry_trusted_room(
configuration=room.configuration,
participant_id=participant_id,
role=None,
cluster=get_default_livekit_cluster(),
)
@@ -401,6 +404,7 @@ def test_request_entry_accepted_participant(
configuration=room.configuration,
participant_id="test-participant-id",
role=None,
cluster=get_default_livekit_cluster(),
)
@@ -481,6 +485,7 @@ def test_request_entry_participant_with_role(
configuration=room.configuration,
participant_id="test-participant-id",
role="administrator",
cluster=get_default_livekit_cluster(),
)
@@ -0,0 +1,103 @@
"""Tests for the ParticipantsManagement service."""
from unittest import mock
from uuid import uuid4
import pytest
from core.models import Placement
from core.services.participants_management import ParticipantsManagement
pytestmark = pytest.mark.django_db
PARTICIPANT_CALLS = [
pytest.param(
lambda room_name, **kwargs: ParticipantsManagement().mute(
room_name, "participant", "track-sid", **kwargs
),
id="mute",
),
pytest.param(
lambda room_name, **kwargs: ParticipantsManagement().remove(
room_name, "participant", **kwargs
),
id="remove",
),
pytest.param(
lambda room_name, **kwargs: ParticipantsManagement().update(
room_name, "participant", attributes={"room_role": "member"}, **kwargs
),
id="update",
),
pytest.param(
lambda room_name, **kwargs: ParticipantsManagement().check_if_in_meeting(
room_name, "participant", **kwargs
),
id="check_if_in_meeting",
),
]
@pytest.mark.parametrize(
"manage_participant",
[
*PARTICIPANT_CALLS,
pytest.param(
lambda room_name: ParticipantsManagement().check_if_in_meeting_cached(
room_name, "participant"
),
id="check_if_in_meeting_cached",
),
],
)
@mock.patch("core.utils.create_livekit_client")
def test_participants_managed_on_the_placed_cluster(
mock_create_livekit_client,
manage_participant,
two_livekit_clusters,
django_assert_num_queries,
):
"""Participants are managed on the LiveKit cluster their room is placed on."""
_paris, lyon = two_livekit_clusters
room_name = str(uuid4())
Placement.objects.create(room_name=room_name, cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(1):
manage_participant(room_name)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
@pytest.mark.parametrize("manage_participant", PARTICIPANT_CALLS)
@mock.patch("core.utils.create_livekit_client")
def test_participants_managed_on_the_given_cluster(
mock_create_livekit_client,
manage_participant,
two_livekit_clusters,
django_assert_num_queries,
):
"""Participants are managed on the cluster given, whatever the placement."""
paris, _lyon = two_livekit_clusters
room_name = str(uuid4())
Placement.objects.create(room_name=room_name, cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(0):
manage_participant(room_name, cluster=paris)
mock_create_livekit_client.assert_called_once_with(cluster=paris)
@pytest.mark.usefixtures("two_livekit_clusters")
@mock.patch("core.utils.create_livekit_client")
def test_check_if_in_meeting_without_room_or_identity(
mock_create_livekit_client, django_assert_num_queries
):
"""Without a room or an identity, nobody is looked for, nor any placement."""
with django_assert_num_queries(0):
assert ParticipantsManagement().check_if_in_meeting("", "participant") is False
assert ParticipantsManagement().check_if_in_meeting("room", "") is False
mock_create_livekit_client.assert_not_called()
@@ -8,7 +8,7 @@ import pytest
from livekit.api import TwirpError
from core.factories import RoomFactory
from core.models import Room, RoomAccessLevel
from core.models import Placement, Room, RoomAccessLevel
from core.services.room_management import (
RoomManagement,
RoomManagementException,
@@ -297,3 +297,61 @@ def test_sync_room_metadata_pushes_configuration_and_access_level(mock_update_me
"access_level": RoomAccessLevel.RESTRICTED,
},
)
def live_room_client():
"""Return a LiveKit client mock on which the room is live, without metadata."""
client = mock.AsyncMock()
client.room.list_rooms.return_value = mock.Mock(rooms=[mock.Mock(metadata="")])
return client
ROOM_CALLS = [
pytest.param(
lambda room_name, **kwargs: RoomManagement.update_metadata(
room_name, {"key": "value"}, **kwargs
),
id="update_metadata",
),
pytest.param(RoomManagement.delete_room, id="delete_room"),
]
@pytest.mark.django_db
@pytest.mark.parametrize("manage_room", ROOM_CALLS)
@mock.patch("core.services.room_management.utils.create_livekit_client")
def test_room_managed_on_the_placed_cluster(
mock_create_livekit_client,
manage_room,
two_livekit_clusters,
django_assert_num_queries,
):
"""A LiveKit room is managed on the LiveKit cluster it is placed on."""
_paris, lyon = two_livekit_clusters
Placement.objects.create(room_name="room-abc", cluster="lyon")
mock_create_livekit_client.return_value = live_room_client()
with django_assert_num_queries(1):
manage_room("room-abc")
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
@pytest.mark.django_db
@pytest.mark.parametrize("manage_room", ROOM_CALLS)
@mock.patch("core.services.room_management.utils.create_livekit_client")
def test_room_managed_on_the_given_cluster(
mock_create_livekit_client,
manage_room,
two_livekit_clusters,
django_assert_num_queries,
):
"""A LiveKit room is managed on the cluster given, whatever its placement."""
paris, _lyon = two_livekit_clusters
Placement.objects.create(room_name="room-abc", cluster="lyon")
mock_create_livekit_client.return_value = live_room_client()
with django_assert_num_queries(0):
manage_room("room-abc", cluster=paris)
mock_create_livekit_client.assert_called_once_with(cluster=paris)
@@ -8,6 +8,7 @@ from unittest import mock
import pytest
from core.factories import RoomFactory
from core.models import Placement
from core.services.subtitle import SubtitleService
pytestmark = pytest.mark.django_db
@@ -37,6 +38,38 @@ def test_start_subtitle_settings(mock_livekit_client, settings):
assert call_args.room == str(room.id)
@mock.patch("core.utils.create_livekit_client")
def test_start_subtitle_on_the_placed_cluster(
mock_create_livekit_client, two_livekit_clusters, django_assert_num_queries
):
"""The agent is dispatched on the LiveKit cluster the room is placed on."""
_paris, lyon = two_livekit_clusters
room = RoomFactory()
Placement.objects.create(room_name=str(room.id), room=room, cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(1):
SubtitleService().start_subtitle(room)
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
@mock.patch("core.utils.create_livekit_client")
def test_start_subtitle_on_the_given_cluster(
mock_create_livekit_client, two_livekit_clusters, django_assert_num_queries
):
"""The agent is dispatched on the cluster given, whatever the placement."""
paris, _lyon = two_livekit_clusters
room = RoomFactory()
Placement.objects.create(room_name=str(room.id), room=room, cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(0):
SubtitleService().start_subtitle(room, cluster=paris)
mock_create_livekit_client.assert_called_once_with(cluster=paris)
def test_stop_subtitle_not_implemented():
"""Test that stop_subtitle raises NotImplementedError."""
@@ -1,9 +1,11 @@
"""Tests for connection test Celery tasks."""
import logging
from unittest import mock
from django.test.utils import override_settings
import pytest
from core.livekit.clusters import get_default_livekit_cluster
from core.services.room_management import (
RoomManagementException,
RoomNotFoundException,
@@ -13,11 +15,50 @@ from core.tasks.connection_test import delete_connection_test_room
@mock.patch("core.tasks.connection_test.RoomManagement.delete_room")
def test_delete_connection_test_room_calls_room_management(mock_delete_room, settings):
"""RoomManagement.delete_room is called for connection-test rooms."""
"""RoomManagement.delete_room is called for connection-test rooms, on the
default LiveKit cluster when the task names none, as tasks queued before
connection test rooms were created on a chosen cluster do."""
settings.CONNECTION_TEST_ROOM_PREFIX = "connection-test"
delete_connection_test_room("connection-test-abc")
mock_delete_room.assert_called_once_with("connection-test-abc")
mock_delete_room.assert_called_once_with(
"connection-test-abc", cluster=get_default_livekit_cluster()
)
@mock.patch("core.tasks.connection_test.RoomManagement.delete_room")
def test_delete_connection_test_room_on_its_cluster(
mock_delete_room, settings, two_livekit_clusters
):
"""A connection-test room is deleted on the LiveKit cluster it was created on."""
settings.CONNECTION_TEST_ROOM_PREFIX = "connection-test"
_paris, lyon = two_livekit_clusters
delete_connection_test_room("connection-test-abc", "lyon")
mock_delete_room.assert_called_once_with("connection-test-abc", cluster=lyon)
@pytest.mark.usefixtures("two_livekit_clusters")
@mock.patch("core.tasks.connection_test.RoomManagement.delete_room")
def test_delete_connection_test_room_on_unregistered_cluster(
mock_delete_room, settings, caplog
):
"""A connection-test room created on a LiveKit cluster since removed from
the registry is deleted on the default cluster, with a warning."""
settings.CONNECTION_TEST_ROOM_PREFIX = "connection-test"
with caplog.at_level(logging.WARNING, logger="core.tasks.connection_test"):
delete_connection_test_room("connection-test-abc", "lille")
mock_delete_room.assert_called_once_with(
"connection-test-abc", cluster=get_default_livekit_cluster()
)
assert get_default_livekit_cluster().name == "paris"
assert (
"Connection test room 'connection-test-abc' was created on LiveKit cluster "
"lille, which is not registered: deleting it on the default cluster paris."
) in caplog.text
@mock.patch("core.tasks.connection_test.RoomManagement.delete_room")
@@ -35,9 +76,11 @@ def test_delete_connection_test_room_ignores_missing_room(mock_delete_room, sett
settings.CONNECTION_TEST_ROOM_PREFIX = "connection-test"
mock_delete_room.side_effect = RoomNotFoundException("Room does not exist")
delete_connection_test_room("connection-test-gone")
delete_connection_test_room("connection-test-gone", "default")
mock_delete_room.assert_called_once_with("connection-test-gone")
mock_delete_room.assert_called_once_with(
"connection-test-gone", cluster=get_default_livekit_cluster()
)
@mock.patch("core.tasks.connection_test.RoomManagement.delete_room")
@@ -46,6 +89,8 @@ def test_delete_connection_test_room_logs_other_failures(mock_delete_room, setti
settings.CONNECTION_TEST_ROOM_PREFIX = "connection-test"
mock_delete_room.side_effect = RoomManagementException("Could not delete room")
delete_connection_test_room("connection-test-fail")
delete_connection_test_room("connection-test-fail", "default")
mock_delete_room.assert_called_once_with("connection-test-fail")
mock_delete_room.assert_called_once_with(
"connection-test-fail", cluster=get_default_livekit_cluster()
)
@@ -1,11 +1,38 @@
"""Test the LiveKit reconnection settings exposed by the config endpoint."""
"""Test the LiveKit settings exposed by the config endpoint."""
import json
import pytest
from rest_framework.test import APIClient
from core.tests.utils import livekit_cluster_entry
pytestmark = pytest.mark.django_db
def test_api_config_retrieve_livekit_url(settings):
"""Without LIVEKIT_CLUSTERS, browsers connect to LIVEKIT_CONFIGURATION's URL."""
response = APIClient().get("/api/v1.0/config/")
assert response.status_code == 200
assert response.json()["livekit"]["url"] == settings.LIVEKIT_CONFIGURATION["url"]
def test_api_config_retrieve_livekit_url_default_cluster(settings):
"""Browsers are given the URL of the default LiveKit cluster, not its API URL."""
settings.LIVEKIT_CLUSTERS = json.dumps(
[
livekit_cluster_entry("paris", api_url="http://livekit.paris.svc:7880"),
livekit_cluster_entry("lyon"),
]
)
response = APIClient().get("/api/v1.0/config/")
assert response.status_code == 200
assert response.json()["livekit"]["url"] == "wss://paris.livekit.test"
def test_api_config_retrieve_livekit_reconnect_defaults():
"""Without any override, the frontend gets the default backoff."""
response = APIClient().get("/api/v1.0/config/")
+74 -2
View File
@@ -1,5 +1,6 @@
"""Test diagnostics API endpoints."""
import json
import uuid
from unittest import mock
@@ -15,6 +16,10 @@ from core.api.throttling import (
ConnectionTestUserRateThrottle,
)
from core.factories import UserFactory
from core.livekit.clusters import get_livekit_cluster
from core.livekit.exceptions import LiveKitRoomCreationError
from core.models import Placement
from core.tests.utils import livekit_cluster_entry, random_remote_addr
pytestmark = pytest.mark.django_db
@@ -111,11 +116,78 @@ def test_api_diagnostics_connection_token_for_authenticated_user(settings, clien
assert payload["exp"] - payload["nbf"] == 300
def test_api_diagnostics_connection_chosen_cluster(
settings, fake_livekit_room_creation
):
"""The connection is tested on the LiveKit cluster the placement strategy
picks, where the room is created, without any placement.
Browsers are given its URL, not the API URL Django calls it on.
"""
lyon = livekit_cluster_entry("lyon", api_url="http://livekit.lyon.svc:7880")
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("paris"), lyon])
with mock.patch("core.livekit.strategies.randbelow", return_value=1):
response = APIClient(REMOTE_ADDR=random_remote_addr()).post(
"/api/v1.0/diagnostics/connection/"
)
assert response.status_code == 200
config = response.json()["livekit"]
assert config["url"] == lyon["url"]
payload = jwt.decode(config["token"], lyon["api_secret"], algorithms=["HS256"])
assert payload["iss"] == lyon["api_key"]
assert payload["video"]["room"] == config["room"]
fake_livekit_room_creation.assert_called_once_with(
config["room"], get_livekit_cluster("lyon")
)
assert not Placement.objects.exists()
@mock.patch("core.api.viewsets.delete_connection_test_room.apply_async")
def test_api_diagnostics_connection_room_creation_failure(
mock_apply_async, settings, fake_livekit_room_creation
):
"""A connection cannot be tested for now when its LiveKit cluster fails to
create the room: no deletion is scheduled then."""
settings.CELERY_ENABLED = True
fake_livekit_room_creation.side_effect = LiveKitRoomCreationError("Timed out")
response = APIClient(REMOTE_ADDR=random_remote_addr()).post(
"/api/v1.0/diagnostics/connection/"
)
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
mock_apply_async.assert_not_called()
def test_api_diagnostics_connection_without_candidate(
settings, fake_livekit_room_creation
):
"""A connection cannot be tested for now when no LiveKit cluster may receive
new placements."""
settings.LIVEKIT_CLUSTERS = json.dumps([livekit_cluster_entry("paris", weight=0)])
response = APIClient(REMOTE_ADDR=random_remote_addr()).post(
"/api/v1.0/diagnostics/connection/"
)
assert response.status_code == 503
assert response.json() == {
"detail": "This room cannot be joined for now, please try again."
}
fake_livekit_room_creation.assert_not_called()
@mock.patch("core.api.viewsets.delete_connection_test_room.apply_async")
def test_api_diagnostics_connection_schedules_room_deletion(
mock_apply_async, settings, client
):
"""When Celery is enabled, schedule a hard room delete after max age."""
"""When Celery is enabled, schedule a hard room delete after max age, on the
LiveKit cluster the room was created on."""
settings.CELERY_ENABLED = True
settings.CONNECTION_TEST_TOKEN_TTL_SECONDS = 300
@@ -126,7 +198,7 @@ def test_api_diagnostics_connection_schedules_room_deletion(
assert response.status_code == 200
room = response.json()["livekit"]["room"]
mock_apply_async.assert_called_once_with(args=[room], countdown=310)
mock_apply_async.assert_called_once_with(args=[room, "default"], countdown=310)
@mock.patch("core.api.viewsets.delete_connection_test_room.apply_async")
@@ -0,0 +1,80 @@
"""
Unit tests for the Placement model
"""
from django.core.exceptions import ValidationError
from django.db import connection
import pytest
from core.factories import RoomFactory
from core.models import Placement
pytestmark = pytest.mark.django_db
def test_models_placements_str():
"""The str representation should name the LiveKit room and its cluster."""
placement = Placement.objects.create(room_name="my-room", cluster="paris")
assert str(placement) == "my-room on paris"
def test_models_placements_defaults():
"""A new placement has no room, no SID, no relocation, no absence and epoch 0."""
placement = Placement.objects.create(room_name="my-room", cluster="paris")
placement.refresh_from_db()
assert placement.room is None
assert placement.sid == ""
assert placement.epoch == 0
assert placement.previous_cluster == ""
assert placement.relocated_at is None
assert placement.absent_since is None
def test_models_placements_room_name_unique():
"""A LiveKit room has a single placement, also when writes skip validation."""
Placement.objects.create(room_name="my-room", cluster="paris")
with pytest.raises(ValidationError) as excinfo:
Placement.objects.create(room_name="my-room", cluster="lyon")
assert "Placement with this LiveKit room name already exists." in str(excinfo.value)
Placement.objects.bulk_create(
[Placement(room_name="my-room", cluster="lyon")], ignore_conflicts=True
)
assert Placement.objects.get().cluster == "paris"
def test_models_placements_room_deleted():
"""A placement outlives its room's record, and then points to no room."""
room = RoomFactory()
room_name = str(room.pk)
placement = Placement.objects.create(
room_name=room_name, room=room, cluster="paris"
)
room.delete()
placement.refresh_from_db()
assert placement.room is None
assert placement.room_name == room_name
assert placement.cluster == "paris"
def test_models_placements_room_deleted_without_the_collector():
"""The database unsets the room of a placement when a deletion bypasses Django."""
room = RoomFactory()
placement = Placement.objects.create(
room_name=str(room.pk), room=room, cluster="paris"
)
with connection.cursor() as cursor:
cursor.execute("DELETE FROM meet_room WHERE resource_id = %s", [room.pk])
cursor.execute("DELETE FROM meet_resource WHERE id = %s", [room.pk])
placement.refresh_from_db()
assert placement.room is None
assert placement.cluster == "paris"
@@ -2,7 +2,11 @@
Unit tests for the Recording model
"""
import logging
import uuid
from django.core.exceptions import ValidationError
from django.db import connection
import pytest
@@ -290,3 +294,69 @@ def test_models_recording_is_saved_false_error_states(status):
"""Test is_saved property returns False for error statuses."""
recording = RecordingFactory(status=status)
assert recording.is_saved is False
def test_models_recording_livekit_cluster_default():
"""A recording names no LiveKit cluster by default."""
recording = RecordingFactory()
assert recording.livekit_cluster == ""
def test_models_recording_livekit_cluster_unnamed_by_previous_release():
"""A rollback can still start recordings: the previous release inserts them
without naming a LiveKit cluster, which leaves them on the default cluster."""
room = RoomFactory()
recording_id = uuid.uuid4()
with connection.cursor() as cursor:
cursor.execute(
"INSERT INTO meet_recording (id, created_at, updated_at, room_id, "
"status, worker_id, mode, options, external_process_id) VALUES "
"(%s, NOW(), NOW(), %s, 'initiated', NULL, 'screen_recording', '{}', NULL)",
[recording_id, room.id],
)
assert Recording.objects.get(id=recording_id).livekit_cluster == ""
def test_models_recording_get_livekit_cluster(two_livekit_clusters):
"""A recording is on the registered LiveKit cluster it names."""
_paris, lyon = two_livekit_clusters
recording = RecordingFactory(livekit_cluster="lyon")
assert recording.get_livekit_cluster() == lyon
def test_models_recording_get_livekit_cluster_unnamed(two_livekit_clusters, caplog):
"""A recording naming no LiveKit cluster, as one started before recordings
kept theirs, is on the default cluster."""
paris, _lyon = two_livekit_clusters
recording = RecordingFactory(livekit_cluster="")
with caplog.at_level(logging.WARNING, logger="core.models"):
assert recording.get_livekit_cluster() == paris
assert "not registered" not in caplog.text
def test_models_recording_get_livekit_cluster_unregistered(
two_livekit_clusters, caplog
):
"""A recording on a LiveKit cluster no longer registered is on the default
cluster, which is logged."""
paris, _lyon = two_livekit_clusters
recording = RecordingFactory(livekit_cluster="lille")
with caplog.at_level(logging.WARNING, logger="core.models"):
assert recording.get_livekit_cluster() == paris
assert (
f"Recording {recording.id} is on LiveKit cluster lille, which is not "
"registered: managing it on the default cluster paris"
) in caplog.text
def test_models_recording_livekit_cluster_too_long():
"""A LiveKit cluster name is at most 63 characters long."""
with pytest.raises(ValidationError, match="livekit_cluster"):
RecordingFactory(livekit_cluster="a" * 64)
@@ -0,0 +1,103 @@
"""Unit tests for the validation of the LIVEKIT_CLUSTERS setting."""
import json
import re
from unittest import mock
import pytest
from core.tests.utils import livekit_cluster_entry
from meet import settings
def test_livekit_clusters_invalid_is_refused():
"""An invalid list of clusters stops the settings loading, naming the problem."""
clusters = json.dumps([livekit_cluster_entry("paris", weight=-1)])
with mock.patch.object(settings.Test, "LIVEKIT_CLUSTERS", clusters):
with pytest.raises(ValueError, match=re.escape("LIVEKIT_CLUSTERS[0].weight")):
settings.Test.post_setup()
def test_livekit_clusters_valid_is_accepted():
"""A valid list of clusters is accepted while telephony and roomkit are off."""
clusters = json.dumps(
[livekit_cluster_entry("paris"), livekit_cluster_entry("lyon")]
)
with (
mock.patch.object(settings.Test, "LIVEKIT_CLUSTERS", clusters),
mock.patch.object(settings.Test, "ROOM_TELEPHONY_ENABLED", False),
mock.patch.object(settings.Test, "ROOMKIT_ENABLED", False),
):
settings.Test.post_setup()
def test_livekit_clusters_unset_skips_validation():
"""Unset, as in the Build configuration which has no LiveKit environment.
LIVEKIT_CONFIGURATION, which the registry then falls back to, is not checked.
"""
with (
mock.patch.object(settings.Test, "LIVEKIT_CLUSTERS", None),
mock.patch.object(
settings.Test,
"LIVEKIT_CONFIGURATION",
{"url": None, "api_key": None, "api_secret": None},
),
):
settings.Test.post_setup()
@pytest.mark.parametrize(
"telephony,roomkit,enabled",
[
(True, False, "ROOM_TELEPHONY_ENABLED"),
(False, True, "ROOMKIT_ENABLED"),
(True, True, "ROOM_TELEPHONY_ENABLED and ROOMKIT_ENABLED"),
],
ids=["telephony", "roomkit", "both"],
)
def test_livekit_clusters_several_with_telephony_are_refused(
telephony, roomkit, enabled
):
"""Several clusters with telephony or roomkit stop the settings loading.
Phone and roomkit callers reach the SIP server of one cluster before their PIN
is known, so Django cannot route them. The message names the enabled settings.
"""
clusters = json.dumps(
[livekit_cluster_entry("paris"), livekit_cluster_entry("lyon")]
)
with (
mock.patch.object(settings.Test, "LIVEKIT_CLUSTERS", clusters),
mock.patch.object(settings.Test, "ROOM_TELEPHONY_ENABLED", telephony),
mock.patch.object(settings.Test, "ROOMKIT_ENABLED", roomkit),
):
with pytest.raises(
ValueError,
match=f"^{enabled} cannot be combined with more than one LiveKit "
"cluster, and LIVEKIT_CLUSTERS lists 2:",
):
settings.Test.post_setup()
@pytest.mark.parametrize(
"clusters",
[None, json.dumps([livekit_cluster_entry("paris")])],
ids=["unset", "listed"],
)
def test_livekit_clusters_single_with_telephony_is_accepted(clusters):
"""Telephony and roomkit work with a single cluster.
It is either listed in LIVEKIT_CLUSTERS or, when that is unset, built from
LIVEKIT_CONFIGURATION.
"""
with (
mock.patch.object(settings.Test, "LIVEKIT_CLUSTERS", clusters),
mock.patch.object(settings.Test, "ROOM_TELEPHONY_ENABLED", True),
mock.patch.object(settings.Test, "ROOMKIT_ENABLED", True),
):
settings.Test.post_setup()
@@ -1,4 +1,4 @@
"""Unit tests for the LIVEKIT_API_TIMEOUT_SECONDS setting."""
"""Unit tests for the timeouts of LiveKit API calls in the settings."""
from unittest import mock
@@ -12,3 +12,13 @@ def test_livekit_api_timeout_zero_is_refused():
with mock.patch.object(Test, "LIVEKIT_API_TIMEOUT_SECONDS", 0):
with pytest.raises(ValueError, match="LIVEKIT_API_TIMEOUT_SECONDS"):
Test.post_setup()
def test_livekit_create_room_timeout_zero_is_refused():
"""A zero timeout would fail every LiveKit room creation, and with it every
join, so it stops the settings loading."""
with mock.patch.object(Test, "LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS", 0):
with pytest.raises(
ValueError, match="LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS must be at least 1"
):
Test.post_setup()
+187 -1
View File
@@ -3,9 +3,12 @@ Test utils functions
"""
# pylint: disable=W0621
import asyncio
import contextlib
import json
import socket
import threading
import time
from unittest import mock
from django.conf import settings
@@ -14,18 +17,25 @@ from django.contrib.auth.models import AnonymousUser
import jwt
import pytest
from asgiref.sync import async_to_sync
from livekit.api import TwirpError
from livekit.api import ( # pylint: disable=no-name-in-module
DeleteRoomRequest,
TwirpError,
)
from core.factories import UserFactory
from core.livekit.clusters import get_livekit_cluster
from core.models import Placement
from core.services.participants_management import (
ParticipantsManagement,
ParticipantsManagementException,
)
from core.services.room_management import RoomManagement, RoomManagementException
from core.services.sip_management import SIPException, SIPManagement
from core.tests.utils import livekit_cluster_entry
from core.utils import (
NotificationError,
create_livekit_client,
generate_livekit_config,
generate_token,
notify_participants,
)
@@ -42,6 +52,20 @@ def decode_token(token: str) -> dict:
)
@pytest.fixture
def two_clusters(settings):
"""Register the Paris and Lyon LiveKit clusters, Paris being the default.
Django calls each of them on an API URL apart from the one browsers use.
"""
settings.LIVEKIT_CLUSTERS = json.dumps(
[
livekit_cluster_entry("paris", api_url="http://livekit.paris.svc:7880"),
livekit_cluster_entry("lyon", api_url="http://livekit.lyon.svc:7880"),
]
)
def test_generate_token_authenticated_uses_full_name():
"""The token's display name should default to the user's full name."""
user = UserFactory(full_name="Jane Doe")
@@ -116,6 +140,49 @@ def test_anonymous_falls_back_to_anonymous_label():
assert claims["name"] == "Anonymous"
@pytest.mark.usefixtures("two_clusters")
@pytest.mark.parametrize(
("name", "expected"), [(None, "paris"), ("paris", "paris"), ("lyon", "lyon")]
)
def test_generate_token_cluster(name, expected):
"""A token is signed with the API key and secret of its cluster.
It is for the default cluster when none is given.
"""
cluster = get_livekit_cluster(name) if name else None
expected_cluster = get_livekit_cluster(expected)
token = generate_token(room="my-room", user=AnonymousUser(), cluster=cluster)
claims = jwt.decode(token, expected_cluster.api_secret, algorithms=["HS256"])
assert claims["iss"] == expected_cluster.api_key
@pytest.mark.usefixtures("two_clusters")
@pytest.mark.parametrize("name", ["paris", "lyon"])
def test_generate_livekit_config_cluster(name):
"""Browsers get the URL of the cluster and a token signed for it."""
cluster = get_livekit_cluster(name)
config = generate_livekit_config(
room_id="my-room", user=AnonymousUser(), username="Guest42", cluster=cluster
)
assert config["url"] == cluster.url
assert config["room"] == "my-room"
claims = jwt.decode(config["token"], cluster.api_secret, algorithms=["HS256"])
assert claims["iss"] == cluster.api_key
assert claims["video"]["room"] == "my-room"
def test_generate_livekit_config_without_cluster():
"""No credentials are built without the cluster the LiveKit room is placed on."""
arguments = {"room_id": "my-room", "user": AnonymousUser(), "username": "Guest42"}
with pytest.raises(TypeError, match="cluster"):
generate_livekit_config(**arguments)
@pytest.fixture
def fake_livekit(request):
"""A LiveKit address that takes the connection, sends `request.param`, then waits."""
@@ -187,6 +254,63 @@ def test_create_livekit_client_lets_an_answer_through(fake_livekit, settings):
RoomManagement.delete_room("room-abc")
@pytest.fixture
def slow_livekit():
"""A LiveKit address that answers every call half a second after it is made."""
server = socket.create_server(("127.0.0.1", 0))
held = []
def answer_later(connection):
time.sleep(0.5)
with contextlib.suppress(OSError):
connection.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n")
def accept():
while True:
try:
connection, _ = server.accept()
except OSError:
return
held.append(connection)
threading.Thread(
target=answer_later, args=(connection,), daemon=True
).start()
threading.Thread(target=accept, daemon=True).start()
yield f"http://127.0.0.1:{server.getsockname()[1]:d}"
server.close()
for connection in held:
connection.close()
def test_create_livekit_client_timeout(slow_livekit, settings):
"""A client given a timeout gives up after it, other clients keep the set one."""
settings.LIVEKIT_CONFIGURATION = {
**settings.LIVEKIT_CONFIGURATION,
"url": slow_livekit,
}
settings.LIVEKIT_API_TIMEOUT_SECONDS = 5
async def delete_room(client):
try:
return await client.room.delete_room(DeleteRoomRequest(room="room-abc"))
finally:
await client.aclose()
@async_to_sync
async def delete_rooms():
patient = create_livekit_client()
hasty = create_livekit_client(timeout=0.1)
return await asyncio.gather(
delete_room(patient), delete_room(hasty), return_exceptions=True
)
patient, hasty = delete_rooms()
assert not isinstance(patient, BaseException)
assert isinstance(hasty, TimeoutError)
class TimingOutClient:
"""A LiveKit client whose every call times out."""
@@ -320,6 +444,34 @@ def test_create_livekit_client_custom_configuration():
assert url_of_new_client().startswith("http://mock-url.com")
@pytest.mark.usefixtures("two_clusters")
@pytest.mark.parametrize(("name", "expected"), [(None, "paris"), ("lyon", "lyon")])
def test_create_livekit_client_cluster(name, expected):
"""A client calls its cluster on the API URL, with the cluster's key and secret.
The default cluster is called when none is given.
"""
cluster = get_livekit_cluster(name) if name else None
expected_cluster = get_livekit_cluster(expected)
@async_to_sync
async def target_of_new_client():
client = create_livekit_client(cluster=cluster)
room_service = client.room
await client.aclose()
return (
room_service._client.host, # pylint: disable=protected-access
room_service.api_key,
room_service.api_secret,
)
assert target_of_new_client() == (
expected_cluster.api_url,
expected_cluster.api_key,
expected_cluster.api_secret,
)
@mock.patch("core.utils.create_livekit_client")
def test_notify_participants_error(mock_create_livekit_client):
"""Test participant notification with API error."""
@@ -420,3 +572,37 @@ def test_notify_participants_success(mock_create_livekit_client):
# Verify aclose was called
mock_api_instance.aclose.assert_called_once()
@mock.patch("core.utils.create_livekit_client")
def test_notify_participants_on_the_placed_cluster(
mock_create_livekit_client, two_livekit_clusters, django_assert_num_queries
):
"""Participants are notified on the LiveKit cluster their room is placed on."""
_paris, lyon = two_livekit_clusters
Placement.objects.create(room_name="room-number-1", cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(1):
notify_participants(room_name="room-number-1", notification_data={"foo": "foo"})
mock_create_livekit_client.assert_called_once_with(cluster=lyon)
@mock.patch("core.utils.create_livekit_client")
def test_notify_participants_on_the_given_cluster(
mock_create_livekit_client, two_livekit_clusters, django_assert_num_queries
):
"""Participants are notified on the cluster given, whatever the placement."""
paris, _lyon = two_livekit_clusters
Placement.objects.create(room_name="room-number-1", cluster="lyon")
mock_create_livekit_client.return_value = mock.AsyncMock()
with django_assert_num_queries(0):
notify_participants(
room_name="room-number-1",
notification_data={"foo": "foo"},
cluster=paris,
)
mock_create_livekit_client.assert_called_once_with(cluster=paris)
+30
View File
@@ -1,5 +1,6 @@
"""Shared helpers for tests in the Meet core application"""
import base64
import hashlib
import secrets
from datetime import datetime, timedelta, timezone
@@ -7,10 +8,13 @@ from datetime import datetime, timedelta, timezone
from django.conf import settings
import jwt
from livekit.api import AccessToken
from core.factories import ApplicationFactory
from core.models import ApplicationScope
CREATED_LIVEKIT_ROOM_SID = "RM_created"
def generate_user_access_token(user, application=None, **overrides):
"""Generate a valid user access JWT signed with the token secret.
@@ -57,6 +61,32 @@ def random_remote_addr():
)
def livekit_cluster_entry(name, **overrides):
"""Describe a LiveKit cluster as an entry of LIVEKIT_CLUSTERS.
Each cluster gets its own API key, and a secret long enough to sign tokens.
"""
return {
"name": name,
"url": f"wss://{name}.livekit.test",
"api_key": f"key-{name}",
"api_secret": f"secret-{name}-padded-for-minimum-len!-livekit",
**overrides,
}
def livekit_webhook_token(body, api_key, api_secret):
"""Return the token a LiveKit cluster sends with a webhook body.
It is signed with this API key and secret, and holds the body's SHA-256.
"""
token = AccessToken(api_key, api_secret)
token.claims.sha256 = base64.b64encode(
hashlib.sha256(body.encode()).digest()
).decode()
return token.to_jwt()
def transit_code_target(code):
"""Return the audit target of a code: its SHA-256, never the code itself."""
return {
+90 -22
View File
@@ -36,6 +36,8 @@ from livekit.api import ( # pylint: disable=E0611
VideoGrants,
)
from core.livekit.clusters import LiveKitClusterConfig, get_default_livekit_cluster
logger = logging.getLogger(__name__)
@@ -70,6 +72,8 @@ def generate_token( # noqa: PLR0917
role: Optional[str] = None,
participant_id: Optional[str] = None,
ttl: Optional[timedelta] = None,
*,
cluster: Optional[LiveKitClusterConfig] = None,
) -> str:
"""Generate a LiveKit access token for a user in a specific room.
@@ -86,11 +90,17 @@ def generate_token( # noqa: PLR0917
participant_id (Optional[str]): Stable identifier for anonymous users;
used as identity when user.is_anonymous.
ttl (Optional[timedelta]): Token validity duration. Defaults to LiveKit SDK default.
cluster (Optional[LiveKitClusterConfig]): The LiveKit cluster the token is for,
signed with its API key and secret. Defaults to the default
cluster.
Returns:
str: The LiveKit JWT access token.
"""
if cluster is None:
cluster = get_default_livekit_cluster()
is_admin_or_owner = role in ("owner", "administrator")
if is_admin_or_owner:
sources = settings.LIVEKIT_DEFAULT_SOURCES
@@ -124,10 +134,7 @@ def generate_token( # noqa: PLR0917
display_name = (username or default_username) if can_edit else default_username
token = (
AccessToken(
api_key=settings.LIVEKIT_CONFIGURATION["api_key"],
api_secret=settings.LIVEKIT_CONFIGURATION["api_secret"],
)
AccessToken(api_key=cluster.api_key, api_secret=cluster.api_secret)
.with_grants(video_grants)
.with_identity(identity)
.with_name(display_name)
@@ -153,6 +160,8 @@ def generate_livekit_config( # noqa: PLR0917
color: Optional[str] = None,
configuration: Optional[dict] = None,
participant_id: Optional[str] = None,
*,
cluster: LiveKitClusterConfig,
) -> dict:
"""Generate LiveKit configuration for room access.
@@ -165,9 +174,12 @@ def generate_livekit_config( # noqa: PLR0917
configuration (Optional[dict]): Room configuration dict that can override default settings.
participant_id (Optional[str]): Stable identifier for anonymous users;
used as identity when user.is_anonymous.
cluster (LiveKitClusterConfig): The LiveKit cluster to connect to,
the one the room's LiveKit room is placed on.
Returns:
dict: LiveKit configuration with URL, room and access token
dict: LiveKit configuration with the URL browsers connect to on the
cluster, the room and an access token signed for that cluster
"""
sources = None
@@ -175,7 +187,7 @@ def generate_livekit_config( # noqa: PLR0917
sources = configuration.get("can_publish_sources", None)
return {
"url": settings.LIVEKIT_CONFIGURATION["url"],
"url": cluster.url,
"room": room_id,
"token": generate_token(
room=room_id,
@@ -185,6 +197,7 @@ def generate_livekit_config( # noqa: PLR0917
sources=sources,
role=role,
participant_id=participant_id,
cluster=cluster,
),
}
@@ -219,16 +232,21 @@ def generate_s3_authorization_headers(key):
return request
async def bound_livekit_request(request, handler):
"""Give up on a LiveKit request, its answer included, after the set timeout.
def bound_livekit_requests(timeout: float):
"""Return a client middleware giving up on a LiveKit request, its answer
included, after `timeout` seconds.
The SDK passes timeout=None to aiohttp on every call, which means no limit,
so a timeout set on the session never applies.
"""
async with asyncio.timeout(settings.LIVEKIT_API_TIMEOUT_SECONDS):
response = await handler(request)
await response.read()
return response
async def bound_livekit_request(request, handler):
async with asyncio.timeout(timeout):
response = await handler(request)
await response.read()
return response
return bound_livekit_request
class LiveKitClient(LiveKitAPI):
@@ -239,16 +257,46 @@ class LiveKitClient(LiveKitAPI):
await self._session.close()
def create_livekit_client(custom_configuration=None):
"""Create and return a configured LiveKit API client."""
def livekit_api_configuration(cluster: Optional[LiveKitClusterConfig] = None) -> dict:
"""Return the URL, API key and secret Django calls a LiveKit cluster with.
The URL is the cluster's api_url, not the one browsers connect to. The
default cluster is used when none is given.
"""
if cluster is None:
cluster = get_default_livekit_cluster()
return {
"url": cluster.api_url,
"api_key": cluster.api_key,
"api_secret": cluster.api_secret,
}
def create_livekit_client(
custom_configuration: Optional[dict] = None,
*,
cluster: Optional[LiveKitClusterConfig] = None,
timeout: Optional[float] = None,
) -> LiveKitClient:
"""Create a LiveKit API client calling a LiveKit cluster.
Args:
custom_configuration (Optional[dict]): The URL, API key and secret to call
LiveKit with, in place of the cluster's.
cluster (Optional[LiveKitClusterConfig]): The LiveKit cluster to call.
Defaults to the default cluster.
timeout (Optional[float]): Seconds after which each request of this client
gives up. Defaults to LIVEKIT_API_TIMEOUT_SECONDS.
"""
configuration = custom_configuration or livekit_api_configuration(cluster)
if timeout is None:
timeout = settings.LIVEKIT_API_TIMEOUT_SECONDS
connector = None if settings.LIVEKIT_VERIFY_SSL else aiohttp.TCPConnector(ssl=False)
session = aiohttp.ClientSession(
connector=connector, middlewares=(bound_livekit_request,)
connector=connector, middlewares=(bound_livekit_requests(timeout),)
)
# Use default configuration if none provided
configuration = custom_configuration or settings.LIVEKIT_CONFIGURATION
return LiveKitClient(session=session, **configuration)
@@ -256,11 +304,31 @@ class NotificationError(Exception):
"""Notification delivery to room participants fails."""
@async_to_sync
async def notify_participants(room_name: str, notification_data: dict):
"""Send notification data to all participants in a LiveKit room."""
def notify_participants(
room_name: str,
notification_data: dict,
*,
cluster: Optional[LiveKitClusterConfig] = None,
):
"""Send notification data to all participants in a LiveKit room.
lkapi = create_livekit_client()
The LiveKit room is reached on `cluster`, by default the LiveKit cluster it
is placed on.
"""
# pylint: disable-next=import-outside-toplevel,cyclic-import
from core.livekit.placements import PlacementService # noqa: PLC0415
cluster = cluster or PlacementService.cluster_for(room_name)
_notify_participants(room_name, notification_data, cluster=cluster)
@async_to_sync
async def _notify_participants(
room_name: str, notification_data: dict, *, cluster: LiveKitClusterConfig
):
"""Send notification data to a LiveKit room on a LiveKit cluster."""
lkapi = create_livekit_client(cluster=cluster)
try:
room_response = await lkapi.room.list_rooms(
+49 -2
View File
@@ -29,6 +29,8 @@ from pydantic import ValidationError as PydanticValidationError
from sentry_sdk.integrations.django import DjangoIntegration
from sentry_sdk.integrations.logging import ignore_logger
from core.livekit.specs import parse_livekit_clusters
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = path.dirname(path.dirname(path.abspath(__file__)))
@@ -778,6 +780,15 @@ class Base(Configuration):
),
"url": values.Value(environ_name="LIVEKIT_API_URL", environ_prefix=None),
}
LIVEKIT_CLUSTERS = SecretFileValue(
None, environ_name="LIVEKIT_CLUSTERS", environ_prefix=None
)
LIVEKIT_PLACEMENT_STRATEGY = values.Value(
"core.livekit.strategies.WeightedRandomStrategy",
environ_name="LIVEKIT_PLACEMENT_STRATEGY",
environ_prefix=None,
)
LIVEKIT_FORCE_WSS_PROTOCOL = values.BooleanValue(
False, environ_name="LIVEKIT_FORCE_WSS_PROTOCOL", environ_prefix=None
)
@@ -851,6 +862,9 @@ class Base(Configuration):
LIVEKIT_API_TIMEOUT_SECONDS = values.PositiveIntegerValue(
10, environ_name="LIVEKIT_API_TIMEOUT_SECONDS", environ_prefix=None
)
LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS = values.PositiveIntegerValue(
3, environ_name="LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS", environ_prefix=None
)
# Regex to filter webhook events by room name. Only matching events are processed.
LIVEKIT_WEBHOOK_EVENTS_FILTER_REGEX = values.Value(
None, environ_name="LIVEKIT_WEBHOOK_EVENTS_FILTER_REGEX", environ_prefix=None
@@ -1593,6 +1607,33 @@ class Base(Configuration):
f"'{cls.AUDIT_LOG_DATA_STREAM_NAMESPACE}' must not contain '-'."
)
@classmethod
def _check_livekit_clusters(cls):
"""Ensure LIVEKIT_CLUSTERS, when set, lists valid LiveKit clusters.
Unset, the registry holds a single cluster built from
LIVEKIT_CONFIGURATION, so every source of clusters is covered.
Telephony and roomkit need a single cluster, so LIVEKIT_CLUSTERS may list
more than one only while both are disabled.
"""
if cls.LIVEKIT_CLUSTERS is None:
return
clusters = parse_livekit_clusters(cls.LIVEKIT_CLUSTERS)
enabled = [
name
for name in ("ROOM_TELEPHONY_ENABLED", "ROOMKIT_ENABLED")
if getattr(cls, name)
]
if len(clusters) > 1 and enabled:
raise ValueError(
f"{' and '.join(enabled)} cannot be combined with more than one "
f"LiveKit cluster, and LIVEKIT_CLUSTERS lists {len(clusters)}: "
"phone and roomkit callers reach the SIP server of one cluster "
"before their PIN is known, so Django cannot route them to the "
"cluster their room is placed on."
)
@classmethod
def _configure_database_pool(cls):
"""Inject the psycopg pool options into the default database when enabled."""
@@ -1653,9 +1694,14 @@ class Base(Configuration):
cls._check_recording_encoding_maps()
cls._check_audit_log_data_stream_namespace()
cls._configure_database_pool()
cls._check_livekit_clusters()
if cls.LIVEKIT_API_TIMEOUT_SECONDS < 1:
raise ValueError("LIVEKIT_API_TIMEOUT_SECONDS must be at least 1")
for timeout_setting in (
"LIVEKIT_API_TIMEOUT_SECONDS",
"LIVEKIT_CREATE_ROOM_TIMEOUT_SECONDS",
):
if getattr(cls, timeout_setting) < 1:
raise ValueError(f"{timeout_setting} must be at least 1")
if (
cls.SUMMARY_SERVICE_VERSION == 1
@@ -1821,6 +1867,7 @@ class Test(Base):
"api_secret": "secret-key-padded-for-minimum-len!-livekit",
"url": "http://127.0.0.1.nip.io:7880",
}
LIVEKIT_CLUSTERS = None
APPLICATION_ENABLED = True
APPLICATION_JWT_SECRET_KEY = "secret-key-padded-for-minimum-len!-application" # noqa:S105
@@ -31,7 +31,7 @@ import {
captureMediaEvent,
reportError,
} from '@/features/analytics/telemetry'
import { type ApiConfig, useConfig } from '@/api/useConfig'
import { useConfig } from '@/api/useConfig'
import { isFireFox } from '@/utils/livekit'
import { useIsMobile } from '@/utils/useIsMobile'
import { navigateTo } from '@/navigation/navigateTo'
@@ -46,6 +46,11 @@ import { VOICE_AUDIO_CONSTRAINTS } from '@/features/rooms/livekit/utils/constant
import { resolveAudioOutputDeviceId } from '@/features/rooms/utils/resolveAudioOutputDeviceId'
import { ConnectionEndReason } from '@/features/rooms/utils/connectionEndReason'
import { createLiveKitRoom } from '@/features/rooms/livekit/utils/reconnectPolicy'
import {
getUrlToWarmUp,
shouldConnect,
toConnectionUrl,
} from '@/features/rooms/livekit/utils/clusterConnection'
// Some clients (like DINUM) operate in bandwidth-constrained environments
// These settings help ensure successful connections in poor network conditions
@@ -79,13 +84,17 @@ const handleError = (e: Error) => {
*
* It should cache DNS and TLS keys.
*/
const warmUpConnection = async (room: Room, livekit: ApiConfig['livekit']) => {
await room.prepareConnection(livekit.url)
const warmUpConnection = async (
room: Room,
url: string,
enableFirefoxProxyWorkaround: boolean
) => {
await room.prepareConnection(url)
if (isFireFox() && livekit.enable_firefox_proxy_workaround) {
if (isFireFox() && enableFirefoxProxyWorkaround) {
try {
const wssUrl =
livekit.url.replace('https://', 'wss://').replace(/\/$/, '') + '/rtc'
url.replace('https://', 'wss://').replace(/\/$/, '') + '/rtc'
/**
* FIREFOX + PROXY WORKAROUND:
@@ -147,7 +156,11 @@ export const Conference = ({
const isMobile = useIsMobile()
const [hasJoined, setHasJoined] = useState(false)
const [isConnectionWarmedUp, setIsConnectionWarmedUp] = useState(false)
const [warmedUpUrls, setWarmedUpUrls] = useState<ReadonlySet<string>>(
() => new Set()
)
const [isConnectLatched, setIsConnectLatched] = useState(false)
const startedWarmUpsRef = useRef(new Set<string>())
const hasAutoMutedRef = useRef(false)
const hasConnectedRef = useRef(false)
@@ -212,19 +225,20 @@ export const Conference = ({
[roomOptions, apiConfig?.livekit.reconnect]
)
/*
* Ensure stable WebSocket connection URL. This is critical for legacy browser compatibility
* (Firefox <124, Chrome <125, Edge <125) where HTTPS URLs in WebSocket() constructor
* may fail - the force_wss_protocol flag allows explicit WSS protocol conversion
*/
const serverUrl = useMemo(() => {
const livekit_url = apiConfig?.livekit.url
if (!livekit_url) return
if (apiConfig?.livekit.force_wss_protocol) {
return livekit_url.replace('https://', 'wss://')
}
return livekit_url
}, [apiConfig?.livekit])
const connection = {
configUrl: apiConfig?.livekit.url,
joinUrl: createRoomData?.livekit?.url,
warmedUpUrls,
isConnectLatched,
}
const urlToWarmUp = getUrlToWarmUp(connection)
const connect = shouldConnect(connection)
const serverUrl =
connection.joinUrl &&
toConnectionUrl(
connection.joinUrl,
apiConfig?.livekit.force_wss_protocol ?? false
)
const handleConnected = useCallback(async () => {
hasConnectedRef.current = true
@@ -275,11 +289,19 @@ export const Conference = ({
}, [isLobbyRequired, onLobbyRequired])
useEffect(() => {
if (!apiConfig || isConnectionWarmedUp) return
warmUpConnection(room, apiConfig.livekit).then(() =>
setIsConnectionWarmedUp(true)
)
}, [room, apiConfig, isConnectionWarmedUp])
if (!apiConfig || !urlToWarmUp) return
if (startedWarmUpsRef.current.has(urlToWarmUp)) return
startedWarmUpsRef.current.add(urlToWarmUp)
warmUpConnection(
room,
urlToWarmUp,
apiConfig.livekit.enable_firefox_proxy_workaround
).then(() => setWarmedUpUrls((urls) => new Set(urls).add(urlToWarmUp)))
}, [room, apiConfig, urlToWarmUp])
useEffect(() => {
if (connect) setIsConnectLatched(true)
}, [connect])
if (fetchError?.statusCode == 410) {
return (
@@ -307,7 +329,7 @@ export const Conference = ({
room={room}
serverUrl={serverUrl}
token={createRoomData?.livekit?.token}
connect={isConnectionWarmedUp}
connect={connect}
audio={userConfig.audioEnabled}
video={
userConfig.videoEnabled && {
@@ -0,0 +1,151 @@
import { describe, expect, it } from 'vitest'
import {
getUrlToWarmUp,
shouldConnect,
toConnectionUrl,
} from './clusterConnection'
const DEFAULT_CLUSTER_URL = 'https://livekit.example.org'
const OTHER_CLUSTER_URL = 'https://livekit-b.example.org'
const NOTHING_WARMED_UP = new Set<string>()
describe('toConnectionUrl', () => {
it('switches an https URL to wss when the config forces it', () => {
expect(toConnectionUrl('https://livekit-b.example.org', true)).toBe(
'wss://livekit-b.example.org'
)
})
it('keeps the URL as given otherwise', () => {
expect(toConnectionUrl('https://livekit-b.example.org', false)).toBe(
'https://livekit-b.example.org'
)
expect(toConnectionUrl('wss://livekit-b.example.org', true)).toBe(
'wss://livekit-b.example.org'
)
expect(toConnectionUrl('http://localhost:7880', true)).toBe(
'http://localhost:7880'
)
})
})
describe('getUrlToWarmUp', () => {
it('waits for the config', () => {
expect(
getUrlToWarmUp({
configUrl: undefined,
joinUrl: OTHER_CLUSTER_URL,
isConnectLatched: false,
})
).toBeUndefined()
})
it('warms up the default cluster while the join request is in flight', () => {
expect(
getUrlToWarmUp({
configUrl: DEFAULT_CLUSTER_URL,
joinUrl: undefined,
isConnectLatched: false,
})
).toBe(DEFAULT_CLUSTER_URL)
})
it('warms up the cluster the room is placed on once the join answered', () => {
expect(
getUrlToWarmUp({
configUrl: DEFAULT_CLUSTER_URL,
joinUrl: OTHER_CLUSTER_URL,
isConnectLatched: false,
})
).toBe(OTHER_CLUSTER_URL)
})
it('keeps the default cluster when the room is placed on it', () => {
expect(
getUrlToWarmUp({
configUrl: DEFAULT_CLUSTER_URL,
joinUrl: DEFAULT_CLUSTER_URL,
isConnectLatched: false,
})
).toBe(DEFAULT_CLUSTER_URL)
})
it('warms up nothing once connect is on', () => {
for (const joinUrl of [undefined, DEFAULT_CLUSTER_URL, OTHER_CLUSTER_URL]) {
expect(
getUrlToWarmUp({
configUrl: DEFAULT_CLUSTER_URL,
joinUrl,
isConnectLatched: true,
})
).toBeUndefined()
}
})
})
describe('shouldConnect', () => {
it('waits for the join answer, even once the default cluster is warmed up', () => {
expect(
shouldConnect({
joinUrl: undefined,
warmedUpUrls: new Set([DEFAULT_CLUSTER_URL]),
isConnectLatched: false,
})
).toBe(false)
})
it('waits for the warm-up of the cluster the room is placed on', () => {
for (const warmedUpUrls of [
NOTHING_WARMED_UP,
new Set([DEFAULT_CLUSTER_URL]),
]) {
expect(
shouldConnect({
joinUrl: OTHER_CLUSTER_URL,
warmedUpUrls,
isConnectLatched: false,
})
).toBe(false)
}
})
it('connects once the cluster the room is placed on is warmed up', () => {
expect(
shouldConnect({
joinUrl: OTHER_CLUSTER_URL,
warmedUpUrls: new Set([DEFAULT_CLUSTER_URL, OTHER_CLUSTER_URL]),
isConnectLatched: false,
})
).toBe(true)
})
it('connects on the warm-up of the default cluster when the room is placed on it', () => {
const connection = {
joinUrl: DEFAULT_CLUSTER_URL,
isConnectLatched: false,
}
expect(
shouldConnect({ ...connection, warmedUpUrls: NOTHING_WARMED_UP })
).toBe(false)
expect(
shouldConnect({
...connection,
warmedUpUrls: new Set([DEFAULT_CLUSTER_URL]),
})
).toBe(true)
})
it('never stops once connect is on, whatever the credentials become', () => {
for (const joinUrl of [undefined, DEFAULT_CLUSTER_URL, OTHER_CLUSTER_URL]) {
expect(
shouldConnect({
joinUrl,
warmedUpUrls: NOTHING_WARMED_UP,
isConnectLatched: true,
})
).toBe(true)
}
})
})
@@ -0,0 +1,51 @@
/**
* The LiveKit credentials of a room give the URL of the LiveKit cluster the
* room is placed on, while the config endpoint gives the default cluster's URL
* earlier on. With a single cluster, both are the same URL.
*/
export type ClusterConnection = {
/** The default cluster's URL, once the config is known */
configUrl?: string
/** The URL of the cluster the room is placed on, from its credentials */
joinUrl?: string
/** The URLs whose warm-up is over */
warmedUpUrls: ReadonlySet<string>
/** Whether connect was already turned on */
isConnectLatched: boolean
}
/**
* Ensure stable WebSocket connection URL. This is critical for legacy browser
* compatibility (Firefox <124, Chrome <125, Edge <125) where HTTPS URLs in
* WebSocket() constructor may fail - the force_wss_protocol flag allows
* explicit WSS protocol conversion
*/
export const toConnectionUrl = (url: string, forceWssProtocol: boolean) =>
forceWssProtocol ? url.replace('https://', 'wss://') : url
/**
* The URL to warm up before connecting: the default cluster's as soon as the
* config is known, while the join request is in flight, then the URL of the
* cluster the room is placed on. None once connect is on, as connecting then
* waits for no warm-up.
*/
export const getUrlToWarmUp = ({
configUrl,
joinUrl,
isConnectLatched,
}: Omit<ClusterConnection, 'warmedUpUrls'>) => {
if (configUrl === undefined || isConnectLatched) return undefined
return joinUrl ?? configUrl
}
/**
* Whether LiveKitRoom may connect: once the URL of the cluster the room is
* placed on is warmed up, and from then on for good, since turning connect off
* disconnects the room and stops the local tracks.
*/
export const shouldConnect = ({
joinUrl,
warmedUpUrls,
isConnectLatched,
}: Omit<ClusterConnection, 'configUrl'>) =>
isConnectLatched || (joinUrl !== undefined && warmedUpUrls.has(joinUrl))
@@ -29,6 +29,9 @@ livekit:
urls:
- https://meet.127.0.0.1.nip.io/api/v1.0/rooms/webhooks-livekit/
room:
auto_create: false
loadBalancer:
type: nginx
annotations:
@@ -29,6 +29,9 @@ livekit:
urls:
- https://meet.127.0.0.1.nip.io/api/v1.0/rooms/webhooks-livekit/
room:
auto_create: false
loadBalancer:
type: nginx
annotations: