Compare commits

..

2 Commits

Author SHA1 Message Date
leo f1e0bd7ac5 wip temp 2026-09-01 19:07:34 +02:00
leo 35d2bae0f0 (backend) add per-recording encoding config to start-recording API
Add new options to query start-start recording API. A resolution
("540p", "720p", "1080p") and a profile ("talking_heads", "text", "mixed")
are resolved to provide a width, height, fps and bitrate which
are passed on to the encoder. Using profiles allows for some flexibility
on quality if necessary without changing front facing user config.

Co-authored-by: sarthakbahal <sarthakbahal.45@gmail.com>
2026-09-01 17:35:42 +02:00
38 changed files with 872 additions and 1841 deletions
+5 -7
View File
@@ -10,15 +10,14 @@ and this project adheres to
### Added
- ✨(frontend) add 1080p sending resolution option #1660
- ✨(backend) add Traefik support via configurable media-auth url header #1649
### Fixed
- 🐛(frontend) keep the sending resolution picked while the camera is off #1667
- ✨(backend) add per-recording encoding quality presets to start-recording API
## [1.30.0] - 2026-09-01
### Changed
- ♿️(frontend) close side panel with Escape key #1507
### Added
- ✨(agent) support Voxtral realtime as inference engine
@@ -34,7 +33,6 @@ and this project adheres to
- ⬆️(frontend) upgrade @fontsource/opendyslexic from 5.2.5 to 5.3.0
- ⬆️(addons) upgrade core-js from 3.49.0 to 3.50.0
- ♻️(backend) factorize s3 client creation in utils
- ♿️(frontend) close side panel with Escape key #1507
### Fixed
+86
View File
@@ -16,6 +16,92 @@ the following command inside your docker container:
## [Unreleased]
### Recording encoding settings replaced by a resolution/profile model
The `RECORDING_ENCODING_*` settings introduced in v1.16.0 exposed raw encoder
values (width, height, framerate, bitrate). They are replaced by two named and configurable sets of
dimensions, a **resolution** (default: `540p`, `720p`, `1080p`) and a **profile**
(default: `talking_heads`, `text`, `mixed`, `full`), which are resolved to the width, height,
fps and video bitrate.
**The following environment variables are no longer read. If they are still set in
your deployment they are silently ignored, and your recordings will be encoded with
the new defaults instead of your tuned values.**
| Removed variable | Replaced by |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `RECORDING_ENCODING_ENABLED` | Nothing. A default encoding is now always built (see below). **Not** `RECORDING_CUSTOM_ENCODING_ENABLED`, which gates a different feature. |
| `RECORDING_ENCODING_WIDTH` | The `width` of the entry selected by `RECORDING_ENCODING_DEFAULT_RESOLUTION` in `RECORDING_ENCODING_AVAILABLE_RESOLUTIONS`. |
| `RECORDING_ENCODING_HEIGHT` | The `height` of that same entry. |
| `RECORDING_ENCODING_FRAMERATE` | The `fps` of the profile selected by `RECORDING_ENCODING_DEFAULT_PROFILE` in `RECORDING_ENCODING_AVAILABLE_PROFILES`. |
| `RECORDING_ENCODING_VIDEO_BITRATE_KBPS` | That profile's `kbps`. |
`RECORDING_ENCODING_AUDIO_BITRATE_KBPS` and `RECORDING_ENCODING_KEY_FRAME_INTERVAL_S`
are unchanged and keep their values.
#### If you never set `RECORDING_ENCODING_ENABLED=True`
No action is required. The shipped defaults (`RECORDING_ENCODING_DEFAULT_PROFILE=full`,
`RECORDING_ENCODING_DEFAULT_RESOLUTION=720p`) match LiveKit's built-in
`H264_720P_30` preset: 1280×720, 30 fps, 3000 kbps H.264 MAIN, 128 kbps AAC.
Note that these values are now sent explicitly as advanced `EncodingOptions`
rather than relying on LiveKit's preset, so `RECORDING_ENCODING_AUDIO_BITRATE_KBPS`
and `RECORDING_ENCODING_KEY_FRAME_INTERVAL_S` now apply to every recording. They
previously applied only when `RECORDING_ENCODING_ENABLED` was `True`.
To keep letting LiveKit pick the encoding instead, set either default to an empty
value:
```
RECORDING_ENCODING_DEFAULT_RESOLUTION=
RECORDING_ENCODING_DEFAULT_PROFILE=
```
#### If you had tuned `RECORDING_ENCODING_*` values
Translate your old values into a default resolution and a default profile. Declare your own resolution and/or profile. Both maps are read from the
environment as a single-line Python/JSON dict literal (parsed with
`ast.literal_eval`, so use double-quoted keys and no trailing commas, and do not
add outer quotes in `.env`-style files):
```bash
RECORDING_ENCODING_AVAILABLE_RESOLUTIONS={"540p": {"width": 960, "height": 540}, "720p": {"width": 1280, "height": 720}, "1080p": {"width": 1920, "height": 1080}}
RECORDING_ENCODING_AVAILABLE_PROFILES={"my_old_profile": {"fps": 15, "kbps": {"540p": 350, "720p": 600, "1080p": 1100}}}
RECORDING_ENCODING_DEFAULT_RESOLUTION=720p
RECORDING_ENCODING_DEFAULT_PROFILE=my_old_profile
```
Two constraints are validated at startup and may raise a `ValueError`:
- every profile in `RECORDING_ENCODING_AVAILABLE_PROFILES` must define a `kbps`
entry for **exactly** the keys of `RECORDING_ENCODING_AVAILABLE_RESOLUTIONS`;
overriding one of the two maps usually means overriding both;
- `RECORDING_ENCODING_DEFAULT_RESOLUTION` and `RECORDING_ENCODING_DEFAULT_PROFILE`,
when non-empty, must be keys of their respective map.
#### Optional: per-recording encoding
`RECORDING_CUSTOM_ENCODING_ENABLED` (default `False`) toggles whether the
start-recording API accepts an `encoding` object
(`{"resolution": "720p", "profile": "talking_heads"}`, `profile` optional) that
overrides the default for a single recording. It does not enable or disable the
default encoding, which is built from the two `RECORDING_ENCODING_DEFAULT_*`
settings either way. Leaving it at `False` preserves the previous behaviour, where
every recording uses the server-side encoding: requests carrying
`options.encoding` are rejected with a `400` before the recording is created, so
nothing is persisted and no egress is started.
Before enabling it:
- clients can only pick keys you declared; there is no way to send a raw width or bitrate
- as of this implementation, the frontend never sends `encoding`
- `encoding` is accepted but ignored for `transcript` recordings, whose audio-only
egress has no video encoding to configure.
See [docs/features/recording.md](docs/features/recording.md#tuning-recording-encoding)
for the full setting reference, the shipped profile table and the tuning caveats.
## v1.30.0
### Removing S3 storage-event webhooks for recordings
-2
View File
@@ -213,8 +213,6 @@ services:
- "7880:7880"
- "7881:7881"
- "7882:7882/udp"
- "3478:3478/udp"
- "30000-30100:30000-30100/udp"
volumes:
- ./docker/livekit/config/livekit-server.yaml:/config.yaml
depends_on:
-1
View File
@@ -65,7 +65,6 @@ RUN apk update && apk upgrade \
musl \
musl-utils \
zlib>=1.3.2-r0 \
libexpat>=2.8.4-r0 \
&& apk del curl
USER nginx
-13
View File
@@ -8,16 +8,3 @@ webhook:
api_key: devkey
urls:
- http://app-dev:8000/api/v1.0/rooms/webhooks-livekit/
turn:
enabled: true
domain: turn.127.0.0.1.nip.io
udp_port: 3478
tls_port: 0
external_tls: false
relay_range_start: 30000
relay_range_end: 30100
allow_restricted_peer_cidrs:
- 192.168.0.0/16
- 172.16.0.0/12
+22 -8
View File
@@ -69,17 +69,31 @@ SUMMARY_SERVICE_WEBHOOK_API_TOKEN=webhook-password
RECORDING_DOWNLOAD_BASE_URL=http://localhost:3000/recording
# Recording encoding (LiveKit Egress advanced options).
# When RECORDING_ENCODING_ENABLED is False (default), LiveKit uses its built-in
# H264_720P_30 preset (1280x720, 30fps, 3000 kbps). Enable and tune to reduce
# file size and CPU load on the egress worker.
# RECORDING_ENCODING_ENABLED=False
# RECORDING_ENCODING_WIDTH=1280
# RECORDING_ENCODING_HEIGHT=720
# RECORDING_ENCODING_FRAMERATE=30
# RECORDING_ENCODING_VIDEO_BITRATE_KBPS=3000
# Encoding is described by a named resolution (width/height) and a named profile
# (framerate + video bitrate per resolution) instead of raw encoder values. The
# start-recording API accepts a pair per recording, e.g.
# options.encoding={"resolution": "720p", "profile": "talking_heads"}; only keys
# declared in the two maps below are accepted, "profile" is optional.
# Both maps are read as a one-line Python dict literal (ast.literal_eval): double
# quoted keys, no outer quotes, no trailing comma. Every profile must define a
# kbps entry for exactly the resolutions of the resolutions map, or startup fails.
# RECORDING_ENCODING_AVAILABLE_RESOLUTIONS={"540p": {"width": 960, "height": 540}, "720p": {"width": 1280, "height": 720}, "1080p": {"width": 1920, "height": 1080}}
# RECORDING_ENCODING_AVAILABLE_PROFILES={"talking_heads": {"fps": 15, "kbps": {"540p": 400, "720p": 700, "1080p": 1200}}, "text": {"fps": 15, "kbps": {"540p": 600, "720p": 1000, "1080p": 1800}}, "mixed": {"fps": 20, "kbps": {"540p": 900, "720p": 1500, "1080p": 2500}}, "full": {"fps": 30, "kbps": {"540p": 2000, "720p": 3000, "1080p": 4500}}}
# Defaults for recordings that don't carry an encoding. Must be keys of the maps
# above. Declared but not yet read by the recording code at this commit.
# RECORDING_ENCODING_DEFAULT_RESOLUTION=720p
# RECORDING_ENCODING_DEFAULT_PROFILE=full
# Applied to every resolved encoding, independent of resolution and profile.
# RECORDING_ENCODING_AUDIO_BITRATE_KBPS=128
# RECORDING_ENCODING_KEY_FRAME_INTERVAL_S=4.0
# Server-wide encoding used when a recording carries no encoding of its own.
# Keep False: when True, the worker factory still reads RECORDING_ENCODING_WIDTH,
# _HEIGHT, _FRAMERATE and _VIDEO_BITRATE_KBPS, which no longer exist.
# RECORDING_ENCODING_ENABLED=False
# Telephony
ROOM_TELEPHONY_ENABLED=True
+50 -2
View File
@@ -13,7 +13,12 @@ from django.core.exceptions import SuspiciousOperation
from django.utils.translation import gettext_lazy as _
from django_pydantic_field.rest_framework import SchemaField
from pydantic import BaseModel, Field, field_serializer
from pydantic import (
BaseModel,
Field,
field_serializer,
field_validator,
)
from pydantic import ValidationError as PydanticValidationError
from rest_framework import serializers
from rest_framework.exceptions import PermissionDenied
@@ -244,6 +249,49 @@ class BaseValidationOnlySerializer(serializers.Serializer):
raise NotImplementedError(f"{self.__class__.__name__} is validation-only")
class EncodingConfig(BaseModel):
"""Configuration options for recording encoding.
The allowed `resolution` and `profile` values are derived at validation time
from ``settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS`` and
``settings.RECORDING_ENCODING_AVAILABLE_PROFILES``, so adding a resolution or profile
to those maps is enough to make it accepted here.
Attributes:
resolution: Target video resolution.
profile: Encoding profile to balance quality and CPU usage. When `None`,
LiveKit default framerate/bitrate are used for the resolution.
"""
resolution: str
profile: str | None = None
model_config = {"extra": "forbid"}
@field_validator("resolution")
@classmethod
def _validate_resolution(cls, value):
"""Reject resolutions absent from RECORDING_ENCODING_AVAILABLE_RESOLUTIONS."""
allowed = set(settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS)
if value not in allowed:
raise ValueError(
f"Invalid resolution '{value}'. Choose from {sorted(allowed)}."
)
return value
@field_validator("profile")
@classmethod
def _validate_profile(cls, value):
"""Reject profiles absent from RECORDING_ENCODING_AVAILABLE_PROFILES."""
if value is None:
return None
allowed = set(settings.RECORDING_ENCODING_AVAILABLE_PROFILES)
if value not in allowed:
raise ValueError(
f"Invalid profile '{value}'. Choose from {sorted(allowed)}."
)
return value
class RecordingOptions(BaseModel):
"""Configuration options for recording.
@@ -264,7 +312,7 @@ class RecordingOptions(BaseModel):
transcribe: bool | None = None
collect_metadata: bool | None = None
original_mode: Literal["screen_recording", "transcript"] | None = None
encoding: EncodingConfig | None = None
model_config = {"extra": "forbid"}
+17 -18
View File
@@ -60,6 +60,7 @@ from core.recording.worker.factories import (
from core.recording.worker.mediator import (
WorkerServiceMediator,
)
from core.recording.worker.services import resolve_encoding_config
from core.services.invitation import InvitationService
from core.services.livekit_events import (
LiveKitEventsService,
@@ -400,12 +401,20 @@ class RoomViewSet(
options = serializer.validated_data.get("options")
room = self.get_object()
options_data = options.model_dump(exclude_none=True) if options else {}
if options is not None and options.encoding is not None:
# Persist the resolved encoding (concrete width/height/framerate/
# bitrate) alongside the requested resolution/profile for traceability.
options_data["encoding"]["resolved"] = resolve_encoding_config(
options.encoding
)
try:
with transaction.atomic():
recording = models.Recording.objects.create(
room=room,
mode=mode,
options=options.model_dump(exclude_none=True) if options else {},
options=options_data,
)
models.RecordingAccess.objects.create(
user=self.request.user,
@@ -1076,10 +1085,9 @@ class RecordingViewSet(
def _auth_get_original_url(self, request):
"""
Extracts and parses the original URL from the configured header.
Extracts and parses the original URL from the "HTTP_X_ORIGINAL_URL" header.
Raises PermissionDenied if the header is missing.
The original url is passed by the reverse proxy in the header named by the
MEDIA_AUTH_ORIGINAL_URL_HEADER setting, which defaults to "HTTP_X_ORIGINAL_URL".
The original url is passed by nginx in the "HTTP_X_ORIGINAL_URL" header.
See corresponding ingress configuration in Helm chart and read about the
nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress
is configured to do this.
@@ -1089,13 +1097,9 @@ class RecordingViewSet(
reasons.
"""
# Extract the original URL from the request header
original_url = request.META.get(settings.MEDIA_AUTH_ORIGINAL_URL_HEADER)
original_url = request.META.get("HTTP_X_ORIGINAL_URL")
if not original_url:
logger.warning(
"Missing %s header in subrequest. Set MEDIA_AUTH_ORIGINAL_URL_HEADER "
"to the header your reverse proxy sends.",
settings.MEDIA_AUTH_ORIGINAL_URL_HEADER,
)
logger.warning("Missing HTTP_X_ORIGINAL_URL header in subrequest")
raise drf_exceptions.PermissionDenied()
logger.debug("Original url: '%s'", original_url)
@@ -1420,8 +1424,7 @@ class FileViewSet(
Authorize access based on the original URL of an Nginx subrequest
and user permissions. Returns a dictionary of URL parameters if authorized.
The original url is passed by the reverse proxy in the header named by the
MEDIA_AUTH_ORIGINAL_URL_HEADER setting, which defaults to "HTTP_X_ORIGINAL_URL".
The original url is passed by nginx in the "HTTP_X_ORIGINAL_URL" header.
See corresponding ingress configuration in Helm chart and read about the
nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress
is configured to do this.
@@ -1440,13 +1443,9 @@ class FileViewSet(
- PermissionDenied if authorization fails.
"""
# Extract the original URL from the request header
original_url = request.META.get(settings.MEDIA_AUTH_ORIGINAL_URL_HEADER)
original_url = request.META.get("HTTP_X_ORIGINAL_URL")
if not original_url:
logger.warning(
"Missing %s header in subrequest. Set MEDIA_AUTH_ORIGINAL_URL_HEADER "
"to the header your reverse proxy sends.",
settings.MEDIA_AUTH_ORIGINAL_URL_HEADER,
)
logger.warning("Missing HTTP_X_ORIGINAL_URL header in subrequest")
raise drf_exceptions.PermissionDenied()
parsed_url = urlparse(original_url)
+45 -17
View File
@@ -22,6 +22,37 @@ _RECORDING_AUDIO_CODEC = livekit_api.AudioCodec.AAC
_RECORDING_AUDIO_FREQUENCY_HZ = 48000
def _build_default_encoding_options() -> Optional[Dict[str, Any]]:
"""Build the server-wide EncodingOptions kwargs, or None to keep LiveKit's preset.
Operator-tunable values live in Django settings; the default resolution gives
width / height, the default profile gives framerate and video bitrate, while
codec and frequency are pinned constants. Either default left empty means we
use the livekit defaults.
"""
resolution = settings.RECORDING_ENCODING_DEFAULT_RESOLUTION
profile = settings.RECORDING_ENCODING_DEFAULT_PROFILE
if not resolution or not profile:
return None
dimensions = settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS[resolution]
profile_spec = settings.RECORDING_ENCODING_AVAILABLE_PROFILES[profile]
return {
"width": dimensions["width"],
"height": dimensions["height"],
"framerate": profile_spec["fps"],
"video_bitrate": profile_spec["kbps"][resolution],
"audio_bitrate": settings.RECORDING_ENCODING_AUDIO_BITRATE_KBPS,
"key_frame_interval": settings.RECORDING_ENCODING_KEY_FRAME_INTERVAL_S,
"video_codec": _RECORDING_VIDEO_CODEC,
"audio_codec": _RECORDING_AUDIO_CODEC,
"audio_frequency": _RECORDING_AUDIO_FREQUENCY_HZ,
}
@dataclass(frozen=True)
class WorkerServiceConfig:
"""Declare Worker Service common configurations"""
@@ -38,22 +69,14 @@ class WorkerServiceConfig:
logger.debug("Loading WorkerServiceConfig from settings.")
encoding_options: Optional[Dict[str, Any]] = None
if settings.RECORDING_ENCODING_ENABLED:
# Single source of truth for the EncodingOptions kwargs:
# operator-tunable values live in Django settings, codec / frequency
# are pinned constants. The services layer only unpacks this dict.
encoding_options = {
"width": settings.RECORDING_ENCODING_WIDTH,
"height": settings.RECORDING_ENCODING_HEIGHT,
"framerate": settings.RECORDING_ENCODING_FRAMERATE,
"video_bitrate": settings.RECORDING_ENCODING_VIDEO_BITRATE_KBPS,
"audio_bitrate": settings.RECORDING_ENCODING_AUDIO_BITRATE_KBPS,
"key_frame_interval": settings.RECORDING_ENCODING_KEY_FRAME_INTERVAL_S,
"video_codec": _RECORDING_VIDEO_CODEC,
"audio_codec": _RECORDING_AUDIO_CODEC,
"audio_frequency": _RECORDING_AUDIO_FREQUENCY_HZ,
}
# Single source of truth for the EncodingOptions kwargs; the services
# layer only unpacks this dict. Recordings carrying their own encoding
# resolve it per request and bypass this default.
encoding_options: Optional[Dict[str, Any]] = (
_build_default_encoding_options()
if settings.RECORDING_ENCODING_ENABLED
else None
)
return cls(
output_folder=settings.RECORDING_OUTPUT_FOLDER,
@@ -78,7 +101,12 @@ class WorkerService(Protocol):
def __init__(self, config: WorkerServiceConfig):
"""Initialize the service with the given configuration."""
def start(self, room_id: str, recording_id: str) -> str:
def start(
self,
room_id: str,
recording_id: str,
encoding_options: Optional[Dict[str, Any]] = None,
) -> str:
"""Start a recording for a specified room."""
def stop(self, worker_id: str) -> str:
@@ -51,8 +51,11 @@ class WorkerServiceMediator:
raise RecordingStartError()
room_name = str(recording.room.id)
encoding_options = (recording.options.get("encoding") or {}).get("resolved")
try:
worker_id = self._worker_service.start(room_name, recording.id)
worker_id = self._worker_service.start(
room_name, recording.id, encoding_options=encoding_options
)
except (WorkerRequestError, WorkerConnectionError, WorkerResponseError) as e:
logger.exception(
"Failed to start recording for room %s: %s", recording.room.slug, e
+61 -5
View File
@@ -2,6 +2,8 @@
# pylint: disable=no-member
from django.conf import settings
from asgiref.sync import async_to_sync
from livekit import api as livekit_api
@@ -11,6 +13,41 @@ from .exceptions import WorkerConnectionError, WorkerResponseError
from .factories import WorkerServiceConfig
def resolve_encoding_config(encoding_config):
"""Resolve a per-recording EncodingConfig to concrete encoding fields.
Returns a JSON-serializable dict of the LiveKit ``EncodingOptions`` kwargs
derived from the request's resolution / profile, or None when no
encoding_config is provided. This allows to derive width, height, fps, and
bitrate from (resolution, profile).
Only the fields that can actually be resolved are included: width/height
require a resolution, framerate/video_bitrate require both a resolution and a
profile.
"""
if encoding_config is None:
return None
resolution = encoding_config.resolution
profile = encoding_config.profile
resolved = {
"key_frame_interval": settings.RECORDING_ENCODING_KEY_FRAME_INTERVAL_S,
}
if resolution:
dimensions = settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS[resolution]
resolved["width"] = dimensions["width"]
resolved["height"] = dimensions["height"]
if resolution and profile:
profile_spec = settings.RECORDING_ENCODING_AVAILABLE_PROFILES[profile]
resolved["framerate"] = profile_spec["fps"]
resolved["video_bitrate"] = profile_spec["kbps"][resolution]
return resolved
class BaseEgressService:
"""Base egress defining common methods to manage and interact with LiveKit egress processes."""
@@ -76,7 +113,7 @@ class BaseEgressService:
return "FAILED_TO_STOP"
def start(self, room_name, recording_id):
def start(self, room_name, recording_id, encoding_options=None):
"""Start the egress process for a recording (not implemented in the base class).
Each derived class must implement this method, providing the necessary parameters for
its specific egress type (e.g. audio_only, streaming output).
@@ -99,13 +136,24 @@ class BaseEgressService:
return livekit_api.EncodingOptions(**opts)
def _resolve_encoding_options(self, encoding_options):
"""Build LiveKit EncodingOptions from a resolved per-recording dict, or None.
``encoding_options`` is the dict persisted by the API in
``recording.options["encoding"]["resolved"]``.
"""
if not encoding_options:
return None
return livekit_api.EncodingOptions(**encoding_options)
class VideoCompositeEgressService(BaseEgressService):
"""Record multiple participant video and audio tracks into a single output '.mp4' file."""
hrid = "video-recording-composite-livekit-egress"
def start(self, room_name, recording_id):
def start(self, room_name, recording_id, encoding_options=None):
"""Start the video composite egress process for a recording."""
# Save room's recording as a mp4 video file.
@@ -126,7 +174,10 @@ class VideoCompositeEgressService(BaseEgressService):
"layout": "speaker-light",
}
advanced = self._build_encoding_options()
advanced = (
self._resolve_encoding_options(encoding_options)
or self._build_encoding_options()
)
if advanced is not None:
request_kwargs["advanced"] = advanced
@@ -145,8 +196,13 @@ class AudioCompositeEgressService(BaseEgressService):
hrid = "audio-recording-composite-livekit-egress"
def start(self, room_name, recording_id):
"""Start the audio composite egress process for a recording."""
def start(self, room_name, recording_id, encoding_options=None):
"""Start the audio composite egress process for a recording.
``encoding_options`` is accepted for signature compatibility with the
WorkerService protocol but ignored: audio-only egress has no
encoding to configure.
"""
# Save room's recording as an ogg audio file.
file_type = livekit_api.EncodedFileType.OGG
@@ -7,7 +7,6 @@ from urllib.parse import quote, urlparse
from django.conf import settings
from django.core.files.storage import default_storage
from django.test import override_settings
from django.utils import timezone
import pytest
@@ -144,59 +143,3 @@ def test_api_files_media_auth_own_file_deleted():
)
assert response.status_code == 403
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_files_media_auth_custom_original_url_header():
"""
Authorization should honour the configured original-url header.
Covers the attachment subrequest path, which resolves the header separately
from the recording one. Reverse proxies other than nginx-ingress use
different headers: Traefik's ForwardAuth sends X-Forwarded-Uri and cannot
emit X-Original-URL at all.
"""
user = factories.UserFactory()
file = factories.FileFactory(
type=models.FileTypeChoices.BACKGROUND_IMAGE,
update_upload_state=models.FileUploadStateChoices.READY,
creator=user,
)
client = APIClient()
client.force_login(user)
default_storage.save(file.file_key, BytesIO(b"my prose"))
original_url = f"http://localhost/media/{file.file_key:s}"
response = client.get(
"/api/v1.0/files/media-auth/", HTTP_X_FORWARDED_URI=original_url
)
assert response.status_code == 200
assert "AWS4-HMAC-SHA256 Credential=" in response["Authorization"]
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_files_media_auth_default_header_ignored_when_reconfigured():
"""
Only the configured header should be honoured, never a hardcoded fallback.
"""
user = factories.UserFactory()
file = factories.FileFactory(
type=models.FileTypeChoices.BACKGROUND_IMAGE,
update_upload_state=models.FileUploadStateChoices.READY,
creator=user,
)
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/{file.file_key:s}"
response = client.get(
"/api/v1.0/files/media-auth/", HTTP_X_ORIGINAL_URL=original_url
)
assert response.status_code == 403
@@ -8,7 +8,6 @@ from uuid import uuid4
from django.conf import settings
from django.core.files.storage import default_storage
from django.test import override_settings
from django.utils import timezone
import pytest
@@ -283,63 +282,3 @@ def test_api_recordings_media_auth_success_administrator(mode):
timeout=1,
)
assert response.content.decode("utf-8") == "my prose"
def test_api_recordings_media_auth_missing_header():
"""
Test that a subrequest without the configured original-url header is rejected.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
response = client.get("/api/v1.0/recordings/media-auth/")
assert response.status_code == 403
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_recordings_media_auth_custom_original_url_header():
"""
Test that the header carrying the original URL can be configured.
Reverse proxies other than nginx-ingress use different headers: Traefik's
ForwardAuth sends X-Forwarded-Uri and cannot emit X-Original-URL at all.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/recordings/{uuid4()!s}.mp4"
response = client.get(
"/api/v1.0/recordings/media-auth/", HTTP_X_FORWARDED_URI=original_url
)
# The header was read and parsed: we get as far as looking the recording up,
# rather than being rejected for a missing header.
assert response.status_code == 404
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_recordings_media_auth_default_header_ignored_when_reconfigured():
"""
Test that only the configured header is honoured.
Guards against the header being read from a hardcoded name in parallel with
the setting.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/recordings/{uuid4()!s}.mp4"
response = client.get(
"/api/v1.0/recordings/media-auth/", HTTP_X_ORIGINAL_URL=original_url
)
assert response.status_code == 403
@@ -0,0 +1,109 @@
"""Tests for the per-recording encoding resolution in BaseEgressService."""
# pylint: disable=protected-access,redefined-outer-name,unused-argument
from unittest.mock import Mock
from django.conf import settings
import pytest
from pydantic import ValidationError as PydanticValidationError
from core.api.serializers import EncodingConfig
from core.recording.worker.services import (
VideoCompositeEgressService,
resolve_encoding_config,
)
def make_config():
"""Build a minimal WorkerServiceConfig-like mock for service instantiation."""
config = Mock()
config.bucket_args = {
"endpoint": "https://s3.test.com",
"access_key": "test_key",
"secret": "test_secret",
"region": "test-region",
"bucket": "test-bucket",
"force_path_style": True,
}
config.encoding_options = None
return config
@pytest.fixture
def service():
"""Return a VideoCompositeEgressService with mocked handle_request."""
svc = VideoCompositeEgressService(make_config())
svc._handle_request = Mock()
return svc
# --- resolve_encoding_config ---
def test_resolve_config_returns_none_without_config():
"""Resolver should return None when no encoding config is provided."""
assert resolve_encoding_config(None) is None
def test_resolve_config_without_profile_omits_profile_fields():
"""A resolution-only config should resolve dimensions but no framerate/bitrate."""
resolved = resolve_encoding_config(EncodingConfig(resolution="720p"))
assert resolved == {
"key_frame_interval": settings.RECORDING_ENCODING_KEY_FRAME_INTERVAL_S,
"width": 1280,
"height": 720,
}
def test_encoding_config_requires_resolution():
"""A profile-only or empty encoding config should be rejected at validation."""
with pytest.raises(PydanticValidationError):
EncodingConfig(profile="mixed")
with pytest.raises(PydanticValidationError):
EncodingConfig()
# --- _resolve_encoding_options ---
@pytest.mark.parametrize("encoding_options", [None, {}])
def test_resolve_options_returns_none_when_empty(service, encoding_options):
"""Resolver should return None when the resolved dict is empty or missing."""
assert service._resolve_encoding_options(encoding_options) is None
@pytest.mark.parametrize(
"resolution",
list(settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS),
)
@pytest.mark.parametrize(
"profile",
list(settings.RECORDING_ENCODING_AVAILABLE_PROFILES),
)
def test_resolve_profile_resolution_combinations(service, profile, resolution):
"""Every (profile, resolution) pair should resolve to the values from settings."""
dimensions = settings.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS[resolution]
profile_spec = settings.RECORDING_ENCODING_AVAILABLE_PROFILES[profile]
resolved = resolve_encoding_config(
EncodingConfig(resolution=resolution, profile=profile)
)
result = service._resolve_encoding_options(resolved)
assert result.width == dimensions["width"]
assert result.height == dimensions["height"]
assert result.framerate == profile_spec["fps"]
assert result.video_bitrate == profile_spec["kbps"][resolution]
def test_resolve_options_none_profile_uses_livekit_defaults(service):
"""Missing profile should pass 0 fps/bitrate (LiveKit protobuf default)."""
resolved = resolve_encoding_config(EncodingConfig(resolution="720p"))
result = service._resolve_encoding_options(resolved)
assert result.width == 1280
assert result.height == 720
assert result.framerate == 0
assert result.video_bitrate == 0
@@ -85,18 +85,22 @@ def test_config_immutability(default_config):
AWS_S3_REGION_NAME="test-region",
AWS_STORAGE_BUCKET_NAME="test-bucket",
RECORDING_ENCODING_ENABLED=True,
RECORDING_ENCODING_WIDTH=1280,
RECORDING_ENCODING_HEIGHT=720,
RECORDING_ENCODING_FRAMERATE=15,
RECORDING_ENCODING_VIDEO_BITRATE_KBPS=600,
RECORDING_ENCODING_AVAILABLE_RESOLUTIONS={
"720p": {"width": 1280, "height": 720},
},
RECORDING_ENCODING_AVAILABLE_PROFILES={
"talking_heads": {"fps": 15, "kbps": {"720p": 600}},
},
RECORDING_ENCODING_DEFAULT_RESOLUTION="720p",
RECORDING_ENCODING_DEFAULT_PROFILE="talking_heads",
RECORDING_ENCODING_AUDIO_BITRATE_KBPS=64,
RECORDING_ENCODING_KEY_FRAME_INTERVAL_S=10.0,
)
def test_config_encoding_options_enabled():
"""When RECORDING_ENCODING_ENABLED is True, encoding options are populated.
The dict mixes operator-tunable values from settings with pinned codec /
frequency constants, so the services layer can simply unpack it.
The dict mixes values resolved from the default resolution / profile with
pinned codec / frequency constants, so the services layer can simply unpack it.
"""
WorkerServiceConfig.from_settings.cache_clear()
@@ -115,6 +119,27 @@ def test_config_encoding_options_enabled():
}
@override_settings(
RECORDING_OUTPUT_FOLDER="/test/output",
LIVEKIT_CONFIGURATION={"server": "test.example.com"},
AWS_S3_ENDPOINT_URL="https://s3.test.com",
AWS_S3_ACCESS_KEY_ID="test_key",
AWS_S3_SECRET_ACCESS_KEY="test_secret",
AWS_S3_REGION_NAME="test-region",
AWS_STORAGE_BUCKET_NAME="test-bucket",
RECORDING_ENCODING_ENABLED=True,
RECORDING_ENCODING_DEFAULT_RESOLUTION="",
RECORDING_ENCODING_DEFAULT_PROFILE="",
)
def test_config_encoding_options_without_defaults():
"""An empty default resolution or profile leaves the encoding to LiveKit."""
WorkerServiceConfig.from_settings.cache_clear()
config = WorkerServiceConfig.from_settings()
assert config.encoding_options is None
@override_settings(
RECORDING_OUTPUT_FOLDER="/test/output",
LIVEKIT_CONFIGURATION={"server": "test.example.com"},
@@ -50,7 +50,7 @@ def test_start_recording_success(mock_update_metadata, mediator, mock_worker_ser
# Verify worker service call
expected_room_name = str(mock_recording.room.id)
mock_worker_service.start.assert_called_once_with(
expected_room_name, mock_recording.id
expected_room_name, mock_recording.id, encoding_options=None
)
# Verify recording updates
@@ -64,6 +64,38 @@ def test_start_recording_success(mock_update_metadata, mediator, mock_worker_ser
)
@mock.patch("core.utils.update_room_metadata")
def test_start_recording_passes_resolved_encoding(
mock_update_room_metadata, mediator, mock_worker_service
):
"""The resolved encoding persisted in recording.options reaches the worker."""
mock_worker_service.start.return_value = "test-worker-123"
resolved = {
"key_frame_interval": 4.0,
"width": 1280,
"height": 720,
"framerate": 15,
"video_bitrate": 700,
}
mock_recording = RecordingFactory(
status=RecordingStatusChoices.INITIATED,
worker_id=None,
options={
"encoding": {
"resolution": "720p",
"profile": "talking_heads",
"resolved": resolved,
}
},
)
mediator.start(mock_recording)
mock_worker_service.start.assert_called_once_with(
str(mock_recording.room.id), mock_recording.id, encoding_options=resolved
)
@pytest.mark.parametrize(
"error_class", [WorkerRequestError, WorkerConnectionError, WorkerResponseError]
)
@@ -470,6 +470,160 @@ def test_start_recording_options_unknown_field_rejected(settings):
assert response.status_code == 400
def test_start_recording_options_encoding_valid(
settings, mock_worker_service_factory, mock_worker_manager
):
"""Should accept a valid encoding configuration."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{
"mode": "screen_recording",
"options": {"encoding": {"resolution": "720p", "profile": "talking_heads"}},
},
format="json",
)
assert response.status_code == 201
def test_start_recording_persists_resolved_encoding(
settings, mock_worker_service_factory, mock_worker_manager
):
"""The resolved encoding should be persisted in recording.options alongside
the requested resolution/profile for traceability."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{
"mode": "screen_recording",
"options": {"encoding": {"resolution": "720p", "profile": "talking_heads"}},
},
format="json",
)
assert response.status_code == 201
recording = Recording.objects.get(room=room)
assert recording.options["encoding"] == {
"resolution": "720p",
"profile": "talking_heads",
"resolved": {
"key_frame_interval": settings.RECORDING_ENCODING_KEY_FRAME_INTERVAL_S,
"width": 1280,
"height": 720,
"framerate": 15,
"video_bitrate": 700,
},
}
def test_start_recording_forwards_resolved_encoding_to_worker(
settings, mock_worker_service, mock_worker_service_factory
):
"""The resolved encoding should passed on to the worker."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
mock_worker_service.start.return_value = "egress-123"
with mock.patch("core.utils.update_room_metadata"):
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{
"mode": "screen_recording",
"options": {
"encoding": {"resolution": "720p", "profile": "talking_heads"}
},
},
format="json",
)
assert response.status_code == 201
recording = Recording.objects.get(room=room)
mock_worker_service.start.assert_called_once_with(
str(room.id),
recording.id,
encoding_options=recording.options["encoding"]["resolved"],
)
def test_start_recording_options_encoding_invalid_resolution(settings):
"""Should reject invalid encoding resolution values."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{"mode": "screen_recording", "options": {"encoding": {"resolution": "4K"}}},
format="json",
)
assert response.status_code == 400
def test_start_recording_options_encoding_unknown_key_rejected(settings):
"""Should reject unknown keys in encoding configuration."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{
"mode": "screen_recording",
"options": {"encoding": {"bitrate": 9000}},
},
format="json",
)
assert response.status_code == 400
def test_start_recording_options_without_encoding_unchanged(
settings, mock_worker_service_factory, mock_worker_manager
):
"""Requests without encoding should keep existing options behavior."""
settings.RECORDING_ENABLE = True
room = RoomFactory()
user = UserFactory()
room.accesses.create(user=user, role="owner")
client = APIClient()
client.force_login(user)
response = client.post(
f"/api/v1.0/rooms/{room.id}/start-recording/",
{"mode": "screen_recording", "options": {"language": "fr"}},
format="json",
)
assert response.status_code == 201
recording = Recording.objects.get(room=room)
assert recording.options == {"language": "fr"}
@pytest.mark.parametrize("value", ["foo", 12])
def test_start_recording_options_invalid_transcribe_type(settings, value):
"""Should reject non-boolean transcribe values."""
+98 -24
View File
@@ -129,15 +129,6 @@ class Base(Configuration):
MEDIA_BASE_URL = values.Value(
"", environ_name="MEDIA_BASE_URL", environ_prefix=None
)
# Header the reverse proxy uses to pass the original request URL to the
# media-auth subrequest views. nginx-ingress sends X-Original-URL, which is
# the default. Other proxies use different headers -- Traefik's ForwardAuth,
# for instance, sends X-Forwarded-Uri and cannot emit X-Original-URL at all.
MEDIA_AUTH_ORIGINAL_URL_HEADER = values.Value(
default="HTTP_X_ORIGINAL_URL",
environ_name="MEDIA_AUTH_ORIGINAL_URL_HEADER",
environ_prefix=None,
)
SITE_ID = 1
@@ -756,26 +747,63 @@ class Base(Configuration):
# recordings), whose request never carries advanced EncodingOptions.
# When disabled, LiveKit falls back to its built-in H264_720P_30 preset
# (1280x720, 30 fps, 3000 kbps H.264 MAIN video, 128 kbps AAC audio).
# When enabled, the values below are passed to LiveKit as EncodingOptions
# (advanced) and replace the preset. Lowering framerate and bitrate reduces
# output file size and CPU load on the egress worker.
# When enabled, the encoding parameters are resolved from the default profile
# and resolution below and passed to LiveKit as EncodingOptions (advanced),
# replacing the preset. Lowering framerate and bitrate reduces output file
# size and CPU load on the egress worker.
RECORDING_ENCODING_ENABLED = values.BooleanValue(
False, environ_name="RECORDING_ENCODING_ENABLED", environ_prefix=None
)
RECORDING_ENCODING_WIDTH = values.PositiveIntegerValue(
1280, environ_name="RECORDING_ENCODING_WIDTH", environ_prefix=None
)
RECORDING_ENCODING_HEIGHT = values.PositiveIntegerValue(
720, environ_name="RECORDING_ENCODING_HEIGHT", environ_prefix=None
)
RECORDING_ENCODING_FRAMERATE = values.PositiveIntegerValue(
30, environ_name="RECORDING_ENCODING_FRAMERATE", environ_prefix=None
)
RECORDING_ENCODING_VIDEO_BITRATE_KBPS = values.PositiveIntegerValue(
3000,
environ_name="RECORDING_ENCODING_VIDEO_BITRATE_KBPS",
# Map resolution name -> {"width": ..., "height": ...} in pixels.
RECORDING_ENCODING_AVAILABLE_RESOLUTIONS = values.DictValue(
{
"540p": {"width": 960, "height": 540},
"720p": {"width": 1280, "height": 720},
"1080p": {"width": 1920, "height": 1080},
},
environ_name="RECORDING_ENCODING_AVAILABLE_RESOLUTIONS",
environ_prefix=None,
)
# Bitrate scales with resolution so quality stays consistent across sizes.
RECORDING_ENCODING_AVAILABLE_PROFILES = values.DictValue(
{
"talking_heads": {
"fps": 15,
"kbps": {"540p": 400, "720p": 700, "1080p": 1200},
},
"text": {
"fps": 15,
"kbps": {"540p": 600, "720p": 1000, "1080p": 1800},
},
"mixed": {
"fps": 20,
"kbps": {"540p": 900, "720p": 1500, "1080p": 2500},
},
"full": {
"fps": 30,
"kbps": {"540p": 2000, "720p": 3000, "1080p": 4500},
},
},
environ_name="RECORDING_ENCODING_AVAILABLE_PROFILES",
environ_prefix=None,
)
# Defaults used when no profile/resolution is specified per recording.
# Must be keys of the two dicts above (validated at startup).
RECORDING_ENCODING_DEFAULT_PROFILE = values.Value(
"full",
environ_name="RECORDING_ENCODING_DEFAULT_PROFILE",
environ_prefix=None,
)
RECORDING_ENCODING_DEFAULT_RESOLUTION = values.Value(
"720p",
environ_name="RECORDING_ENCODING_DEFAULT_RESOLUTION",
environ_prefix=None,
)
# Settings independent of profile/resolution.
RECORDING_ENCODING_AUDIO_BITRATE_KBPS = values.PositiveIntegerValue(
128,
environ_name="RECORDING_ENCODING_AUDIO_BITRATE_KBPS",
@@ -790,6 +818,7 @@ class Base(Configuration):
SUMMARY_SERVICE_VERSION = values.PositiveIntegerValue(
1, environ_name="SUMMARY_SERVICE_VERSION", environ_prefix=None
)
SUMMARY_SERVICE_ENDPOINT = values.Value(
None, environ_name="SUMMARY_SERVICE_ENDPOINT", environ_prefix=None
)
@@ -1178,6 +1207,49 @@ class Base(Configuration):
},
}
@classmethod
def _check_recording_encoding_maps(cls):
"""Ensure the per-recording encoding maps are mutually consistent.
Every profile in RECORDING_ENCODING_AVAILABLE_PROFILES must declare an fps and
a bitrate for each resolution declared in RECORDING_ENCODING_AVAILABLE_RESOLUTIONS,
and each non-empty default must name an entry of its map.
"""
# DictValue resolves to a dict at runtime; pylint sees the descriptor.
# pylint: disable=no-member
resolutions = set(cls.RECORDING_ENCODING_AVAILABLE_RESOLUTIONS)
for profile, spec in cls.RECORDING_ENCODING_AVAILABLE_PROFILES.items():
if "fps" not in spec or "kbps" not in spec:
raise ValueError(
f"Profile '{profile}' in RECORDING_ENCODING_AVAILABLE_PROFILES must "
"define both 'fps' and 'kbps'."
)
profile_resolutions = set(spec["kbps"])
if profile_resolutions != resolutions:
raise ValueError(
f"Profile '{profile}' in RECORDING_ENCODING_AVAILABLE_PROFILES must "
"define a bitrate for exactly the resolutions in "
"RECORDING_ENCODING_AVAILABLE_RESOLUTIONS, mismatch on: "
f"{resolutions ^ profile_resolutions}"
)
default_resolution = cls.RECORDING_ENCODING_DEFAULT_RESOLUTION
if default_resolution and default_resolution not in resolutions:
raise ValueError(
f"RECORDING_ENCODING_DEFAULT_RESOLUTION '{default_resolution}' is not a "
"key of RECORDING_ENCODING_AVAILABLE_RESOLUTIONS, choose from "
f"{sorted(resolutions)}."
)
profiles = set(cls.RECORDING_ENCODING_AVAILABLE_PROFILES)
default_profile = cls.RECORDING_ENCODING_DEFAULT_PROFILE
if default_profile and default_profile not in profiles:
raise ValueError(
f"RECORDING_ENCODING_DEFAULT_PROFILE '{default_profile}' is not a key of "
f"RECORDING_ENCODING_AVAILABLE_PROFILES, choose from {sorted(profiles)}."
)
@classmethod
def post_setup(cls):
"""Post setup configuration.
@@ -1191,6 +1263,8 @@ class Base(Configuration):
"FILE_UPLOAD_TMP_PATH cannot be the same as FILE_UPLOAD_PATH"
)
cls._check_recording_encoding_maps()
if (
cls.SUMMARY_SERVICE_VERSION == 1
and cls.SUMMARY_SERVICE_ENDPOINT is not None
-1
View File
@@ -53,7 +53,6 @@ RUN apk update && apk upgrade \
musl \
musl-utils \
zlib>=1.3.2-r0 \
libexpat>=2.8.4-r0 \
&& apk del curl
USER nginx
@@ -8,28 +8,6 @@ const IGNORED_EXCEPTION_PATTERNS = [
// the close reason is already logged by the SDK.
// See: https://github.com/livekit/client-sdk-js/issues/2062
/^Event captured as exception with keys: isTrusted$/,
// MediaPipe's WASM writes its native logs to stderr, which Emscripten
// routes to console.error, which PostHog's console capture then promotes
// to an $exception — even though nothing was thrown. Two flavors:
//
// 1. "INFO: ..." lines are purely informational. In particular
// "INFO: Created TensorFlow Lite XNNPACK delegate for CPU." is a
// SUCCESS message: TFLite prints it when it lazily initializes CPU
// inference on the first segmented frame. It fires on every effects
// init, on every browser and delegate (the GPU delegate still
// instantiates the CPU/XNNPACK delegate for non-delegated ops), so it
// was our single noisiest "error" while carrying zero signal.
/^INFO: /,
//
// 2. absl-formatted log lines, e.g.
// "E0901 19:21:45.443000 1880752 gl_graph_runner_internal.cc:260]
// StartGraph failed: ..."
// (severity letter E/W/I/F + MMDD + timestamp). These are the stderr
// *copies* of failures that MediaPipe also raises as real JS
// exceptions, which we already capture via reportError / thrown
// errors. Dropping them de-duplicates each incident (previously
// counted 2-3x) without losing the actual error report.
/^[EWIF]\d{4} \d{2}:\d{2}:\d{2}\./,
]
const shouldIgnoreException = (value: unknown): boolean =>
@@ -1,595 +0,0 @@
import { CSSProperties, useState } from 'react'
import { useConnectionState, useRoomContext } from '@livekit/components-react'
import { ConnectionState } from 'livekit-client'
import { StatsSnapshot, TrackRow, useWebRTCStats } from './useWebRTCStats'
import { readRoomConfig } from './roomConfig'
import {
forceTransport,
releaseForcedTransport,
Scenario,
SCENARIOS,
setDownlinkCap,
setUplinkCap,
stepBitrate,
TransportMode,
transportModeFromRoute,
} from './simulation'
const SANS =
"-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif"
const MONO = 'ui-monospace, SFMono-Regular, Menlo, Consolas, monospace'
const COLOR = {
text: '#e8e8e8',
muted: '#9a9a9a',
faint: '#6f6f6f',
hairline: '#2c2d31',
border: '#3d3e44',
surface: '#1b1c1e',
chartBg: '#141517',
down: '#f6821f',
up: '#5a9cf8',
ok: '#10b981',
bad: '#f05a4a',
busy: '#e5a13c',
}
const stateColor = (state: ConnectionState): string => {
switch (state) {
case ConnectionState.Connected:
return COLOR.ok
case ConnectionState.Reconnecting:
case ConnectionState.SignalReconnecting:
return COLOR.busy
case ConnectionState.Connecting:
return COLOR.up
case ConnectionState.Disconnected:
return COLOR.bad
}
}
const styles: Record<string, CSSProperties> = {
toggle: {
position: 'fixed',
bottom: 12,
right: 12,
zIndex: 9999,
fontFamily: SANS,
fontSize: 12,
lineHeight: 1,
padding: '8px 12px',
borderRadius: 999,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
background: COLOR.surface,
color: COLOR.text,
cursor: 'pointer',
boxShadow: '0 2px 10px #0006',
},
panel: {
position: 'fixed',
bottom: 12,
right: 12,
zIndex: 9999,
width: 460,
maxWidth: 'calc(100vw - 24px)',
maxHeight: 'calc(100vh - 24px)',
overflowY: 'auto',
fontFamily: SANS,
fontSize: 11,
lineHeight: 1.5,
color: COLOR.text,
background: COLOR.surface,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
borderRadius: 10,
padding: 14,
boxShadow: '0 10px 34px #00000080',
},
sectionTitle: {
margin: '14px 0 5px',
color: COLOR.faint,
textTransform: 'uppercase',
letterSpacing: 1.2,
fontSize: 9,
fontWeight: 600,
display: 'flex',
justifyContent: 'space-between',
},
metricsLine: {
display: 'flex',
alignItems: 'center',
gap: 16,
margin: '0 0 10px',
paddingBottom: 10,
borderBottom: `1px solid ${COLOR.hairline}`,
fontVariantNumeric: 'tabular-nums',
},
legend: {
display: 'flex',
gap: 12,
marginTop: 3,
color: COLOR.faint,
fontSize: 10,
},
list: {
display: 'flex',
flexDirection: 'column',
maxHeight: '400px',
overflowY: 'auto',
},
configList: { display: 'flex', flexWrap: 'wrap' },
configRow: {
display: 'flex',
gap: 6,
flex: '0 0 50%',
minWidth: 0,
boxSizing: 'border-box',
padding: '1px 8px 1px 0',
},
configLabel: {
color: COLOR.muted,
flex: '0 0 45%',
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
configValue: {
flex: 1,
minWidth: 0,
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
trackRow: {
display: 'flex',
gap: 8,
padding: '3px 0',
borderBottom: `1px solid ${COLOR.hairline}`,
alignItems: 'baseline',
},
trackLabel: {
flex: '0 0 34%',
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
trackCodec: {
flex: '0 0 11%',
fontFamily: MONO,
fontSize: 10,
color: COLOR.muted,
},
trackKbps: {
flex: '0 0 9%',
textAlign: 'right',
fontVariantNumeric: 'tabular-nums',
},
trackDetail: {
flex: 1,
minWidth: 0,
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
mono: { fontFamily: MONO, fontSize: 10, color: COLOR.muted },
button: {
fontFamily: SANS,
fontSize: 11,
padding: '3px 9px',
borderRadius: 5,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
background: '#232428',
color: COLOR.text,
cursor: 'pointer',
},
buttonActive: { borderColor: COLOR.down, color: COLOR.down },
buttonDisabled: { color: COLOR.faint, cursor: 'not-allowed' },
row: { display: 'flex', alignItems: 'center', gap: 6, flexWrap: 'wrap' },
stepperValue: {
minWidth: 92,
textAlign: 'center',
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.hairline,
borderRadius: 4,
padding: '2px 6px',
background: COLOR.chartBg,
fontVariantNumeric: 'tabular-nums',
},
close: {
background: 'none',
border: 'none',
color: COLOR.faint,
cursor: 'pointer',
fontSize: 14,
},
}
const Sparkline = ({ history }: { history: StatsSnapshot[] }) => {
const width = 430
const height = 54
const max = Math.max(...history.map((s) => Math.max(s.upKbps, s.downKbps)), 1)
const points = (pick: (s: StatsSnapshot) => number) =>
history
.map(
(s, i) =>
`${((i / Math.max(history.length - 1, 1)) * width).toFixed(1)},${(
height -
(pick(s) / max) * (height - 6) -
3
).toFixed(1)}`
)
.join(' ')
return (
<>
<svg
width={width}
height={height}
role="img"
aria-label="Bandwidth over the last minute"
style={{
display: 'block',
background: COLOR.chartBg,
borderRadius: 6,
border: `1px solid ${COLOR.hairline}`,
}}
>
{history.length >= 2 && (
<>
<polyline
points={points((s) => s.downKbps)}
fill="none"
stroke={COLOR.down}
strokeWidth="1.5"
/>
<polyline
points={points((s) => s.upKbps)}
fill="none"
stroke={COLOR.up}
strokeWidth="1.5"
/>
</>
)}
</svg>
{/* Legend outside the plot: text over the lines was unreadable. */}
<div style={styles.legend}>
<span>
<span style={{ color: COLOR.down }}></span> down
</span>
<span>
<span style={{ color: COLOR.up }}></span> up
</span>
<span>last 60s · max {Math.round(max)} kbps</span>
</div>
</>
)
}
const CapStepper = ({
label,
hint,
valueKbps,
onChange,
}: {
label: string
hint: string
valueKbps: number | null
onChange: (kbps: number | null) => void
}) => (
<div style={styles.row}>
<span style={{ flex: '0 0 70px' }}>{label}</span>
<button
type="button"
style={styles.button}
aria-label={`Decrease ${label} bandwidth`}
onClick={() => onChange(stepBitrate(valueKbps, 'decrease'))}
>
</button>
<span
style={{
...styles.stepperValue,
...(valueKbps !== null ? { color: COLOR.down } : {}),
}}
>
{valueKbps === null ? 'unlimited' : `${valueKbps} kbps`}
</span>
<button
type="button"
style={styles.button}
aria-label={`Increase ${label} bandwidth`}
onClick={() => onChange(stepBitrate(valueKbps, 'increase'))}
>
+
</button>
<span style={styles.mono}>{hint}</span>
</div>
)
const TrackList = ({ title, rows }: { title: string; rows: TrackRow[] }) => (
<>
<div style={styles.sectionTitle}>
<span>
{title} ({rows.length})
</span>
<span>
{Math.round(rows.reduce((sum, t) => sum + t.kbps, 0))} kbps media
</span>
</div>
<div style={styles.list}>
{rows.map((track) => (
<div key={track.key} style={styles.trackRow}>
<span style={styles.trackLabel}>{track.label}</span>
<span style={styles.trackCodec}>{track.codec ?? ''}</span>
<span style={styles.trackKbps}>{Math.round(track.kbps)}</span>
<span style={styles.trackDetail}>
{[
track.res,
track.fps !== undefined && `${Math.round(track.fps)}fps`,
]
.filter(Boolean)
.join(' · ')}
</span>
</div>
))}
{rows.length === 0 && <span style={{ color: COLOR.faint }}>none</span>}
</div>
</>
)
const MeetDevtools = () => {
const room = useRoomContext()
const connState = useConnectionState(room)
const [open, setOpen] = useState(false)
const [firedScenario, setFiredScenario] = useState<string>()
const [uplinkCap, setUplinkCapState] = useState<number | null>(null)
const [downlinkCap, setDownlinkCapState] = useState<number | null>(null)
const { snapshot, history } = useWebRTCStats(room, open)
const transportMode: TransportMode = transportModeFromRoute(snapshot?.route)
const fireScenario = (scenario: Scenario) => {
setFiredScenario(scenario.id)
scenario.run(room).catch((e) => {
console.warn('[MeetDevtools] simulateScenario failed', e)
})
window.setTimeout(
() =>
setFiredScenario((current) =>
current === scenario.id ? undefined : current
),
1200
)
}
if (!open) {
return (
<button
type="button"
style={styles.toggle}
onClick={() => setOpen(true)}
aria-label="Open WebRTC devtools"
>
<span style={{ color: stateColor(connState) }}></span> rtc
</button>
)
}
const config = readRoomConfig(room)
const published = snapshot?.tracks.filter((t) => t.dir === 'up') ?? []
const subscribed = snapshot?.tracks.filter((t) => t.dir === 'down') ?? []
const turnProtocols = snapshot?.turnProtocols ?? []
const transports: Array<{
mode: TransportMode
label: string
requires?: string
title: string
}> = [
{
mode: 'auto',
label: 'auto (udp)',
title:
'clears the server-cached transport preference and relay-only policy, then full reconnect',
},
{
mode: 'tcp',
label: 'tcp',
title: 'force-tcp — server prefers TCP candidates (ICE/TCP)',
},
{
mode: 'turn-udp',
label: 'turn:udp',
requires: 'udp',
title:
'client-side: iceTransportPolicy relay + full reconnect (no server hook exists)',
},
{
mode: 'turn-tcp',
label: 'turn:tcp',
requires: 'tcp',
title: 'force-tcp — lands on TURN/TCP when it is the TCP path',
},
{
mode: 'turn-tls',
label: 'turn:tls',
requires: 'tls',
title: 'force-tls — server switches you to TURN over TLS',
},
]
return (
<section style={styles.panel} aria-label="WebRTC devtools">
<div style={styles.metricsLine}>
<span>
<span style={{ color: COLOR.down }}></span> {snapshot?.downKbps ?? 0}{' '}
kbps
</span>
<span>
<span style={{ color: COLOR.up }}></span> {snapshot?.upKbps ?? 0}{' '}
kbps
</span>
<span style={{ color: COLOR.muted }}>
rtt{' '}
<span style={{ color: COLOR.text }}>{snapshot?.rttMs ?? ''}</span> ms
</span>
<span style={{ color: COLOR.muted }}>
jitter{' '}
<span style={{ color: COLOR.text }}>{snapshot?.jitterMs ?? ''}</span>{' '}
ms
</span>
<span
style={{
color: stateColor(connState),
fontSize: 10,
marginLeft: 'auto',
}}
>
{connState}
</span>
<button
type="button"
style={styles.close}
onClick={() => setOpen(false)}
aria-label="Close WebRTC devtools"
>
</button>
</div>
<Sparkline history={history} />
<div style={styles.sectionTitle}>room configuration (live)</div>
<div style={styles.configList}>
{config.map((entry) => (
<div key={entry.label} style={styles.configRow}>
<span
style={{
color:
entry.on === undefined
? COLOR.faint
: entry.on
? COLOR.ok
: COLOR.bad,
}}
>
</span>
<span style={styles.configLabel}>{entry.label}</span>
<span style={styles.configValue} title={entry.value}>
{entry.value}
</span>
</div>
))}
</div>
<TrackList title="published ↑" rows={published} />
<TrackList title="subscribed ↓" rows={subscribed} />
<div style={styles.sectionTitle}>bandwidth</div>
<div style={{ ...styles.list, gap: 4 }}>
<CapStepper
label="uplink"
hint="encoder cap (setParameters)"
valueKbps={uplinkCap}
onChange={(kbps) => {
setUplinkCapState(kbps)
void setUplinkCap(room, kbps)
}}
/>
<CapStepper
label="downlink"
hint="SFU limit (subscriber-bandwidth)"
valueKbps={downlinkCap}
onChange={(kbps) => {
setDownlinkCapState(kbps)
void setDownlinkCap(room, kbps).catch((e) =>
console.warn('[MeetDevtools] downlink cap failed', e)
)
}}
/>
</div>
<div style={styles.sectionTitle}>transport</div>
<div style={styles.row}>
{transports.map(({ mode, label, requires, title }) => {
const available = !requires || turnProtocols.includes(requires)
const active = transportMode === mode
const clickable = available && !active
return (
<button
key={mode}
type="button"
aria-pressed={active}
disabled={!clickable}
title={available ? title : `${title} — not configured`}
style={{
...styles.button,
...(clickable || active ? {} : styles.buttonDisabled),
...(active ? styles.buttonActive : {}),
}}
onClick={() => {
if (!clickable) return
void (
mode === 'auto'
? releaseForcedTransport(room)
: forceTransport(
room,
mode as 'tcp' | 'turn-udp' | 'turn-tcp' | 'turn-tls'
)
).catch((e) =>
console.warn('[MeetDevtools] transport change failed', e)
)
}}
>
{requires && (
<span style={{ color: available ? COLOR.ok : COLOR.faint }}>
{' '}
</span>
)}
{label}
</button>
)
})}
<span style={styles.mono}>
route:{' '}
{snapshot?.route
? [
snapshot.route.protocol,
snapshot.route.type,
snapshot.route.relayProtocol &&
`relay:${snapshot.route.relayProtocol}`,
]
.filter(Boolean)
.join('·')
: ''}
</span>
</div>
<div style={styles.sectionTitle}>connection scenarios</div>
<div style={styles.row}>
{SCENARIOS.map((scenario) => (
<button
key={scenario.id}
type="button"
title={`${scenario.side}-side simulation — momentary, watch the state dot`}
style={{
...styles.button,
...(firedScenario === scenario.id ? styles.buttonActive : {}),
}}
onClick={() => fireScenario(scenario)}
>
{scenario.side === 'server' ? '☁ ' : ''}
{scenario.label}
</button>
))}
</div>
</section>
)
}
export default MeetDevtools
@@ -1,14 +0,0 @@
import { lazy, Suspense } from 'react'
const LazyPanel = import.meta.env.DEV
? lazy(() => import('./MeetDevtools'))
: null
export const MeetDevtools = () => {
if (!LazyPanel) return null
return (
<Suspense fallback={null}>
<LazyPanel />
</Suspense>
)
}
@@ -1,88 +0,0 @@
import { Room } from 'livekit-client'
export type ConfigEntry = {
label: string
value: string
/** true = feature actively on, false = off, undefined = informational */
on?: boolean
}
const formatBackupCodec = (backup: unknown): string => {
if (backup === undefined || backup === true) return 'auto'
if (backup === false) return 'off'
if (typeof backup === 'object' && backup !== null && 'codec' in backup) {
return String((backup as { codec: unknown }).codec)
}
return String(backup)
}
export const readRoomConfig = (room: Room): ConfigEntry[] => {
const options = room.options
const publish = options.publishDefaults
const adaptive = options.adaptiveStream
const entries: ConfigEntry[] = [
{
label: 'adaptiveStream',
value:
typeof adaptive === 'object'
? JSON.stringify(adaptive)
: String(!!adaptive),
on: !!adaptive,
},
{
label: 'dynacast',
value: String(!!options.dynacast),
on: !!options.dynacast,
},
{
label: 'e2ee',
value: String(room.isE2EEEnabled),
on: room.isE2EEEnabled,
},
{
label: 'videoCodec',
value: publish?.videoCodec ?? 'default',
},
{
label: 'backupCodec',
value: formatBackupCodec(publish?.backupCodec),
},
{
label: 'simulcast',
value: String(publish?.simulcast ?? true),
on: publish?.simulcast ?? true,
},
{
label: 'audio dtx',
value: String(publish?.dtx ?? true),
on: publish?.dtx ?? true,
},
{
label: 'audio red',
value: String(publish?.red ?? true),
on: publish?.red ?? true,
},
{
label: 'quality (local)',
value: room.localParticipant.connectionQuality,
},
]
const server = room.serverInfo
if (server) {
entries.push({
label: 'server',
value: [
server.version && `v${server.version}`,
server.region,
server.protocol !== undefined && `proto ${server.protocol}`,
server.edition !== undefined && `edition ${server.edition}`,
]
.filter(Boolean)
.join(' · '),
})
}
return entries
}
@@ -1,192 +0,0 @@
import { Room, Track } from 'livekit-client'
// Transitive dependency of livekit-client (pinned by it); used only to
// build the one signal request room.simulateScenario cannot express.
import { SimulateScenario } from '@livekit/protocol'
/** Shared bandwidth ladder for the /+ steppers; null = unlimited. */
export const BITRATE_LADDER_KBPS: Array<number | null> = [
null,
2000,
1000,
600,
300,
150,
]
export const stepBitrate = (
current: number | null,
direction: 'decrease' | 'increase'
): number | null => {
const index = BITRATE_LADDER_KBPS.indexOf(current)
const safeIndex = index === -1 ? 0 : index
const next =
direction === 'decrease'
? Math.min(safeIndex + 1, BITRATE_LADDER_KBPS.length - 1)
: Math.max(safeIndex - 1, 0)
return BITRATE_LADDER_KBPS[next]
}
const savedEncodings = new WeakMap<RTCRtpSender, Array<number | undefined>>()
export const setUplinkCap = async (
room: Room,
kbps: number | null
): Promise<void> => {
const senders: RTCRtpSender[] = []
room.localParticipant.trackPublications.forEach((pub) => {
const track = pub.track
if (track?.kind === Track.Kind.Video && track.sender) {
senders.push(track.sender)
}
})
for (const sender of senders) {
const params = sender.getParameters()
if (!params.encodings || params.encodings.length === 0) continue
if (kbps === null) {
const original = savedEncodings.get(sender)
params.encodings.forEach((encoding, i) => {
encoding.maxBitrate = original?.[i]
})
savedEncodings.delete(sender)
} else {
if (!savedEncodings.has(sender)) {
savedEncodings.set(
sender,
params.encodings.map((encoding) => encoding.maxBitrate)
)
}
const activeCount =
params.encodings.filter((encoding) => encoding.active !== false)
.length || 1
// Split the budget across active simulcast layers / SVC encoding.
const perEncoding = Math.max(
30_000,
Math.floor((kbps * 1000) / activeCount)
)
params.encodings.forEach((encoding) => {
encoding.maxBitrate = perEncoding
})
}
try {
await sender.setParameters(params)
} catch (e) {
console.warn('[MeetDevtools] setParameters failed', e)
}
}
}
export const setDownlinkCap = (
room: Room,
kbps: number | null
): Promise<void> =>
room.simulateScenario('subscriber-bandwidth', kbps === null ? 0 : kbps * 1000)
export type Scenario = {
id: string
label: string
/** where the simulation happens */
side: 'client' | 'server'
run: (room: Room) => Promise<void>
}
export const SCENARIOS: Scenario[] = [
{
id: 'resume',
label: 'reconnect (resume)',
side: 'client',
// Replays a signaling WebSocket loss; media keeps flowing, client resumes.
run: (room) => room.simulateScenario('signal-reconnect'),
},
{
id: 'resume-fail',
label: 'reconnect (resume fails)',
side: 'client',
// Same, but the next resume attempt fails → exercises the retry ladder.
run: (room) => room.simulateScenario('resume-reconnect'),
},
{
id: 'full-reconnect',
label: 'full reconnect',
side: 'client',
// Complete rejoin with brand-new peer connections.
run: (room) => room.simulateScenario('full-reconnect'),
},
{
id: 'migration',
label: 'server migration',
side: 'server',
run: (room) => room.simulateScenario('migration'),
},
{
id: 'node-failure',
label: 'SFU node failure',
side: 'server',
run: (room) => room.simulateScenario('node-failure'),
},
{
id: 'server-leave',
label: 'server disconnect',
side: 'server',
// Server-initiated leave: the closest thing to "you got kicked".
run: (room) => room.simulateScenario('server-leave'),
},
]
export type TransportMode =
| 'auto'
| 'tcp'
| 'turn-udp'
| 'turn-tcp'
| 'turn-tls'
const clearServerTransportPreference = (room: Room): Promise<void> =>
room.engine.client.sendSimulateScenario(
new SimulateScenario({
scenario: { case: 'switchCandidateProtocol', value: 0 },
})
)
const setRelayOnly = (room: Room, relay: boolean) => {
room.engine.rtcConfig = {
...room.engine.rtcConfig,
iceTransportPolicy: relay ? 'relay' : 'all',
}
}
const settle = () => new Promise((resolve) => setTimeout(resolve, 300))
export const forceTransport = async (
room: Room,
mode: 'tcp' | 'turn-udp' | 'turn-tcp' | 'turn-tls'
): Promise<void> => {
if (mode === 'turn-udp') {
await clearServerTransportPreference(room)
setRelayOnly(room, true)
await settle()
return room.simulateScenario('full-reconnect')
}
setRelayOnly(room, false)
return room.simulateScenario(mode === 'turn-tls' ? 'force-tls' : 'force-tcp')
}
export const releaseForcedTransport = async (room: Room): Promise<void> => {
setRelayOnly(room, false)
await clearServerTransportPreference(room)
await settle()
await room.simulateScenario('full-reconnect')
}
export const transportModeFromRoute = (route?: {
protocol?: string
relayProtocol?: string
}): TransportMode => {
// relayProtocol is the client→TURN leg; protocol alone means no relay.
if (route?.relayProtocol === 'tls') return 'turn-tls'
if (route?.relayProtocol === 'tcp') return 'turn-tcp'
if (route?.relayProtocol === 'udp') return 'turn-udp'
if (route?.protocol === 'tcp') return 'tcp'
return 'auto'
}
@@ -1,243 +0,0 @@
import { useEffect, useRef, useState } from 'react'
import { Room } from 'livekit-client'
export type TrackRow = {
key: string
dir: 'up' | 'down'
label: string
codec?: string
kbps: number
fps?: number
res?: string
}
export type StatsSnapshot = {
ts: number
/** wire totals from transport stats (includes headers, RTCP, FEC) */
upKbps: number
downKbps: number
rttMs?: number
/** worst inbound RTP jitter across subscribed tracks */
jitterMs?: number
availableOutKbps?: number
/** selected ICE route of the publisher transport (measured, not assumed) */
route?: { protocol?: string; type?: string; relayProtocol?: string }
/**
* relayProtocol values of gathered relay local candidates i.e. which
* clientTURN transports are actually configured (udp/tcp/tls). Empty
* when no TURN server is configured.
*/
turnProtocols: string[]
tracks: TrackRow[]
}
type StatDict = Record<string, unknown>
type Counters = Record<string, number>
const asNumber = (v: unknown): number | undefined =>
typeof v === 'number' && Number.isFinite(v) ? v : undefined
const asString = (v: unknown): string | undefined =>
typeof v === 'string' ? v : undefined
const shortCodec = (mimeType?: string) =>
mimeType ? mimeType.replace(/^(audio|video)\//, '') : undefined
const buildTrackLabels = (room: Room) => {
const labels = new Map<string, string>()
room.localParticipant.trackPublications.forEach((pub) => {
const id = pub.track?.mediaStreamTrack?.id
if (id) labels.set(id, `local ${pub.source}`)
})
room.remoteParticipants.forEach((participant) => {
// Keep rows scannable: participant names capped at 10 chars.
const rawName = participant.name || participant.identity
const name = rawName.length > 10 ? `${rawName.slice(0, 10)}.` : rawName
participant.trackPublications.forEach((pub) => {
const id = pub.track?.mediaStreamTrack?.id
if (id) labels.set(id, `${name} ${pub.source}`)
})
})
return labels
}
export const useWebRTCStats = (
room: Room,
enabled: boolean,
intervalMs = 1000
) => {
// Single source of truth: the snapshot is just the last history entry.
const [history, setHistory] = useState<StatsSnapshot[]>([])
const prevRef = useRef(new Map<string, Counters>())
useEffect(() => {
if (!enabled) return
let cancelled = false
const prev = prevRef.current
/** Per-stat counter deltas; returns 0 on the first sighting. */
const deltas = (key: string, now: Counters): Counters => {
const before = prev.get(key)
prev.set(key, now)
const out: Counters = {}
for (const [name, value] of Object.entries(now)) {
out[name] = before?.[name] !== undefined ? value - before[name] : 0
}
return out
}
const collect = async () => {
const reports: Array<{ pc: 'pub' | 'sub'; report: RTCStatsReport }> = []
try {
// Not public API — see file header.
const manager = room.engine?.pcManager
const pub = await manager?.publisher?.getStats()
if (pub) reports.push({ pc: 'pub', report: pub })
const sub = await manager?.subscriber?.getStats()
if (sub) reports.push({ pc: 'sub', report: sub })
} catch {
// Engine not ready or SDK internals changed; panel shows nothing.
}
if (cancelled || reports.length === 0) return
const labels = buildTrackLabels(room)
const tracks: TrackRow[] = []
let upKbps = 0
let downKbps = 0
let rttMs: number | undefined
let jitterMs: number | undefined
let availableOutKbps: number | undefined
let route: StatsSnapshot['route']
const turnProtocols = new Set<string>()
for (const { pc, report } of reports) {
const byId = new Map<string, StatDict>()
report.forEach((stat) => byId.set(stat.id as string, stat as StatDict))
report.forEach((raw) => {
const stat = raw as StatDict
const type = asString(stat.type)
const ts = asNumber(stat.timestamp) ?? Date.now()
const key = `${pc}:${asString(stat.id) ?? ''}`
if (
type === 'local-candidate' &&
asString(stat.candidateType) === 'relay'
) {
// Relay candidates are only gathered when a TURN server is
// configured and reachable; relayProtocol says how the client
// reaches it (udp/tcp/tls).
turnProtocols.add(
asString(stat.relayProtocol) ?? asString(stat.protocol) ?? 'udp'
)
}
if (type === 'transport') {
const d = deltas(key, {
sent: asNumber(stat.bytesSent) ?? 0,
received: asNumber(stat.bytesReceived) ?? 0,
ts,
})
if (d.ts > 0) {
upKbps += Math.max(0, (d.sent * 8) / d.ts)
downKbps += Math.max(0, (d.received * 8) / d.ts)
}
}
if (type === 'candidate-pair' && stat.nominated === true) {
const rtt = asNumber(stat.currentRoundTripTime)
if (rtt !== undefined) rttMs = Math.round(rtt * 1000)
const available = asNumber(stat.availableOutgoingBitrate)
if (available !== undefined && pc === 'pub') {
availableOutKbps = Math.round(available / 1000)
}
if (pc === 'pub') {
const local = byId.get(asString(stat.localCandidateId) ?? '')
route = {
protocol: asString(local?.protocol),
type: asString(local?.candidateType),
relayProtocol: asString(local?.relayProtocol),
}
}
}
if (type === 'outbound-rtp' || type === 'inbound-rtp') {
const isUp = type === 'outbound-rtp'
if (!isUp) {
const jitter = asNumber(stat.jitter)
if (jitter !== undefined) {
const ms = Math.round(jitter * 1000)
if (jitterMs === undefined || ms > jitterMs) jitterMs = ms
}
}
const d = deltas(key, {
bytes: asNumber(isUp ? stat.bytesSent : stat.bytesReceived) ?? 0,
ts,
})
const kbps = d.ts > 0 ? Math.max(0, (d.bytes * 8) / d.ts) : 0
// Resolve codec + source track.
const codecStat = byId.get(asString(stat.codecId) ?? '')
let msTrackId = asString(stat.trackIdentifier)
if (!msTrackId && isUp) {
const mediaSource = byId.get(asString(stat.mediaSourceId) ?? '')
msTrackId = asString(mediaSource?.trackIdentifier)
}
const rid = asString(stat.rid)
const baseLabel =
(msTrackId && labels.get(msTrackId)) ??
`${asString(stat.kind) ?? 'media'} ssrc ${asNumber(stat.ssrc) ?? '?'}`
const width = asNumber(stat.frameWidth)
const height = asNumber(stat.frameHeight)
tracks.push({
key,
dir: isUp ? 'up' : 'down',
label: rid ? `${baseLabel} [${rid}]` : baseLabel,
codec: shortCodec(asString(codecStat?.mimeType)),
kbps,
fps: asNumber(stat.framesPerSecond),
res: width && height ? `${width}x${height}` : undefined,
})
}
})
}
// Stable order (direction, then label): sorting by bitrate would
// reshuffle rows on every tick as kbps fluctuates.
tracks.sort((a, b) =>
a.dir === b.dir
? a.label.localeCompare(b.label)
: a.dir === 'up'
? -1
: 1
)
const next: StatsSnapshot = {
ts: Date.now(),
upKbps: Math.round(upKbps),
downKbps: Math.round(downKbps),
rttMs,
jitterMs,
availableOutKbps,
route,
turnProtocols: Array.from(turnProtocols).sort(),
tracks,
}
setHistory((h) => [...h.slice(-59), next])
}
void collect()
const id = window.setInterval(() => void collect(), intervalMs)
return () => {
cancelled = true
window.clearInterval(id)
}
}, [room, enabled, intervalMs])
return { snapshot: history[history.length - 1], history }
}
@@ -43,7 +43,6 @@ import { useSnapshot } from 'valtio'
import { userPreferencesStore } from '@/stores/userPreferences'
import { userStore } from '@/stores/user'
import { WatchMediaDeviceErrors } from './WatchMediaDeviceErrors'
import { MeetDevtools } from '@/features/devtools'
import { VOICE_AUDIO_CONSTRAINTS } from '@/features/rooms/livekit/utils/constants'
export const Conference = ({
@@ -299,7 +298,6 @@ export const Conference = ({
<VideoConference />
{!isMobile && <InviteDialog mode={mode} />}
<PictureInPictureConference />
<MeetDevtools />
</LiveKitRoom>
</Screen>
</QueryAware>
@@ -17,7 +17,7 @@ import {
type ProcessorType,
MEDIAPIPE_PATH_WASM,
} from '.'
import { captureEvent, reportError } from '@/features/analytics/telemetry'
import { captureEvent } from '@/features/analytics/telemetry.ts'
const PROCESSING_WIDTH = 256
const PROCESSING_HEIGHT = 144
@@ -26,40 +26,6 @@ const SEGMENTATION_MASK_CANVAS_ID = 'background-blur-local-segmentation'
const BLUR_CANVAS_ID = 'background-blur-local'
const DEFAULT_BLUR = '10'
const CONCEALING_BLUR = '25'
const FRAME_INTERVAL_MS = 1000 / 30
// After this many consecutive failed frames, stop segmenting and fall back to
// publishing a fully blurred frame: the user keeps a live camera instead of a
// frozen one, without ever exposing the surroundings they chose to conceal.
const MAX_CONSECUTIVE_ERRORS = 5
let webgl2Supported: boolean | undefined
/**
* MediaPipe's ImageSegmenter requires a WebGL2 context on the web even with
* `delegate: 'CPU'` (only inference runs on CPU; the mask post-processing in
* TensorsToSegmentationCalculator is GL-based). Without this check, machines
* with WebGL disabled or blocklisted fail at StartGraph with
* `emscripten_webgl_create_context() returned error 0`.
*
* The result is cached and the probe context is explicitly released so that
* repeated support checks do not count against the browser's limit on live
* WebGL contexts.
*/
const isWebGL2Supported = () => {
if (webgl2Supported === undefined) {
try {
const canvas = document.createElement('canvas')
const gl = canvas.getContext('webgl2')
webgl2Supported = !!gl
gl?.getExtension('WEBGL_lose_context')?.loseContext()
} catch {
webgl2Supported = false
}
}
return webgl2Supported
}
/**
* This implementation of video blurring is made to be run on CPU for browser that are
@@ -76,12 +42,14 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
source?: MediaStreamTrack
sourceSettings?: MediaTrackSettings
videoElement?: HTMLVideoElement
videoElementLoaded?: boolean
// Canvas containing the video processing result, of which we extract as stream.
outputCanvas?: HTMLCanvasElement
outputCanvasCtx?: CanvasRenderingContext2D
imageSegmenter?: ImageSegmenter
imageSegmenterResult?: ImageSegmenterResult
// Canvas used for resizing video source and projecting mask.
segmentationMaskCanvas?: HTMLCanvasElement
@@ -98,13 +66,6 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
type: ProcessorType
virtualBackgroundImage?: HTMLImageElement
private virtualBackgroundImagePath?: string
private destroyed = false
private degraded = false
private consecutiveErrors = 0
private processing?: Promise<void>
private onVideoLoaded?: () => void
constructor(opts: ProcessorConfig) {
this.name = 'blur'
this.options = opts
@@ -112,10 +73,7 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
}
static get isSupported() {
return (
navigator.userAgent.toLowerCase().includes('firefox') &&
isWebGL2Supported()
)
return navigator.userAgent.toLowerCase().includes('firefox')
}
async init(opts: ProcessorOptions<Track.Kind>) {
@@ -123,10 +81,6 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
throw new Error('Element is required for processing')
}
this.destroyed = false
this.degraded = false
this.consecutiveErrors = 0
this.source = opts.track as MediaStreamTrack
this.sourceSettings = this.source!.getSettings()
this.videoElement = opts.element as HTMLVideoElement
@@ -143,56 +97,26 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.processedTrack = tracks[0]
this.segmentationMask = new ImageData(PROCESSING_WIDTH, PROCESSING_HEIGHT)
const t0 = performance.now()
await this.initSegmenter()
const segmenterInitMs = Math.round(performance.now() - t0)
this._initWorker()
captureEvent('legacy-background-processor', {
effect_type: this.options.type,
hw_concurrency: navigator.hardwareConcurrency,
video_width: this.videoElement?.videoWidth,
video_height: this.videoElement?.videoHeight,
segmenter_init_ms: segmenterInitMs,
})
captureEvent('firefox-blurring-init', {})
}
_initVirtualBackgroundImage() {
if (this.options.type !== 'virtual' || !this.options.imagePath) {
if (this.options.type !== 'virtual') {
return
}
if (
const needsUpdate =
this.options.imagePath &&
this.virtualBackgroundImage &&
this.virtualBackgroundImagePath === this.options.imagePath
) {
return
this.virtualBackgroundImage.src !== this.options.imagePath
if (this.options.imagePath || needsUpdate) {
this.virtualBackgroundImage = document.createElement('img')
this.virtualBackgroundImage.crossOrigin = 'anonymous'
this.virtualBackgroundImage.src = this.options.imagePath!
}
const image = document.createElement('img')
image.crossOrigin = 'anonymous'
image.src = this.options.imagePath
// Surface load failures once instead of letting drawImage throw on a
// broken image inside the processing loop.
image.decode().catch((error) => {
reportError('effects_processor_failure', error, {
context: 'Failed to load virtual background image',
image_path:
this.options.type === 'virtual' ? this.options.imagePath : undefined,
})
})
this.virtualBackgroundImage = image
this.virtualBackgroundImagePath = this.options.imagePath
}
_isVirtualBackgroundImageReady() {
return (
!!this.virtualBackgroundImage &&
this.virtualBackgroundImage.complete &&
this.virtualBackgroundImage.naturalWidth > 0
)
}
async update(opts: ProcessorConfig): Promise<void> {
@@ -205,26 +129,26 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
name: 'Blurring',
})
this.timerWorker.onmessage = (data) => this.onTimerMessage(data)
const startLoop = () => {
this.onVideoLoaded = undefined
this._syncOutputCanvasSize()
this._scheduleNextFrame()
}
if (this.videoElement!.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA) {
startLoop()
} else {
this.onVideoLoaded = startLoop
this.videoElement!.addEventListener('loadeddata', this.onVideoLoaded, {
once: true,
// When hiding camera then showing it again, the onloadeddata callback is not fired again.
if (this.videoElementLoaded) {
this.timerWorker!.postMessage({
id: SET_TIMEOUT,
timeMs: 1000 / 30,
})
} else {
this.videoElement!.onloadeddata = () => {
this.videoElementLoaded = true
this.timerWorker!.postMessage({
id: SET_TIMEOUT,
timeMs: 1000 / 30,
})
}
}
}
onTimerMessage(response: { data: { id: number } }) {
if (response.data.id === TIMEOUT_TICK) {
this.processing = this.process()
this.process()
}
}
@@ -270,47 +194,30 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
*/
async segment() {
const startTimeMs = performance.now()
return new Promise<void>((resolve, reject) => {
try {
this.imageSegmenter!.segmentForVideo(
this.sourceImageData!,
startTimeMs,
(result: ImageSegmenterResult) => {
try {
// The mask is only valid for the duration of this callback:
// MediaPipe frees the underlying WASM memory as soon as it
// returns, so the data must be copied out synchronously here.
this._applyMaskToAlphaChannel(result)
resolve()
} catch (error) {
reject(error)
}
}
)
} catch (error) {
reject(error)
}
return new Promise<void>((resolve) => {
this.imageSegmenter!.segmentForVideo(
this.sourceImageData!,
startTimeMs,
(result: ImageSegmenterResult) => {
this.imageSegmenterResult = result
resolve()
}
)
})
}
_applyMaskToAlphaChannel(result: ImageSegmenterResult) {
const categoryMask = result.categoryMask
if (!categoryMask) {
return
}
const mask = categoryMask.getAsUint8Array()
const alpha = this.segmentationMask!.data
const length = Math.min(mask.length, alpha.length / 4)
for (let i = 0; i < length; ++i) {
alpha[i * 4 + 3] = 255 - mask[i]
}
}
/**
* Composite the segmentation mask over the output canvas: mask first, then
* the clear body, leaving the background to be filled by the caller.
* TODO: future improvement with WebGL.
*/
_compositeMaskAndBody() {
async blur() {
if (this.options.type !== 'blur') {
throw new Error('Blurring is only supported for blur background')
}
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
}
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
@@ -333,16 +240,6 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
}
/**
* TODO: future improvement with WebGL.
*/
async blur() {
if (this.options.type !== 'blur') {
throw new Error('Blurring is only supported for blur background')
}
this._compositeMaskAndBody()
// Draw blurry background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
@@ -354,150 +251,87 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
* TODO: future improvement with WebGL.
*/
async drawVirtualBackground() {
this._compositeMaskAndBody()
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
this.outputCanvasCtx!.filter = 'none'
if (this._isVirtualBackgroundImageReady()) {
// Draw virtual background.
this.outputCanvasCtx!.drawImage(
this.virtualBackgroundImage!,
0,
0,
this.outputCanvas!.width,
this.outputCanvas!.height
)
} else {
// Image not decoded (yet, or failed to load): fill the background with
// a heavy blur instead. Never fall back to the raw video here — the
// user selected this effect to conceal their surroundings, so the
// fallback must keep concealing them.
this.outputCanvasCtx!.filter = `blur(${CONCEALING_BLUR}px)`
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
}
}
/**
* Draw the whole frame heavily blurred (person included). Used when
* segmentation is broken: the outgoing video keeps flowing instead of
* freezing on a stale frame, while the surroundings the user chose to
* conceal stay concealed. Requires no segmenter, only one filtered draw.
*/
_drawDegradedFrame() {
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
this.outputCanvasCtx!.filter = `blur(${CONCEALING_BLUR}px)`
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
this.outputCanvasCtx!.filter = 'blur(8px)'
// Put opacity mask.
this.outputCanvasCtx!.drawImage(
this.segmentationMaskCanvas!,
0,
0,
PROCESSING_WIDTH,
PROCESSING_HEIGHT,
0,
0,
this.videoElement!.videoWidth,
this.videoElement!.videoHeight
)
// Draw clear body.
this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
// Draw virtual background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
this.outputCanvasCtx!.drawImage(
this.virtualBackgroundImage!,
0,
0,
this.outputCanvas!.width,
this.outputCanvas!.height
)
}
async process() {
if (this.destroyed) {
return
await this.sizeSource()
await this.segment()
if (this.options.type === 'blur') {
await this.blur()
} else {
await this.drawVirtualBackground()
}
try {
this._syncOutputCanvasSize()
// No decoded frame available (e.g. right after a device switch): skip
// this tick rather than processing a 0x0 source.
if (
!this.videoElement ||
this.videoElement.videoWidth === 0 ||
this.videoElement.videoHeight === 0
) {
this._scheduleNextFrame()
return
}
if (this.degraded) {
this._drawDegradedFrame()
this._scheduleNextFrame()
return
}
await this.sizeSource()
await this.segment()
if (this.destroyed) {
return
}
if (this.options.type === 'blur') {
await this.blur()
} else {
await this.drawVirtualBackground()
}
this.consecutiveErrors = 0
} catch (error) {
if (this.destroyed) {
return
}
this.consecutiveErrors += 1
if (this.consecutiveErrors >= MAX_CONSECUTIVE_ERRORS) {
// Degrade to a fully blurred frame: a live camera beats a frozen
// one, and concealment must survive the failure.
this.degraded = true
reportError('effects_processor_failure', error, {
context:
'Background processing failed repeatedly, falling back to fully blurred video',
consecutive_errors: this.consecutiveErrors,
})
this.imageSegmenter?.close()
this.imageSegmenter = undefined
}
}
this._scheduleNextFrame()
}
_scheduleNextFrame() {
if (this.destroyed) {
return
}
this.timerWorker?.postMessage({
this.timerWorker!.postMessage({
id: SET_TIMEOUT,
timeMs: FRAME_INTERVAL_MS,
timeMs: 1000 / 30,
})
}
/**
* Keep the output canvas in sync with the actual decoded video dimensions.
* `MediaStreamTrack.getSettings()` can be incomplete or stale on Firefox,
* so the video element is the source of truth.
*/
_syncOutputCanvasSize() {
const width = this.videoElement?.videoWidth
const height = this.videoElement?.videoHeight
if (!width || !height || !this.outputCanvas) {
return
}
if (
this.outputCanvas.width !== width ||
this.outputCanvas.height !== height
) {
this.outputCanvas.width = width
this.outputCanvas.height = height
}
}
_createMainCanvas() {
const width =
this.sourceSettings?.width || this.videoElement?.videoWidth || 1280
const height =
this.sourceSettings?.height || this.videoElement?.videoHeight || 720
this.outputCanvas = this._createCanvas(BLUR_CANVAS_ID, width, height)
this.outputCanvas = document.querySelector(
'canvas#background-blur-local'
) as HTMLCanvasElement
if (!this.outputCanvas) {
this.outputCanvas = this._createCanvas(
BLUR_CANVAS_ID,
this.sourceSettings!.width!,
this.sourceSettings!.height!
)
}
this.outputCanvasCtx = this.outputCanvas.getContext('2d')!
}
_createMaskCanvas() {
this.segmentationMaskCanvas = this._createCanvas(
SEGMENTATION_MASK_CANVAS_ID,
PROCESSING_WIDTH,
PROCESSING_HEIGHT
)
// getImageData is called on this canvas 30 times per second: opt out of
// GPU backing to avoid a costly readback on every frame.
this.segmentationMaskCanvasCtx = this.segmentationMaskCanvas.getContext(
'2d',
{ willReadFrequently: true }
)!
this.segmentationMaskCanvas = document.querySelector(
`#${SEGMENTATION_MASK_CANVAS_ID}`
) as HTMLCanvasElement
if (!this.segmentationMaskCanvas) {
this.segmentationMaskCanvas = this._createCanvas(
SEGMENTATION_MASK_CANVAS_ID,
PROCESSING_WIDTH,
PROCESSING_HEIGHT
)
}
this.segmentationMaskCanvasCtx =
this.segmentationMaskCanvas.getContext('2d')!
}
_createCanvas(id: string, width: number, height: number) {
@@ -514,39 +348,11 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
}
async destroy() {
this.destroyed = true
this.timerWorker?.postMessage({
id: CLEAR_TIMEOUT,
})
// Let any in-flight frame finish before releasing the resources it uses,
// so segmentForVideo is never called on a closed segmenter.
try {
await this.processing
} catch {
// Failures are already handled inside process().
}
this.processing = undefined
if (this.onVideoLoaded && this.videoElement) {
this.videoElement.removeEventListener('loadeddata', this.onVideoLoaded)
}
this.onVideoLoaded = undefined
this.timerWorker?.terminate()
this.timerWorker = undefined
this.imageSegmenter?.close()
this.imageSegmenter = undefined
this.processedTrack?.stop()
this.processedTrack = undefined
this.outputCanvas = undefined
this.outputCanvasCtx = undefined
this.segmentationMaskCanvas = undefined
this.segmentationMaskCanvasCtx = undefined
this.sourceImageData = undefined
}
}
@@ -6,7 +6,6 @@ import type { Track, TrackProcessor } from 'livekit-client'
import { BackgroundCustomProcessor } from './BackgroundCustomProcessor'
import { UnifiedBackgroundTrackProcessor } from './UnifiedBackgroundTrackProcessor'
import { FaceLandmarksOptions } from './FaceLandmarksProcessor'
import { captureEvent } from '@/features/analytics/telemetry'
export const SELFIE_SEGMENTER_MODEL_PATH =
'/assets/mediapipe/models/selfie_segmenter_landscape.tflite'
@@ -32,29 +31,15 @@ export interface BackgroundProcessorInterface extends TrackProcessor<Track.Kind>
options: ProcessorConfig
}
let unsupportedReported = false
export class BackgroundProcessorFactory {
private static _isSupported?: boolean
static hasModernApiSupport() {
return ProcessorWrapper.hasModernApiSupport
}
static isSupported() {
if (this._isSupported === undefined) {
this._isSupported =
supportsBackgroundProcessors() || BackgroundCustomProcessor.isSupported
}
if (!this._isSupported && !unsupportedReported) {
unsupportedReported = true
captureEvent('background-processor-unsupported', {
path: 'isSupported',
})
}
return this._isSupported
return (
supportsBackgroundProcessors() || BackgroundCustomProcessor.isSupported
)
}
static getProcessor(
@@ -63,8 +48,6 @@ export class BackgroundProcessorFactory {
const isBlur = config.type === ProcessorType.BLUR
const isVirtual = config.type === ProcessorType.VIRTUAL
return new BackgroundCustomProcessor(config)
if (!isBlur && !isVirtual) return undefined
if (supportsBackgroundProcessors()) {
@@ -75,12 +58,6 @@ export class BackgroundProcessorFactory {
return new BackgroundCustomProcessor(config)
}
if (!unsupportedReported) {
captureEvent('background-processor-unsupported', {
path: 'getProcessor',
})
}
return undefined
}
@@ -2,7 +2,7 @@ import { useTranslation } from 'react-i18next'
import { useTrackToggle, UseTrackToggleProps } from '@livekit/components-react'
import { Button, Popover } from '@/primitives'
import { RiArrowUpSLine, RiImageCircleAiFill } from '@remixicon/react'
import { Track, type VideoCaptureOptions, VideoPresets } from 'livekit-client'
import { Track, type VideoCaptureOptions } from 'livekit-client'
import { ToggleDevice } from './ToggleDevice'
import { css } from '@/styled-system/css'
@@ -57,8 +57,7 @@ export const VideoDeviceControl = ({
}: VideoDeviceControlProps) => {
const { t } = useTranslation('rooms', { keyPrefix: 'selectDevice' })
const { videoDeviceId, processorConfig, videoPublishResolution } =
useSnapshot(userChoicesStore)
const { videoDeviceId, processorConfig } = useSnapshot(userChoicesStore)
const onChange = React.useCallback(
(enabled: boolean, isUserInitiated: boolean) =>
@@ -98,9 +97,6 @@ export const VideoDeviceControl = ({
await toggle(!trackProps.enabled, {
processor: processor,
...(videoPublishResolution && {
resolution: VideoPresets[videoPublishResolution].resolution,
}),
} as VideoCaptureOptions)
}
@@ -25,11 +25,7 @@ import {
} from '@/features/files/api/listFiles.ts'
import { useCreateFile } from '@/features/files/api/createFile.ts'
import { FileTrigger } from 'react-aria-components'
import {
RiDeleteBinLine,
RiImageAddFill,
RiProhibitedLine,
} from '@remixicon/react'
import { RiDeleteBinLine, RiImageAddFill } from '@remixicon/react'
import { useDeleteFile } from '@/features/files/api/deleteFile.ts'
import { useUser } from '@/features/auth/api/useUser'
import { ApiFileItem } from '@/features/files/api/types.ts'
@@ -201,23 +197,10 @@ export const EffectsConfiguration = ({
*
* We arrive in this condition when we enter the room with the camera already off.
*/
try {
const newProcessorTmp =
BackgroundProcessorFactory.getProcessor(config)!
await toggle(true, {
processor: newProcessorTmp,
})
} catch (error) {
reportError('effects_processor_failure', error, {
context: 'Error applying effect while enabling camera:',
})
saveProcessorConfig(undefined)
try {
await toggle(true)
} catch {
// Camera errors are handled by the toggle's own error path.
}
}
const newProcessorTmp = BackgroundProcessorFactory.getProcessor(config)!
await toggle(true, {
processor: newProcessorTmp,
})
setTimeout(() => setProcessorPending(false))
return
}
@@ -259,14 +242,6 @@ export const EffectsConfiguration = ({
reportError('effects_processor_failure', error, {
context: 'Error applying effect:',
})
try {
if (videoTrack.getProcessor()) {
await videoTrack.stopProcessor()
}
} catch {
// Best effort: the processor may already be broken.
}
saveProcessorConfig(undefined)
} finally {
// Without setTimeout the DOM is not refreshing when updating the options.
setTimeout(() => setProcessorPending(false))
@@ -275,24 +250,6 @@ export const EffectsConfiguration = ({
[enabled, selectedId, toggle, updateEffectStatusMessage, videoTrack]
)
const clearEffect = useCallback(async () => {
if (selectedId === 'none') return
setProcessorPending(true)
try {
if (videoTrack?.getProcessor()) {
await videoTrack.stopProcessor()
}
saveProcessorConfig(undefined)
announceEffectStatusMessage(t('blur.status.none'))
} catch (error) {
reportError('effects_processor_failure', error, {
context: 'Error clearing effect:',
})
} finally {
setTimeout(() => setProcessorPending(false))
}
}, [announceEffectStatusMessage, selectedId, t, videoTrack])
const { data: appConfig } = useConfig()
const { isLoggedIn } = useUser()
const canUploadBackground =
@@ -690,17 +647,6 @@ export const EffectsConfiguration = ({
gap: '1.25rem',
})}
>
<ToggleButton
variant="bigSquare"
aria-label={t('clear')}
tooltip={t('clear')}
isDisabled={processorOptions.isDisabled}
onChange={clearEffect}
isSelected={selectedId === 'none'}
data-attr="toggle-effect-none"
>
<RiProhibitedLine />
</ToggleButton>
{processorOptions.blurBased.map(({ Icon, ...option }) => (
<ToggleButton
key={option.id}
@@ -22,12 +22,10 @@ import { VOICE_AUDIO_CONSTRAINTS } from '../utils/constants'
import {
saveAudioInputDeviceId,
saveAudioInputEnabled,
saveProcessorConfig,
saveVideoInputDeviceId,
saveVideoInputEnabled,
userChoicesStore,
} from '@/stores/userChoices'
import { reportError } from '@/features/analytics/telemetry'
import { useSyncTrackDeviceId } from './useSyncTrackDeviceId'
// Module-level: effect dependencies, must be referentially stable.
@@ -223,32 +221,15 @@ export function useJoinTracks(): {
[audioDeviceId]
)
const createVideo = useCallback(async () => {
const processor =
BackgroundProcessorFactory.fromProcessorConfig(processorConfig)
if (!processor) {
return createLocalVideoTrack({ deviceId: videoDeviceId })
}
try {
return await createLocalVideoTrack({
const createVideo = useCallback(
() =>
createLocalVideoTrack({
deviceId: videoDeviceId,
processor,
})
} catch (error) {
// A camera problem (permission, device missing/busy) is not the
// effect's fault: let the normal media error handling deal with it
// without touching the user's saved effect.
const e = getMediaDeviceFailure(error as Error)
if (e !== MediaDeviceFailure.Other && !!e) {
throw error
}
reportError('effects_processor_failure', error, {
context: 'Restoring saved effect failed, retrying without it',
})
saveProcessorConfig(undefined)
return createLocalVideoTrack({ deviceId: videoDeviceId })
}
}, [videoDeviceId, processorConfig])
processor:
BackgroundProcessorFactory.fromProcessorConfig(processorConfig),
}),
[videoDeviceId, processorConfig]
)
const audioTrack = useLocalTrack({
ready: audioReady,
@@ -18,7 +18,6 @@ import {
saveVideoPublishResolution,
saveVideoSubscribeQuality,
userChoicesStore,
VIDEO_RESOLUTIONS,
VideoResolution,
} from '@/stores/userChoices'
import { RowWrapper } from './layout/RowWrapper'
@@ -33,8 +32,7 @@ const EMPTY_PROPS = {}
export const VideoTab = ({ id }: VideoTabProps) => {
const { t } = useTranslation('settings', { keyPrefix: 'video' })
const room = useRoomContext()
const { localParticipant, remoteParticipants } = room
const { localParticipant, remoteParticipants } = useRoomContext()
const {
videoDeviceId,
@@ -71,21 +69,20 @@ export const VideoTab = ({ id }: VideoTabProps) => {
isDisabled: true,
}
const handleVideoResolutionChange = async (key: VideoResolution) => {
saveVideoPublishResolution(key)
const videoTrack = localParticipant.getTrackPublication(
const handleVideoResolutionChange = async (key: 'h720' | 'h360' | 'h180') => {
const videoPublication = localParticipant.getTrackPublication(
Track.Source.Camera
)?.track
if (!videoTrack) {
return
)
const videoTrack = videoPublication?.track
if (videoTrack) {
saveVideoPublishResolution(key)
await videoTrack.restartTrack({
resolution: VideoPresets[key].resolution,
deviceId: { exact: videoDeviceId },
processor:
BackgroundProcessorFactory.fromProcessorConfig(processorConfig),
})
}
await videoTrack.restartTrack({
resolution: VideoPresets[key].resolution,
deviceId: { exact: videoDeviceId },
processor:
BackgroundProcessorFactory.fromProcessorConfig(processorConfig),
})
}
/**
@@ -125,13 +122,20 @@ export const VideoTab = ({ id }: VideoTabProps) => {
}, [videoDeviceId, videoElement])
const resolutionItems = useMemo(() => {
const labels: Record<VideoResolution, string> = {
h1080: `${t('resolution.publish.items.veryHigh')} (1080p)`,
h720: `${t('resolution.publish.items.high')} (720p)`,
h360: `${t('resolution.publish.items.medium')} (360p)`,
h180: `${t('resolution.publish.items.low')} (180p)`,
}
return VIDEO_RESOLUTIONS.map((value) => ({ value, label: labels[value] }))
return [
{
value: 'h720',
label: `${t('resolution.publish.items.high')} (720p)`,
},
{
value: 'h360',
label: `${t('resolution.publish.items.medium')} (360p)`,
},
{
value: 'h180',
label: `${t('resolution.publish.items.low')} (180p)`,
},
]
}, [t])
const videoQualityItems = useMemo(() => {
@@ -56,7 +56,6 @@
"publish": {
"label": "Wähle die maximale Auflösung beim Senden",
"items": {
"veryHigh": "Sehr hohe Auflösung",
"high": "Hohe Auflösung",
"medium": "Mittlere Auflösung",
"low": "Niedrige Auflösung"
@@ -56,7 +56,6 @@
"publish": {
"label": "Select your sending resolution (max.)",
"items": {
"veryHigh": "Very high definition",
"high": "High definition",
"medium": "Standard definition",
"low": "Low definition"
@@ -56,7 +56,6 @@
"publish": {
"label": "Selecciona tu resolución de envío (máx.)",
"items": {
"veryHigh": "Muy alta definición",
"high": "Alta definición",
"medium": "Definición estándar",
"low": "Baja definición"
@@ -56,7 +56,6 @@
"publish": {
"label": "Sélectionner votre résolution d'envoi (max.)",
"items": {
"veryHigh": "Très haute définition",
"high": "Haute définition",
"medium": "Définition standard",
"low": "Basse définition"
@@ -56,7 +56,6 @@
"publish": {
"label": "Selecteer uw verzendresolutie (max.)",
"items": {
"veryHigh": "Zeer hoge definitie",
"high": "Hoge definitie",
"medium": "Standaarddefinitie",
"low": "Lage definitie"
+2 -11
View File
@@ -10,12 +10,7 @@ import {
} from '@livekit/components-core'
import { VideoQuality } from 'livekit-client'
export const VIDEO_RESOLUTIONS = ['h1080', 'h720', 'h360', 'h180'] as const
export type VideoResolution = (typeof VIDEO_RESOLUTIONS)[number]
const isVideoResolution = (value: unknown): value is VideoResolution =>
VIDEO_RESOLUTIONS.includes(value as VideoResolution)
export type VideoResolution = 'h720' | 'h360' | 'h180'
export type LocalUserChoices = Omit<LocalUserChoicesLK, 'username'> & {
processorConfig?: ProcessorConfig
@@ -26,17 +21,13 @@ export type LocalUserChoices = Omit<LocalUserChoicesLK, 'username'> & {
}
function getUserChoicesState(): LocalUserChoices {
const stored: LocalUserChoices = {
return {
noiseReductionEnabled: false,
audioOutputDeviceId: 'default', // Use 'default' to match LiveKit's standard device selection behavior
videoPublishResolution: 'h720',
videoSubscribeQuality: VideoQuality.HIGH,
...loadUserChoices(),
}
if (!isVideoResolution(stored.videoPublishResolution)) {
stored.videoPublishResolution = 'h720'
}
return stored
}
export const userChoicesStore = proxy<LocalUserChoices>(getUserChoicesState())