Compare commits

...

21 Commits

Author SHA1 Message Date
lebaudantoine 98d0b48015 fixup! wip 2026-09-02 23:47:39 +02:00
lebaudantoine 7412c02341 fixup! wip enhance analytics 2026-09-02 23:47:09 +02:00
lebaudantoine f2b3af6633 fixup! wip 2026-09-02 23:12:35 +02:00
lebaudantoine 2aafd93b46 fixup! wip 2026-09-02 22:46:15 +02:00
lebaudantoine 87931f50cd fixup! wip try to enhance error handling 2026-09-02 22:30:16 +02:00
lebaudantoine 35cb114160 wip try to enhance error handling 2026-09-02 20:48:16 +02:00
lebaudantoine c783fefa57 fixup! wip 2026-09-02 20:36:33 +02:00
lebaudantoine 09a0238936 wip propose back a no effect options to help user deactivate their effect 2026-09-02 20:09:52 +02:00
lebaudantoine 03870ad1fa wip enhance analytics 2026-09-02 19:54:44 +02:00
lebaudantoine f37992a089 fixup! wip 2026-09-02 19:08:57 +02:00
lebaudantoine c7c6d20b91 wip silence
info log which are purely informational and log with std err
and failure that mediapipe also raises as real js exceptions
2026-09-02 19:05:12 +02:00
lebaudantoine 2a71e5b836 wip 2026-09-02 18:53:33 +02:00
lebaudantoine 089d016d12 🐛(frontend) cache supportsBackgroundProcessors WebGL2 probe result
`supportsBackgroundProcessors()` creates a live WebGL2 context on
every call and never releases it. The method is called from render
paths (e.g. the Effects button on the join screen), so without
caching, each re-render leaks a context until the browser hits its
live-context limit.

This is one of the ways MediaPipe later fails with:

  "emscripten_webgl_create_context() returned error 0"

Support cannot change within a session, so probe once and cache the
result.
2026-09-02 18:26:12 +02:00
Paul Csiki cf3960db95 (backend) add Traefik reverse proxy support for media-auth
Adds support for serving media behind Traefik, which currently cannot work
at all.

The media-auth subrequest views read the original request URL from a
hardcoded HTTP_X_ORIGINAL_URL header. That header is an nginx-ingress
convention. Traefik's ForwardAuth middleware sends X-Forwarded-Uri instead
and has no mechanism to emit X-Original-URL, so behind Traefik every
recording download and file attachment is rejected with a bare 403 --
indistinguishable from a legitimate permission denial, which makes it
painful to diagnose.

Add MEDIA_AUTH_ORIGINAL_URL_HEADER, defaulting to HTTP_X_ORIGINAL_URL so
existing nginx-ingress deployments are unaffected. Traefik deployments set
it to HTTP_X_FORWARDED_URI. It is used in both places that resolve the
header: RecordingViewSet._auth_get_original_url and the file attachment
_authorize_subrequest. The log message on a missing header now names the
header actually expected, which is what makes the failure diagnosable.

This mirrors the setting the sibling Docs project already exposes
(suitenumerique/docs, MEDIA_AUTH_ORIGINAL_URL_HEADER) for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 18:07:23 +02:00
lebaudantoine d80d31897c 🔒️(frontend) fix HIGH CVEs in libexpat 2.8.2-r0
Address the following HIGH severity CVEs in libexpat 2.8.2-r0,
reported by Trivy:

* CVE-2026-66046
* CVE-2026-76641
2026-09-02 15:05:01 +02:00
kaelvar 63a7751072 (frontend) add 1080p sending resolution option
The sending resolution selector stopped at 720p while `VideoPresets` already
exposes `h1080` (1920x1080), so publishers on a good uplink could not make use
of the capacity they had. Add "Very high definition (1080p)" above the existing
entries, translated in the five supported locales.

The default stays `h720`, so nothing changes unless a user goes and picks the
new entry. Being explicit about what that costs, since 1080p roughly doubles a
publisher's uplink: this is a per-user choice, and an instance operator has no
way today to decline it. Whether that warrants a server-side setting alongside
the existing `ApiConfig` flags is a call for maintainers — happy to add one if
you want it, rather than change the API contract unasked in a frontend PR.

While here, make the option list harder to get wrong. Resolutions now come from
a single `VIDEO_RESOLUTIONS` tuple that `VideoResolution` derives from, the
selector items are built by mapping over it against a
`Record<VideoResolution, string>` of labels — so a resolution cannot be added
to one and forgotten in the other — and a persisted value that is not in the
tuple falls back to `h720` instead of reaching `VideoPresets[...]` as
`undefined`, since `loadUserChoices` spreads localStorage without validating
it.

Known limitation, unchanged by this patch: `restartTrack` passes the resolution
as an `ideal` constraint, so a camera that cannot reach the selected height
degrades silently. That is already true of 720p on a 480p webcam; 1080p is the
first step where the gap is the common case rather than the edge one.
2026-09-02 15:05:01 +02:00
kaelvar 1ac1778521 🐛(frontend) keep the sending resolution picked while the camera is off
`handleVideoResolutionChange` did all of its work inside `if (videoTrack)`,
including `saveVideoPublishResolution`. With the camera off there is no camera
publication, so choosing a resolution did nothing at all: it was neither applied
nor recorded, while the selector went on showing the value the user had just
picked. Turning the camera back on then published at the old resolution, and so
did the next session.

Found on a self-hosted instance: a user set the sending resolution with the
camera off, turned it back on, and the publisher kept sending 720p. Nothing in
the UI suggested the choice had been dropped.

Persist the choice first and unconditionally, then restart the track only when
there is one to restart.

Persisting alone is not enough within a session. `roomOptions` is only read by
`new Room(...)`, so a store update never reaches a room that is already built.

Sync the VideoDeviceControl with the userChoiesStore resolution, as we did for
the device id and the processor configuration.

The early return is the honest shape here: with no live track there is nothing
to await, and the defaults above already cover what happens next.
2026-09-02 00:11:43 +02:00
lebaudantoine fcc58065d2 🩹(changelog) fix changelog entry ordering
Restore the correct order of entries in the CHANGELOG, which got
shuffled somewhere between rebases.
2026-09-01 22:08:37 +02:00
lebaudantoine 21c57bffb4 🧑‍💻(devx) add a WebRTC stats and network throttling devtool
Introduce an in-app devtool that monitors WebRTC statistics in
real time and lets developers simulate various network scenarios,
including constraining the uplink and downlink bandwidth.

Makes it much easier to reproduce and investigate connectivity or
quality issues locally without depending on external tools.

The code was AI generated, and might contain some smell.
It's only enabled in dev, and not included in the production
build. Feel free to enhance it as needed.
2026-09-01 22:05:53 +02:00
lebaudantoine bd81c99495 🔧(devx) configure a TURN server on the local LiveKit dev stack
Wire a TURN server into the local LiveKit server used by the dev
stack, so ICE negotiation has more candidate types available during
local testing.

Makes it easier to reproduce connectivity scenarios that would
otherwise only show up on stricter networks in production.
2026-09-01 22:05:53 +02:00
lebaudantoine 839cfa4b80 📝(docs) document v1.30.0 in UPGRADE.md
Add the missing v1.30.0 entry in `UPGRADE.md`, which was overlooked
when the release was published.
2026-09-01 16:22:57 +02:00
29 changed files with 1791 additions and 169 deletions
+10 -4
View File
@@ -8,12 +8,17 @@ and this project adheres to
## [Unreleased] ## [Unreleased]
### Added
- ✨(frontend) add 1080p sending resolution option #1660
- ✨(backend) add Traefik support via configurable media-auth url header #1649
### Fixed
- 🐛(frontend) keep the sending resolution picked while the camera is off #1667
## [1.30.0] - 2026-09-01 ## [1.30.0] - 2026-09-01
### Changed
- ♿️(frontend) close side panel with Escape key #1507
### Added ### Added
- ✨(agent) support Voxtral realtime as inference engine - ✨(agent) support Voxtral realtime as inference engine
@@ -29,6 +34,7 @@ and this project adheres to
- ⬆️(frontend) upgrade @fontsource/opendyslexic from 5.2.5 to 5.3.0 - ⬆️(frontend) upgrade @fontsource/opendyslexic from 5.2.5 to 5.3.0
- ⬆️(addons) upgrade core-js from 3.49.0 to 3.50.0 - ⬆️(addons) upgrade core-js from 3.49.0 to 3.50.0
- ♻️(backend) factorize s3 client creation in utils - ♻️(backend) factorize s3 client creation in utils
- ♿️(frontend) close side panel with Escape key #1507
### Fixed ### Fixed
+2
View File
@@ -16,6 +16,8 @@ the following command inside your docker container:
## [Unreleased] ## [Unreleased]
## v1.30.0
### Removing S3 storage-event webhooks for recordings ### Removing S3 storage-event webhooks for recordings
Recordings were previously confirmed as saved by an S3 storage-event webhook posting to `/api/v1.0/recordings/storage-hook/`. That endpoint has been removed: recordings are now always finalized from LiveKit's own `egress_ended` webhook, which has been the default path since v1.22.0. Recordings were previously confirmed as saved by an S3 storage-event webhook posting to `/api/v1.0/recordings/storage-hook/`. That endpoint has been removed: recordings are now always finalized from LiveKit's own `egress_ended` webhook, which has been the default path since v1.22.0.
+2
View File
@@ -213,6 +213,8 @@ services:
- "7880:7880" - "7880:7880"
- "7881:7881" - "7881:7881"
- "7882:7882/udp" - "7882:7882/udp"
- "3478:3478/udp"
- "30000-30100:30000-30100/udp"
volumes: volumes:
- ./docker/livekit/config/livekit-server.yaml:/config.yaml - ./docker/livekit/config/livekit-server.yaml:/config.yaml
depends_on: depends_on:
+1
View File
@@ -65,6 +65,7 @@ RUN apk update && apk upgrade \
musl \ musl \
musl-utils \ musl-utils \
zlib>=1.3.2-r0 \ zlib>=1.3.2-r0 \
libexpat>=2.8.4-r0 \
&& apk del curl && apk del curl
USER nginx USER nginx
+13
View File
@@ -8,3 +8,16 @@ webhook:
api_key: devkey api_key: devkey
urls: urls:
- http://app-dev:8000/api/v1.0/rooms/webhooks-livekit/ - http://app-dev:8000/api/v1.0/rooms/webhooks-livekit/
turn:
enabled: true
domain: turn.127.0.0.1.nip.io
udp_port: 3478
tls_port: 0
external_tls: false
relay_range_start: 30000
relay_range_end: 30100
allow_restricted_peer_cidrs:
- 192.168.0.0/16
- 172.16.0.0/12
+17 -7
View File
@@ -1076,9 +1076,10 @@ class RecordingViewSet(
def _auth_get_original_url(self, request): def _auth_get_original_url(self, request):
""" """
Extracts and parses the original URL from the "HTTP_X_ORIGINAL_URL" header. Extracts and parses the original URL from the configured header.
Raises PermissionDenied if the header is missing. Raises PermissionDenied if the header is missing.
The original url is passed by nginx in the "HTTP_X_ORIGINAL_URL" header. The original url is passed by the reverse proxy in the header named by the
MEDIA_AUTH_ORIGINAL_URL_HEADER setting, which defaults to "HTTP_X_ORIGINAL_URL".
See corresponding ingress configuration in Helm chart and read about the See corresponding ingress configuration in Helm chart and read about the
nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress
is configured to do this. is configured to do this.
@@ -1088,9 +1089,13 @@ class RecordingViewSet(
reasons. reasons.
""" """
# Extract the original URL from the request header # Extract the original URL from the request header
original_url = request.META.get("HTTP_X_ORIGINAL_URL") original_url = request.META.get(settings.MEDIA_AUTH_ORIGINAL_URL_HEADER)
if not original_url: if not original_url:
logger.warning("Missing HTTP_X_ORIGINAL_URL header in subrequest") logger.warning(
"Missing %s header in subrequest. Set MEDIA_AUTH_ORIGINAL_URL_HEADER "
"to the header your reverse proxy sends.",
settings.MEDIA_AUTH_ORIGINAL_URL_HEADER,
)
raise drf_exceptions.PermissionDenied() raise drf_exceptions.PermissionDenied()
logger.debug("Original url: '%s'", original_url) logger.debug("Original url: '%s'", original_url)
@@ -1415,7 +1420,8 @@ class FileViewSet(
Authorize access based on the original URL of an Nginx subrequest Authorize access based on the original URL of an Nginx subrequest
and user permissions. Returns a dictionary of URL parameters if authorized. and user permissions. Returns a dictionary of URL parameters if authorized.
The original url is passed by nginx in the "HTTP_X_ORIGINAL_URL" header. The original url is passed by the reverse proxy in the header named by the
MEDIA_AUTH_ORIGINAL_URL_HEADER setting, which defaults to "HTTP_X_ORIGINAL_URL".
See corresponding ingress configuration in Helm chart and read about the See corresponding ingress configuration in Helm chart and read about the
nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress nginx.ingress.kubernetes.io/auth-url annotation to understand how the Nginx ingress
is configured to do this. is configured to do this.
@@ -1434,9 +1440,13 @@ class FileViewSet(
- PermissionDenied if authorization fails. - PermissionDenied if authorization fails.
""" """
# Extract the original URL from the request header # Extract the original URL from the request header
original_url = request.META.get("HTTP_X_ORIGINAL_URL") original_url = request.META.get(settings.MEDIA_AUTH_ORIGINAL_URL_HEADER)
if not original_url: if not original_url:
logger.warning("Missing HTTP_X_ORIGINAL_URL header in subrequest") logger.warning(
"Missing %s header in subrequest. Set MEDIA_AUTH_ORIGINAL_URL_HEADER "
"to the header your reverse proxy sends.",
settings.MEDIA_AUTH_ORIGINAL_URL_HEADER,
)
raise drf_exceptions.PermissionDenied() raise drf_exceptions.PermissionDenied()
parsed_url = urlparse(original_url) parsed_url = urlparse(original_url)
@@ -7,6 +7,7 @@ from urllib.parse import quote, urlparse
from django.conf import settings from django.conf import settings
from django.core.files.storage import default_storage from django.core.files.storage import default_storage
from django.test import override_settings
from django.utils import timezone from django.utils import timezone
import pytest import pytest
@@ -143,3 +144,59 @@ def test_api_files_media_auth_own_file_deleted():
) )
assert response.status_code == 403 assert response.status_code == 403
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_files_media_auth_custom_original_url_header():
"""
Authorization should honour the configured original-url header.
Covers the attachment subrequest path, which resolves the header separately
from the recording one. Reverse proxies other than nginx-ingress use
different headers: Traefik's ForwardAuth sends X-Forwarded-Uri and cannot
emit X-Original-URL at all.
"""
user = factories.UserFactory()
file = factories.FileFactory(
type=models.FileTypeChoices.BACKGROUND_IMAGE,
update_upload_state=models.FileUploadStateChoices.READY,
creator=user,
)
client = APIClient()
client.force_login(user)
default_storage.save(file.file_key, BytesIO(b"my prose"))
original_url = f"http://localhost/media/{file.file_key:s}"
response = client.get(
"/api/v1.0/files/media-auth/", HTTP_X_FORWARDED_URI=original_url
)
assert response.status_code == 200
assert "AWS4-HMAC-SHA256 Credential=" in response["Authorization"]
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_files_media_auth_default_header_ignored_when_reconfigured():
"""
Only the configured header should be honoured, never a hardcoded fallback.
"""
user = factories.UserFactory()
file = factories.FileFactory(
type=models.FileTypeChoices.BACKGROUND_IMAGE,
update_upload_state=models.FileUploadStateChoices.READY,
creator=user,
)
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/{file.file_key:s}"
response = client.get(
"/api/v1.0/files/media-auth/", HTTP_X_ORIGINAL_URL=original_url
)
assert response.status_code == 403
@@ -8,6 +8,7 @@ from uuid import uuid4
from django.conf import settings from django.conf import settings
from django.core.files.storage import default_storage from django.core.files.storage import default_storage
from django.test import override_settings
from django.utils import timezone from django.utils import timezone
import pytest import pytest
@@ -282,3 +283,63 @@ def test_api_recordings_media_auth_success_administrator(mode):
timeout=1, timeout=1,
) )
assert response.content.decode("utf-8") == "my prose" assert response.content.decode("utf-8") == "my prose"
def test_api_recordings_media_auth_missing_header():
"""
Test that a subrequest without the configured original-url header is rejected.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
response = client.get("/api/v1.0/recordings/media-auth/")
assert response.status_code == 403
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_recordings_media_auth_custom_original_url_header():
"""
Test that the header carrying the original URL can be configured.
Reverse proxies other than nginx-ingress use different headers: Traefik's
ForwardAuth sends X-Forwarded-Uri and cannot emit X-Original-URL at all.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/recordings/{uuid4()!s}.mp4"
response = client.get(
"/api/v1.0/recordings/media-auth/", HTTP_X_FORWARDED_URI=original_url
)
# The header was read and parsed: we get as far as looking the recording up,
# rather than being rejected for a missing header.
assert response.status_code == 404
@override_settings(MEDIA_AUTH_ORIGINAL_URL_HEADER="HTTP_X_FORWARDED_URI")
def test_api_recordings_media_auth_default_header_ignored_when_reconfigured():
"""
Test that only the configured header is honoured.
Guards against the header being read from a hardcoded name in parallel with
the setting.
"""
user = UserFactory()
client = APIClient()
client.force_login(user)
original_url = f"http://localhost/media/recordings/{uuid4()!s}.mp4"
response = client.get(
"/api/v1.0/recordings/media-auth/", HTTP_X_ORIGINAL_URL=original_url
)
assert response.status_code == 403
+9
View File
@@ -129,6 +129,15 @@ class Base(Configuration):
MEDIA_BASE_URL = values.Value( MEDIA_BASE_URL = values.Value(
"", environ_name="MEDIA_BASE_URL", environ_prefix=None "", environ_name="MEDIA_BASE_URL", environ_prefix=None
) )
# Header the reverse proxy uses to pass the original request URL to the
# media-auth subrequest views. nginx-ingress sends X-Original-URL, which is
# the default. Other proxies use different headers -- Traefik's ForwardAuth,
# for instance, sends X-Forwarded-Uri and cannot emit X-Original-URL at all.
MEDIA_AUTH_ORIGINAL_URL_HEADER = values.Value(
default="HTTP_X_ORIGINAL_URL",
environ_name="MEDIA_AUTH_ORIGINAL_URL_HEADER",
environ_prefix=None,
)
SITE_ID = 1 SITE_ID = 1
+1
View File
@@ -53,6 +53,7 @@ RUN apk update && apk upgrade \
musl \ musl \
musl-utils \ musl-utils \
zlib>=1.3.2-r0 \ zlib>=1.3.2-r0 \
libexpat>=2.8.4-r0 \
&& apk del curl && apk del curl
USER nginx USER nginx
@@ -8,6 +8,28 @@ const IGNORED_EXCEPTION_PATTERNS = [
// the close reason is already logged by the SDK. // the close reason is already logged by the SDK.
// See: https://github.com/livekit/client-sdk-js/issues/2062 // See: https://github.com/livekit/client-sdk-js/issues/2062
/^Event captured as exception with keys: isTrusted$/, /^Event captured as exception with keys: isTrusted$/,
// MediaPipe's WASM writes its native logs to stderr, which Emscripten
// routes to console.error, which PostHog's console capture then promotes
// to an $exception — even though nothing was thrown. Two flavors:
//
// 1. "INFO: ..." lines are purely informational. In particular
// "INFO: Created TensorFlow Lite XNNPACK delegate for CPU." is a
// SUCCESS message: TFLite prints it when it lazily initializes CPU
// inference on the first segmented frame. It fires on every effects
// init, on every browser and delegate (the GPU delegate still
// instantiates the CPU/XNNPACK delegate for non-delegated ops), so it
// was our single noisiest "error" while carrying zero signal.
/^INFO: /,
//
// 2. absl-formatted log lines, e.g.
// "E0901 19:21:45.443000 1880752 gl_graph_runner_internal.cc:260]
// StartGraph failed: ..."
// (severity letter E/W/I/F + MMDD + timestamp). These are the stderr
// *copies* of failures that MediaPipe also raises as real JS
// exceptions, which we already capture via reportError / thrown
// errors. Dropping them de-duplicates each incident (previously
// counted 2-3x) without losing the actual error report.
/^[EWIF]\d{4} \d{2}:\d{2}:\d{2}\./,
] ]
const shouldIgnoreException = (value: unknown): boolean => const shouldIgnoreException = (value: unknown): boolean =>
@@ -0,0 +1,595 @@
import { CSSProperties, useState } from 'react'
import { useConnectionState, useRoomContext } from '@livekit/components-react'
import { ConnectionState } from 'livekit-client'
import { StatsSnapshot, TrackRow, useWebRTCStats } from './useWebRTCStats'
import { readRoomConfig } from './roomConfig'
import {
forceTransport,
releaseForcedTransport,
Scenario,
SCENARIOS,
setDownlinkCap,
setUplinkCap,
stepBitrate,
TransportMode,
transportModeFromRoute,
} from './simulation'
const SANS =
"-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif"
const MONO = 'ui-monospace, SFMono-Regular, Menlo, Consolas, monospace'
const COLOR = {
text: '#e8e8e8',
muted: '#9a9a9a',
faint: '#6f6f6f',
hairline: '#2c2d31',
border: '#3d3e44',
surface: '#1b1c1e',
chartBg: '#141517',
down: '#f6821f',
up: '#5a9cf8',
ok: '#10b981',
bad: '#f05a4a',
busy: '#e5a13c',
}
const stateColor = (state: ConnectionState): string => {
switch (state) {
case ConnectionState.Connected:
return COLOR.ok
case ConnectionState.Reconnecting:
case ConnectionState.SignalReconnecting:
return COLOR.busy
case ConnectionState.Connecting:
return COLOR.up
case ConnectionState.Disconnected:
return COLOR.bad
}
}
const styles: Record<string, CSSProperties> = {
toggle: {
position: 'fixed',
bottom: 12,
right: 12,
zIndex: 9999,
fontFamily: SANS,
fontSize: 12,
lineHeight: 1,
padding: '8px 12px',
borderRadius: 999,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
background: COLOR.surface,
color: COLOR.text,
cursor: 'pointer',
boxShadow: '0 2px 10px #0006',
},
panel: {
position: 'fixed',
bottom: 12,
right: 12,
zIndex: 9999,
width: 460,
maxWidth: 'calc(100vw - 24px)',
maxHeight: 'calc(100vh - 24px)',
overflowY: 'auto',
fontFamily: SANS,
fontSize: 11,
lineHeight: 1.5,
color: COLOR.text,
background: COLOR.surface,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
borderRadius: 10,
padding: 14,
boxShadow: '0 10px 34px #00000080',
},
sectionTitle: {
margin: '14px 0 5px',
color: COLOR.faint,
textTransform: 'uppercase',
letterSpacing: 1.2,
fontSize: 9,
fontWeight: 600,
display: 'flex',
justifyContent: 'space-between',
},
metricsLine: {
display: 'flex',
alignItems: 'center',
gap: 16,
margin: '0 0 10px',
paddingBottom: 10,
borderBottom: `1px solid ${COLOR.hairline}`,
fontVariantNumeric: 'tabular-nums',
},
legend: {
display: 'flex',
gap: 12,
marginTop: 3,
color: COLOR.faint,
fontSize: 10,
},
list: {
display: 'flex',
flexDirection: 'column',
maxHeight: '400px',
overflowY: 'auto',
},
configList: { display: 'flex', flexWrap: 'wrap' },
configRow: {
display: 'flex',
gap: 6,
flex: '0 0 50%',
minWidth: 0,
boxSizing: 'border-box',
padding: '1px 8px 1px 0',
},
configLabel: {
color: COLOR.muted,
flex: '0 0 45%',
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
configValue: {
flex: 1,
minWidth: 0,
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
trackRow: {
display: 'flex',
gap: 8,
padding: '3px 0',
borderBottom: `1px solid ${COLOR.hairline}`,
alignItems: 'baseline',
},
trackLabel: {
flex: '0 0 34%',
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
trackCodec: {
flex: '0 0 11%',
fontFamily: MONO,
fontSize: 10,
color: COLOR.muted,
},
trackKbps: {
flex: '0 0 9%',
textAlign: 'right',
fontVariantNumeric: 'tabular-nums',
},
trackDetail: {
flex: 1,
minWidth: 0,
whiteSpace: 'nowrap',
overflow: 'hidden',
textOverflow: 'ellipsis',
},
mono: { fontFamily: MONO, fontSize: 10, color: COLOR.muted },
button: {
fontFamily: SANS,
fontSize: 11,
padding: '3px 9px',
borderRadius: 5,
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.border,
background: '#232428',
color: COLOR.text,
cursor: 'pointer',
},
buttonActive: { borderColor: COLOR.down, color: COLOR.down },
buttonDisabled: { color: COLOR.faint, cursor: 'not-allowed' },
row: { display: 'flex', alignItems: 'center', gap: 6, flexWrap: 'wrap' },
stepperValue: {
minWidth: 92,
textAlign: 'center',
borderWidth: 1,
borderStyle: 'solid',
borderColor: COLOR.hairline,
borderRadius: 4,
padding: '2px 6px',
background: COLOR.chartBg,
fontVariantNumeric: 'tabular-nums',
},
close: {
background: 'none',
border: 'none',
color: COLOR.faint,
cursor: 'pointer',
fontSize: 14,
},
}
const Sparkline = ({ history }: { history: StatsSnapshot[] }) => {
const width = 430
const height = 54
const max = Math.max(...history.map((s) => Math.max(s.upKbps, s.downKbps)), 1)
const points = (pick: (s: StatsSnapshot) => number) =>
history
.map(
(s, i) =>
`${((i / Math.max(history.length - 1, 1)) * width).toFixed(1)},${(
height -
(pick(s) / max) * (height - 6) -
3
).toFixed(1)}`
)
.join(' ')
return (
<>
<svg
width={width}
height={height}
role="img"
aria-label="Bandwidth over the last minute"
style={{
display: 'block',
background: COLOR.chartBg,
borderRadius: 6,
border: `1px solid ${COLOR.hairline}`,
}}
>
{history.length >= 2 && (
<>
<polyline
points={points((s) => s.downKbps)}
fill="none"
stroke={COLOR.down}
strokeWidth="1.5"
/>
<polyline
points={points((s) => s.upKbps)}
fill="none"
stroke={COLOR.up}
strokeWidth="1.5"
/>
</>
)}
</svg>
{/* Legend outside the plot: text over the lines was unreadable. */}
<div style={styles.legend}>
<span>
<span style={{ color: COLOR.down }}></span> down
</span>
<span>
<span style={{ color: COLOR.up }}></span> up
</span>
<span>last 60s · max {Math.round(max)} kbps</span>
</div>
</>
)
}
const CapStepper = ({
label,
hint,
valueKbps,
onChange,
}: {
label: string
hint: string
valueKbps: number | null
onChange: (kbps: number | null) => void
}) => (
<div style={styles.row}>
<span style={{ flex: '0 0 70px' }}>{label}</span>
<button
type="button"
style={styles.button}
aria-label={`Decrease ${label} bandwidth`}
onClick={() => onChange(stepBitrate(valueKbps, 'decrease'))}
>
</button>
<span
style={{
...styles.stepperValue,
...(valueKbps !== null ? { color: COLOR.down } : {}),
}}
>
{valueKbps === null ? 'unlimited' : `${valueKbps} kbps`}
</span>
<button
type="button"
style={styles.button}
aria-label={`Increase ${label} bandwidth`}
onClick={() => onChange(stepBitrate(valueKbps, 'increase'))}
>
+
</button>
<span style={styles.mono}>{hint}</span>
</div>
)
const TrackList = ({ title, rows }: { title: string; rows: TrackRow[] }) => (
<>
<div style={styles.sectionTitle}>
<span>
{title} ({rows.length})
</span>
<span>
{Math.round(rows.reduce((sum, t) => sum + t.kbps, 0))} kbps media
</span>
</div>
<div style={styles.list}>
{rows.map((track) => (
<div key={track.key} style={styles.trackRow}>
<span style={styles.trackLabel}>{track.label}</span>
<span style={styles.trackCodec}>{track.codec ?? ''}</span>
<span style={styles.trackKbps}>{Math.round(track.kbps)}</span>
<span style={styles.trackDetail}>
{[
track.res,
track.fps !== undefined && `${Math.round(track.fps)}fps`,
]
.filter(Boolean)
.join(' · ')}
</span>
</div>
))}
{rows.length === 0 && <span style={{ color: COLOR.faint }}>none</span>}
</div>
</>
)
const MeetDevtools = () => {
const room = useRoomContext()
const connState = useConnectionState(room)
const [open, setOpen] = useState(false)
const [firedScenario, setFiredScenario] = useState<string>()
const [uplinkCap, setUplinkCapState] = useState<number | null>(null)
const [downlinkCap, setDownlinkCapState] = useState<number | null>(null)
const { snapshot, history } = useWebRTCStats(room, open)
const transportMode: TransportMode = transportModeFromRoute(snapshot?.route)
const fireScenario = (scenario: Scenario) => {
setFiredScenario(scenario.id)
scenario.run(room).catch((e) => {
console.warn('[MeetDevtools] simulateScenario failed', e)
})
window.setTimeout(
() =>
setFiredScenario((current) =>
current === scenario.id ? undefined : current
),
1200
)
}
if (!open) {
return (
<button
type="button"
style={styles.toggle}
onClick={() => setOpen(true)}
aria-label="Open WebRTC devtools"
>
<span style={{ color: stateColor(connState) }}></span> rtc
</button>
)
}
const config = readRoomConfig(room)
const published = snapshot?.tracks.filter((t) => t.dir === 'up') ?? []
const subscribed = snapshot?.tracks.filter((t) => t.dir === 'down') ?? []
const turnProtocols = snapshot?.turnProtocols ?? []
const transports: Array<{
mode: TransportMode
label: string
requires?: string
title: string
}> = [
{
mode: 'auto',
label: 'auto (udp)',
title:
'clears the server-cached transport preference and relay-only policy, then full reconnect',
},
{
mode: 'tcp',
label: 'tcp',
title: 'force-tcp — server prefers TCP candidates (ICE/TCP)',
},
{
mode: 'turn-udp',
label: 'turn:udp',
requires: 'udp',
title:
'client-side: iceTransportPolicy relay + full reconnect (no server hook exists)',
},
{
mode: 'turn-tcp',
label: 'turn:tcp',
requires: 'tcp',
title: 'force-tcp — lands on TURN/TCP when it is the TCP path',
},
{
mode: 'turn-tls',
label: 'turn:tls',
requires: 'tls',
title: 'force-tls — server switches you to TURN over TLS',
},
]
return (
<section style={styles.panel} aria-label="WebRTC devtools">
<div style={styles.metricsLine}>
<span>
<span style={{ color: COLOR.down }}></span> {snapshot?.downKbps ?? 0}{' '}
kbps
</span>
<span>
<span style={{ color: COLOR.up }}></span> {snapshot?.upKbps ?? 0}{' '}
kbps
</span>
<span style={{ color: COLOR.muted }}>
rtt{' '}
<span style={{ color: COLOR.text }}>{snapshot?.rttMs ?? ''}</span> ms
</span>
<span style={{ color: COLOR.muted }}>
jitter{' '}
<span style={{ color: COLOR.text }}>{snapshot?.jitterMs ?? ''}</span>{' '}
ms
</span>
<span
style={{
color: stateColor(connState),
fontSize: 10,
marginLeft: 'auto',
}}
>
{connState}
</span>
<button
type="button"
style={styles.close}
onClick={() => setOpen(false)}
aria-label="Close WebRTC devtools"
>
</button>
</div>
<Sparkline history={history} />
<div style={styles.sectionTitle}>room configuration (live)</div>
<div style={styles.configList}>
{config.map((entry) => (
<div key={entry.label} style={styles.configRow}>
<span
style={{
color:
entry.on === undefined
? COLOR.faint
: entry.on
? COLOR.ok
: COLOR.bad,
}}
>
</span>
<span style={styles.configLabel}>{entry.label}</span>
<span style={styles.configValue} title={entry.value}>
{entry.value}
</span>
</div>
))}
</div>
<TrackList title="published ↑" rows={published} />
<TrackList title="subscribed ↓" rows={subscribed} />
<div style={styles.sectionTitle}>bandwidth</div>
<div style={{ ...styles.list, gap: 4 }}>
<CapStepper
label="uplink"
hint="encoder cap (setParameters)"
valueKbps={uplinkCap}
onChange={(kbps) => {
setUplinkCapState(kbps)
void setUplinkCap(room, kbps)
}}
/>
<CapStepper
label="downlink"
hint="SFU limit (subscriber-bandwidth)"
valueKbps={downlinkCap}
onChange={(kbps) => {
setDownlinkCapState(kbps)
void setDownlinkCap(room, kbps).catch((e) =>
console.warn('[MeetDevtools] downlink cap failed', e)
)
}}
/>
</div>
<div style={styles.sectionTitle}>transport</div>
<div style={styles.row}>
{transports.map(({ mode, label, requires, title }) => {
const available = !requires || turnProtocols.includes(requires)
const active = transportMode === mode
const clickable = available && !active
return (
<button
key={mode}
type="button"
aria-pressed={active}
disabled={!clickable}
title={available ? title : `${title} — not configured`}
style={{
...styles.button,
...(clickable || active ? {} : styles.buttonDisabled),
...(active ? styles.buttonActive : {}),
}}
onClick={() => {
if (!clickable) return
void (
mode === 'auto'
? releaseForcedTransport(room)
: forceTransport(
room,
mode as 'tcp' | 'turn-udp' | 'turn-tcp' | 'turn-tls'
)
).catch((e) =>
console.warn('[MeetDevtools] transport change failed', e)
)
}}
>
{requires && (
<span style={{ color: available ? COLOR.ok : COLOR.faint }}>
{' '}
</span>
)}
{label}
</button>
)
})}
<span style={styles.mono}>
route:{' '}
{snapshot?.route
? [
snapshot.route.protocol,
snapshot.route.type,
snapshot.route.relayProtocol &&
`relay:${snapshot.route.relayProtocol}`,
]
.filter(Boolean)
.join('·')
: ''}
</span>
</div>
<div style={styles.sectionTitle}>connection scenarios</div>
<div style={styles.row}>
{SCENARIOS.map((scenario) => (
<button
key={scenario.id}
type="button"
title={`${scenario.side}-side simulation — momentary, watch the state dot`}
style={{
...styles.button,
...(firedScenario === scenario.id ? styles.buttonActive : {}),
}}
onClick={() => fireScenario(scenario)}
>
{scenario.side === 'server' ? '☁ ' : ''}
{scenario.label}
</button>
))}
</div>
</section>
)
}
export default MeetDevtools
@@ -0,0 +1,14 @@
import { lazy, Suspense } from 'react'
const LazyPanel = import.meta.env.DEV
? lazy(() => import('./MeetDevtools'))
: null
export const MeetDevtools = () => {
if (!LazyPanel) return null
return (
<Suspense fallback={null}>
<LazyPanel />
</Suspense>
)
}
@@ -0,0 +1,88 @@
import { Room } from 'livekit-client'
export type ConfigEntry = {
label: string
value: string
/** true = feature actively on, false = off, undefined = informational */
on?: boolean
}
const formatBackupCodec = (backup: unknown): string => {
if (backup === undefined || backup === true) return 'auto'
if (backup === false) return 'off'
if (typeof backup === 'object' && backup !== null && 'codec' in backup) {
return String((backup as { codec: unknown }).codec)
}
return String(backup)
}
export const readRoomConfig = (room: Room): ConfigEntry[] => {
const options = room.options
const publish = options.publishDefaults
const adaptive = options.adaptiveStream
const entries: ConfigEntry[] = [
{
label: 'adaptiveStream',
value:
typeof adaptive === 'object'
? JSON.stringify(adaptive)
: String(!!adaptive),
on: !!adaptive,
},
{
label: 'dynacast',
value: String(!!options.dynacast),
on: !!options.dynacast,
},
{
label: 'e2ee',
value: String(room.isE2EEEnabled),
on: room.isE2EEEnabled,
},
{
label: 'videoCodec',
value: publish?.videoCodec ?? 'default',
},
{
label: 'backupCodec',
value: formatBackupCodec(publish?.backupCodec),
},
{
label: 'simulcast',
value: String(publish?.simulcast ?? true),
on: publish?.simulcast ?? true,
},
{
label: 'audio dtx',
value: String(publish?.dtx ?? true),
on: publish?.dtx ?? true,
},
{
label: 'audio red',
value: String(publish?.red ?? true),
on: publish?.red ?? true,
},
{
label: 'quality (local)',
value: room.localParticipant.connectionQuality,
},
]
const server = room.serverInfo
if (server) {
entries.push({
label: 'server',
value: [
server.version && `v${server.version}`,
server.region,
server.protocol !== undefined && `proto ${server.protocol}`,
server.edition !== undefined && `edition ${server.edition}`,
]
.filter(Boolean)
.join(' · '),
})
}
return entries
}
@@ -0,0 +1,192 @@
import { Room, Track } from 'livekit-client'
// Transitive dependency of livekit-client (pinned by it); used only to
// build the one signal request room.simulateScenario cannot express.
import { SimulateScenario } from '@livekit/protocol'
/** Shared bandwidth ladder for the /+ steppers; null = unlimited. */
export const BITRATE_LADDER_KBPS: Array<number | null> = [
null,
2000,
1000,
600,
300,
150,
]
export const stepBitrate = (
current: number | null,
direction: 'decrease' | 'increase'
): number | null => {
const index = BITRATE_LADDER_KBPS.indexOf(current)
const safeIndex = index === -1 ? 0 : index
const next =
direction === 'decrease'
? Math.min(safeIndex + 1, BITRATE_LADDER_KBPS.length - 1)
: Math.max(safeIndex - 1, 0)
return BITRATE_LADDER_KBPS[next]
}
const savedEncodings = new WeakMap<RTCRtpSender, Array<number | undefined>>()
export const setUplinkCap = async (
room: Room,
kbps: number | null
): Promise<void> => {
const senders: RTCRtpSender[] = []
room.localParticipant.trackPublications.forEach((pub) => {
const track = pub.track
if (track?.kind === Track.Kind.Video && track.sender) {
senders.push(track.sender)
}
})
for (const sender of senders) {
const params = sender.getParameters()
if (!params.encodings || params.encodings.length === 0) continue
if (kbps === null) {
const original = savedEncodings.get(sender)
params.encodings.forEach((encoding, i) => {
encoding.maxBitrate = original?.[i]
})
savedEncodings.delete(sender)
} else {
if (!savedEncodings.has(sender)) {
savedEncodings.set(
sender,
params.encodings.map((encoding) => encoding.maxBitrate)
)
}
const activeCount =
params.encodings.filter((encoding) => encoding.active !== false)
.length || 1
// Split the budget across active simulcast layers / SVC encoding.
const perEncoding = Math.max(
30_000,
Math.floor((kbps * 1000) / activeCount)
)
params.encodings.forEach((encoding) => {
encoding.maxBitrate = perEncoding
})
}
try {
await sender.setParameters(params)
} catch (e) {
console.warn('[MeetDevtools] setParameters failed', e)
}
}
}
export const setDownlinkCap = (
room: Room,
kbps: number | null
): Promise<void> =>
room.simulateScenario('subscriber-bandwidth', kbps === null ? 0 : kbps * 1000)
export type Scenario = {
id: string
label: string
/** where the simulation happens */
side: 'client' | 'server'
run: (room: Room) => Promise<void>
}
export const SCENARIOS: Scenario[] = [
{
id: 'resume',
label: 'reconnect (resume)',
side: 'client',
// Replays a signaling WebSocket loss; media keeps flowing, client resumes.
run: (room) => room.simulateScenario('signal-reconnect'),
},
{
id: 'resume-fail',
label: 'reconnect (resume fails)',
side: 'client',
// Same, but the next resume attempt fails → exercises the retry ladder.
run: (room) => room.simulateScenario('resume-reconnect'),
},
{
id: 'full-reconnect',
label: 'full reconnect',
side: 'client',
// Complete rejoin with brand-new peer connections.
run: (room) => room.simulateScenario('full-reconnect'),
},
{
id: 'migration',
label: 'server migration',
side: 'server',
run: (room) => room.simulateScenario('migration'),
},
{
id: 'node-failure',
label: 'SFU node failure',
side: 'server',
run: (room) => room.simulateScenario('node-failure'),
},
{
id: 'server-leave',
label: 'server disconnect',
side: 'server',
// Server-initiated leave: the closest thing to "you got kicked".
run: (room) => room.simulateScenario('server-leave'),
},
]
export type TransportMode =
| 'auto'
| 'tcp'
| 'turn-udp'
| 'turn-tcp'
| 'turn-tls'
const clearServerTransportPreference = (room: Room): Promise<void> =>
room.engine.client.sendSimulateScenario(
new SimulateScenario({
scenario: { case: 'switchCandidateProtocol', value: 0 },
})
)
const setRelayOnly = (room: Room, relay: boolean) => {
room.engine.rtcConfig = {
...room.engine.rtcConfig,
iceTransportPolicy: relay ? 'relay' : 'all',
}
}
const settle = () => new Promise((resolve) => setTimeout(resolve, 300))
export const forceTransport = async (
room: Room,
mode: 'tcp' | 'turn-udp' | 'turn-tcp' | 'turn-tls'
): Promise<void> => {
if (mode === 'turn-udp') {
await clearServerTransportPreference(room)
setRelayOnly(room, true)
await settle()
return room.simulateScenario('full-reconnect')
}
setRelayOnly(room, false)
return room.simulateScenario(mode === 'turn-tls' ? 'force-tls' : 'force-tcp')
}
export const releaseForcedTransport = async (room: Room): Promise<void> => {
setRelayOnly(room, false)
await clearServerTransportPreference(room)
await settle()
await room.simulateScenario('full-reconnect')
}
export const transportModeFromRoute = (route?: {
protocol?: string
relayProtocol?: string
}): TransportMode => {
// relayProtocol is the client→TURN leg; protocol alone means no relay.
if (route?.relayProtocol === 'tls') return 'turn-tls'
if (route?.relayProtocol === 'tcp') return 'turn-tcp'
if (route?.relayProtocol === 'udp') return 'turn-udp'
if (route?.protocol === 'tcp') return 'tcp'
return 'auto'
}
@@ -0,0 +1,243 @@
import { useEffect, useRef, useState } from 'react'
import { Room } from 'livekit-client'
export type TrackRow = {
key: string
dir: 'up' | 'down'
label: string
codec?: string
kbps: number
fps?: number
res?: string
}
export type StatsSnapshot = {
ts: number
/** wire totals from transport stats (includes headers, RTCP, FEC) */
upKbps: number
downKbps: number
rttMs?: number
/** worst inbound RTP jitter across subscribed tracks */
jitterMs?: number
availableOutKbps?: number
/** selected ICE route of the publisher transport (measured, not assumed) */
route?: { protocol?: string; type?: string; relayProtocol?: string }
/**
* relayProtocol values of gathered relay local candidates — i.e. which
* client→TURN transports are actually configured (udp/tcp/tls). Empty
* when no TURN server is configured.
*/
turnProtocols: string[]
tracks: TrackRow[]
}
type StatDict = Record<string, unknown>
type Counters = Record<string, number>
const asNumber = (v: unknown): number | undefined =>
typeof v === 'number' && Number.isFinite(v) ? v : undefined
const asString = (v: unknown): string | undefined =>
typeof v === 'string' ? v : undefined
const shortCodec = (mimeType?: string) =>
mimeType ? mimeType.replace(/^(audio|video)\//, '') : undefined
const buildTrackLabels = (room: Room) => {
const labels = new Map<string, string>()
room.localParticipant.trackPublications.forEach((pub) => {
const id = pub.track?.mediaStreamTrack?.id
if (id) labels.set(id, `local ${pub.source}`)
})
room.remoteParticipants.forEach((participant) => {
// Keep rows scannable: participant names capped at 10 chars.
const rawName = participant.name || participant.identity
const name = rawName.length > 10 ? `${rawName.slice(0, 10)}.` : rawName
participant.trackPublications.forEach((pub) => {
const id = pub.track?.mediaStreamTrack?.id
if (id) labels.set(id, `${name} ${pub.source}`)
})
})
return labels
}
export const useWebRTCStats = (
room: Room,
enabled: boolean,
intervalMs = 1000
) => {
// Single source of truth: the snapshot is just the last history entry.
const [history, setHistory] = useState<StatsSnapshot[]>([])
const prevRef = useRef(new Map<string, Counters>())
useEffect(() => {
if (!enabled) return
let cancelled = false
const prev = prevRef.current
/** Per-stat counter deltas; returns 0 on the first sighting. */
const deltas = (key: string, now: Counters): Counters => {
const before = prev.get(key)
prev.set(key, now)
const out: Counters = {}
for (const [name, value] of Object.entries(now)) {
out[name] = before?.[name] !== undefined ? value - before[name] : 0
}
return out
}
const collect = async () => {
const reports: Array<{ pc: 'pub' | 'sub'; report: RTCStatsReport }> = []
try {
// Not public API — see file header.
const manager = room.engine?.pcManager
const pub = await manager?.publisher?.getStats()
if (pub) reports.push({ pc: 'pub', report: pub })
const sub = await manager?.subscriber?.getStats()
if (sub) reports.push({ pc: 'sub', report: sub })
} catch {
// Engine not ready or SDK internals changed; panel shows nothing.
}
if (cancelled || reports.length === 0) return
const labels = buildTrackLabels(room)
const tracks: TrackRow[] = []
let upKbps = 0
let downKbps = 0
let rttMs: number | undefined
let jitterMs: number | undefined
let availableOutKbps: number | undefined
let route: StatsSnapshot['route']
const turnProtocols = new Set<string>()
for (const { pc, report } of reports) {
const byId = new Map<string, StatDict>()
report.forEach((stat) => byId.set(stat.id as string, stat as StatDict))
report.forEach((raw) => {
const stat = raw as StatDict
const type = asString(stat.type)
const ts = asNumber(stat.timestamp) ?? Date.now()
const key = `${pc}:${asString(stat.id) ?? ''}`
if (
type === 'local-candidate' &&
asString(stat.candidateType) === 'relay'
) {
// Relay candidates are only gathered when a TURN server is
// configured and reachable; relayProtocol says how the client
// reaches it (udp/tcp/tls).
turnProtocols.add(
asString(stat.relayProtocol) ?? asString(stat.protocol) ?? 'udp'
)
}
if (type === 'transport') {
const d = deltas(key, {
sent: asNumber(stat.bytesSent) ?? 0,
received: asNumber(stat.bytesReceived) ?? 0,
ts,
})
if (d.ts > 0) {
upKbps += Math.max(0, (d.sent * 8) / d.ts)
downKbps += Math.max(0, (d.received * 8) / d.ts)
}
}
if (type === 'candidate-pair' && stat.nominated === true) {
const rtt = asNumber(stat.currentRoundTripTime)
if (rtt !== undefined) rttMs = Math.round(rtt * 1000)
const available = asNumber(stat.availableOutgoingBitrate)
if (available !== undefined && pc === 'pub') {
availableOutKbps = Math.round(available / 1000)
}
if (pc === 'pub') {
const local = byId.get(asString(stat.localCandidateId) ?? '')
route = {
protocol: asString(local?.protocol),
type: asString(local?.candidateType),
relayProtocol: asString(local?.relayProtocol),
}
}
}
if (type === 'outbound-rtp' || type === 'inbound-rtp') {
const isUp = type === 'outbound-rtp'
if (!isUp) {
const jitter = asNumber(stat.jitter)
if (jitter !== undefined) {
const ms = Math.round(jitter * 1000)
if (jitterMs === undefined || ms > jitterMs) jitterMs = ms
}
}
const d = deltas(key, {
bytes: asNumber(isUp ? stat.bytesSent : stat.bytesReceived) ?? 0,
ts,
})
const kbps = d.ts > 0 ? Math.max(0, (d.bytes * 8) / d.ts) : 0
// Resolve codec + source track.
const codecStat = byId.get(asString(stat.codecId) ?? '')
let msTrackId = asString(stat.trackIdentifier)
if (!msTrackId && isUp) {
const mediaSource = byId.get(asString(stat.mediaSourceId) ?? '')
msTrackId = asString(mediaSource?.trackIdentifier)
}
const rid = asString(stat.rid)
const baseLabel =
(msTrackId && labels.get(msTrackId)) ??
`${asString(stat.kind) ?? 'media'} ssrc ${asNumber(stat.ssrc) ?? '?'}`
const width = asNumber(stat.frameWidth)
const height = asNumber(stat.frameHeight)
tracks.push({
key,
dir: isUp ? 'up' : 'down',
label: rid ? `${baseLabel} [${rid}]` : baseLabel,
codec: shortCodec(asString(codecStat?.mimeType)),
kbps,
fps: asNumber(stat.framesPerSecond),
res: width && height ? `${width}x${height}` : undefined,
})
}
})
}
// Stable order (direction, then label): sorting by bitrate would
// reshuffle rows on every tick as kbps fluctuates.
tracks.sort((a, b) =>
a.dir === b.dir
? a.label.localeCompare(b.label)
: a.dir === 'up'
? -1
: 1
)
const next: StatsSnapshot = {
ts: Date.now(),
upKbps: Math.round(upKbps),
downKbps: Math.round(downKbps),
rttMs,
jitterMs,
availableOutKbps,
route,
turnProtocols: Array.from(turnProtocols).sort(),
tracks,
}
setHistory((h) => [...h.slice(-59), next])
}
void collect()
const id = window.setInterval(() => void collect(), intervalMs)
return () => {
cancelled = true
window.clearInterval(id)
}
}, [room, enabled, intervalMs])
return { snapshot: history[history.length - 1], history }
}
@@ -43,6 +43,7 @@ import { useSnapshot } from 'valtio'
import { userPreferencesStore } from '@/stores/userPreferences' import { userPreferencesStore } from '@/stores/userPreferences'
import { userStore } from '@/stores/user' import { userStore } from '@/stores/user'
import { WatchMediaDeviceErrors } from './WatchMediaDeviceErrors' import { WatchMediaDeviceErrors } from './WatchMediaDeviceErrors'
import { MeetDevtools } from '@/features/devtools'
import { VOICE_AUDIO_CONSTRAINTS } from '@/features/rooms/livekit/utils/constants' import { VOICE_AUDIO_CONSTRAINTS } from '@/features/rooms/livekit/utils/constants'
export const Conference = ({ export const Conference = ({
@@ -298,6 +299,7 @@ export const Conference = ({
<VideoConference /> <VideoConference />
{!isMobile && <InviteDialog mode={mode} />} {!isMobile && <InviteDialog mode={mode} />}
<PictureInPictureConference /> <PictureInPictureConference />
<MeetDevtools />
</LiveKitRoom> </LiveKitRoom>
</Screen> </Screen>
</QueryAware> </QueryAware>
@@ -17,7 +17,7 @@ import {
type ProcessorType, type ProcessorType,
MEDIAPIPE_PATH_WASM, MEDIAPIPE_PATH_WASM,
} from '.' } from '.'
import { captureEvent } from '@/features/analytics/telemetry.ts' import { captureEvent, reportError } from '@/features/analytics/telemetry'
const PROCESSING_WIDTH = 256 const PROCESSING_WIDTH = 256
const PROCESSING_HEIGHT = 144 const PROCESSING_HEIGHT = 144
@@ -26,6 +26,40 @@ const SEGMENTATION_MASK_CANVAS_ID = 'background-blur-local-segmentation'
const BLUR_CANVAS_ID = 'background-blur-local' const BLUR_CANVAS_ID = 'background-blur-local'
const DEFAULT_BLUR = '10' const DEFAULT_BLUR = '10'
const CONCEALING_BLUR = '25'
const FRAME_INTERVAL_MS = 1000 / 30
// After this many consecutive failed frames, stop segmenting and fall back to
// publishing a fully blurred frame: the user keeps a live camera instead of a
// frozen one, without ever exposing the surroundings they chose to conceal.
const MAX_CONSECUTIVE_ERRORS = 5
let webgl2Supported: boolean | undefined
/**
* MediaPipe's ImageSegmenter requires a WebGL2 context on the web even with
* `delegate: 'CPU'` (only inference runs on CPU; the mask post-processing in
* TensorsToSegmentationCalculator is GL-based). Without this check, machines
* with WebGL disabled or blocklisted fail at StartGraph with
* `emscripten_webgl_create_context() returned error 0`.
*
* The result is cached and the probe context is explicitly released so that
* repeated support checks do not count against the browser's limit on live
* WebGL contexts.
*/
const isWebGL2Supported = () => {
if (webgl2Supported === undefined) {
try {
const canvas = document.createElement('canvas')
const gl = canvas.getContext('webgl2')
webgl2Supported = !!gl
gl?.getExtension('WEBGL_lose_context')?.loseContext()
} catch {
webgl2Supported = false
}
}
return webgl2Supported
}
/** /**
* This implementation of video blurring is made to be run on CPU for browser that are * This implementation of video blurring is made to be run on CPU for browser that are
@@ -42,14 +76,12 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
source?: MediaStreamTrack source?: MediaStreamTrack
sourceSettings?: MediaTrackSettings sourceSettings?: MediaTrackSettings
videoElement?: HTMLVideoElement videoElement?: HTMLVideoElement
videoElementLoaded?: boolean
// Canvas containing the video processing result, of which we extract as stream. // Canvas containing the video processing result, of which we extract as stream.
outputCanvas?: HTMLCanvasElement outputCanvas?: HTMLCanvasElement
outputCanvasCtx?: CanvasRenderingContext2D outputCanvasCtx?: CanvasRenderingContext2D
imageSegmenter?: ImageSegmenter imageSegmenter?: ImageSegmenter
imageSegmenterResult?: ImageSegmenterResult
// Canvas used for resizing video source and projecting mask. // Canvas used for resizing video source and projecting mask.
segmentationMaskCanvas?: HTMLCanvasElement segmentationMaskCanvas?: HTMLCanvasElement
@@ -66,6 +98,13 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
type: ProcessorType type: ProcessorType
virtualBackgroundImage?: HTMLImageElement virtualBackgroundImage?: HTMLImageElement
private virtualBackgroundImagePath?: string
private destroyed = false
private degraded = false
private consecutiveErrors = 0
private processing?: Promise<void>
private onVideoLoaded?: () => void
constructor(opts: ProcessorConfig) { constructor(opts: ProcessorConfig) {
this.name = 'blur' this.name = 'blur'
this.options = opts this.options = opts
@@ -73,7 +112,10 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
} }
static get isSupported() { static get isSupported() {
return navigator.userAgent.toLowerCase().includes('firefox') return (
navigator.userAgent.toLowerCase().includes('firefox') &&
isWebGL2Supported()
)
} }
async init(opts: ProcessorOptions<Track.Kind>) { async init(opts: ProcessorOptions<Track.Kind>) {
@@ -81,6 +123,10 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
throw new Error('Element is required for processing') throw new Error('Element is required for processing')
} }
this.destroyed = false
this.degraded = false
this.consecutiveErrors = 0
this.source = opts.track as MediaStreamTrack this.source = opts.track as MediaStreamTrack
this.sourceSettings = this.source!.getSettings() this.sourceSettings = this.source!.getSettings()
this.videoElement = opts.element as HTMLVideoElement this.videoElement = opts.element as HTMLVideoElement
@@ -97,26 +143,56 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.processedTrack = tracks[0] this.processedTrack = tracks[0]
this.segmentationMask = new ImageData(PROCESSING_WIDTH, PROCESSING_HEIGHT) this.segmentationMask = new ImageData(PROCESSING_WIDTH, PROCESSING_HEIGHT)
const t0 = performance.now()
await this.initSegmenter() await this.initSegmenter()
const segmenterInitMs = Math.round(performance.now() - t0)
this._initWorker() this._initWorker()
captureEvent('firefox-blurring-init', {}) captureEvent('legacy-background-processor', {
effect_type: this.options.type,
hw_concurrency: navigator.hardwareConcurrency,
video_width: this.videoElement?.videoWidth,
video_height: this.videoElement?.videoHeight,
segmenter_init_ms: segmenterInitMs,
})
} }
_initVirtualBackgroundImage() { _initVirtualBackgroundImage() {
if (this.options.type !== 'virtual') { if (this.options.type !== 'virtual' || !this.options.imagePath) {
return return
} }
const needsUpdate = if (
this.options.imagePath &&
this.virtualBackgroundImage && this.virtualBackgroundImage &&
this.virtualBackgroundImage.src !== this.options.imagePath this.virtualBackgroundImagePath === this.options.imagePath
if (this.options.imagePath || needsUpdate) { ) {
this.virtualBackgroundImage = document.createElement('img') return
this.virtualBackgroundImage.crossOrigin = 'anonymous'
this.virtualBackgroundImage.src = this.options.imagePath!
} }
const image = document.createElement('img')
image.crossOrigin = 'anonymous'
image.src = this.options.imagePath
// Surface load failures once instead of letting drawImage throw on a
// broken image inside the processing loop.
image.decode().catch((error) => {
reportError('effects_processor_failure', error, {
context: 'Failed to load virtual background image',
image_path:
this.options.type === 'virtual' ? this.options.imagePath : undefined,
})
})
this.virtualBackgroundImage = image
this.virtualBackgroundImagePath = this.options.imagePath
}
_isVirtualBackgroundImageReady() {
return (
!!this.virtualBackgroundImage &&
this.virtualBackgroundImage.complete &&
this.virtualBackgroundImage.naturalWidth > 0
)
} }
async update(opts: ProcessorConfig): Promise<void> { async update(opts: ProcessorConfig): Promise<void> {
@@ -129,26 +205,26 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
name: 'Blurring', name: 'Blurring',
}) })
this.timerWorker.onmessage = (data) => this.onTimerMessage(data) this.timerWorker.onmessage = (data) => this.onTimerMessage(data)
// When hiding camera then showing it again, the onloadeddata callback is not fired again.
if (this.videoElementLoaded) { const startLoop = () => {
this.timerWorker!.postMessage({ this.onVideoLoaded = undefined
id: SET_TIMEOUT, this._syncOutputCanvasSize()
timeMs: 1000 / 30, this._scheduleNextFrame()
})
} else {
this.videoElement!.onloadeddata = () => {
this.videoElementLoaded = true
this.timerWorker!.postMessage({
id: SET_TIMEOUT,
timeMs: 1000 / 30,
})
} }
if (this.videoElement!.readyState >= HTMLMediaElement.HAVE_CURRENT_DATA) {
startLoop()
} else {
this.onVideoLoaded = startLoop
this.videoElement!.addEventListener('loadeddata', this.onVideoLoaded, {
once: true,
})
} }
} }
onTimerMessage(response: { data: { id: number } }) { onTimerMessage(response: { data: { id: number } }) {
if (response.data.id === TIMEOUT_TICK) { if (response.data.id === TIMEOUT_TICK) {
this.process() this.processing = this.process()
} }
} }
@@ -194,30 +270,47 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
*/ */
async segment() { async segment() {
const startTimeMs = performance.now() const startTimeMs = performance.now()
return new Promise<void>((resolve) => { return new Promise<void>((resolve, reject) => {
try {
this.imageSegmenter!.segmentForVideo( this.imageSegmenter!.segmentForVideo(
this.sourceImageData!, this.sourceImageData!,
startTimeMs, startTimeMs,
(result: ImageSegmenterResult) => { (result: ImageSegmenterResult) => {
this.imageSegmenterResult = result try {
// The mask is only valid for the duration of this callback:
// MediaPipe frees the underlying WASM memory as soon as it
// returns, so the data must be copied out synchronously here.
this._applyMaskToAlphaChannel(result)
resolve() resolve()
} catch (error) {
reject(error)
}
} }
) )
} catch (error) {
reject(error)
}
}) })
} }
/** _applyMaskToAlphaChannel(result: ImageSegmenterResult) {
* TODO: future improvement with WebGL. const categoryMask = result.categoryMask
*/ if (!categoryMask) {
async blur() { return
if (this.options.type !== 'blur') { }
throw new Error('Blurring is only supported for blur background') const mask = categoryMask.getAsUint8Array()
const alpha = this.segmentationMask!.data
const length = Math.min(mask.length, alpha.length / 4)
for (let i = 0; i < length; ++i) {
alpha[i * 4 + 3] = 255 - mask[i]
} }
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
} }
/**
* Composite the segmentation mask over the output canvas: mask first, then
* the clear body, leaving the background to be filled by the caller.
*/
_compositeMaskAndBody() {
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0) this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.outputCanvasCtx!.globalCompositeOperation = 'copy' this.outputCanvasCtx!.globalCompositeOperation = 'copy'
@@ -240,6 +333,16 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.outputCanvasCtx!.globalCompositeOperation = 'source-in' this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none' this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0) this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
}
/**
* TODO: future improvement with WebGL.
*/
async blur() {
if (this.options.type !== 'blur') {
throw new Error('Blurring is only supported for blur background')
}
this._compositeMaskAndBody()
// Draw blurry background. // Draw blurry background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over' this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
@@ -251,36 +354,12 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
* TODO: future improvement with WebGL. * TODO: future improvement with WebGL.
*/ */
async drawVirtualBackground() { async drawVirtualBackground() {
const mask = this.imageSegmenterResult!.categoryMask!.getAsUint8Array() this._compositeMaskAndBody()
for (let i = 0; i < mask.length; ++i) {
this.segmentationMask!.data[i * 4 + 3] = 255 - mask[i]
}
this.segmentationMaskCanvasCtx!.putImageData(this.segmentationMask!, 0, 0)
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
this.outputCanvasCtx!.filter = 'blur(8px)'
// Put opacity mask.
this.outputCanvasCtx!.drawImage(
this.segmentationMaskCanvas!,
0,
0,
PROCESSING_WIDTH,
PROCESSING_HEIGHT,
0,
0,
this.videoElement!.videoWidth,
this.videoElement!.videoHeight
)
// Draw clear body.
this.outputCanvasCtx!.globalCompositeOperation = 'source-in'
this.outputCanvasCtx!.filter = 'none'
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
// Draw virtual background.
this.outputCanvasCtx!.globalCompositeOperation = 'destination-over' this.outputCanvasCtx!.globalCompositeOperation = 'destination-over'
this.outputCanvasCtx!.filter = 'none'
if (this._isVirtualBackgroundImageReady()) {
// Draw virtual background.
this.outputCanvasCtx!.drawImage( this.outputCanvasCtx!.drawImage(
this.virtualBackgroundImage!, this.virtualBackgroundImage!,
0, 0,
@@ -288,50 +367,137 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
this.outputCanvas!.width, this.outputCanvas!.width,
this.outputCanvas!.height this.outputCanvas!.height
) )
} else {
// Image not decoded (yet, or failed to load): fill the background with
// a heavy blur instead. Never fall back to the raw video here — the
// user selected this effect to conceal their surroundings, so the
// fallback must keep concealing them.
this.outputCanvasCtx!.filter = `blur(${CONCEALING_BLUR}px)`
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
}
}
/**
* Draw the whole frame heavily blurred (person included). Used when
* segmentation is broken: the outgoing video keeps flowing instead of
* freezing on a stale frame, while the surroundings the user chose to
* conceal stay concealed. Requires no segmenter, only one filtered draw.
*/
_drawDegradedFrame() {
this.outputCanvasCtx!.globalCompositeOperation = 'copy'
this.outputCanvasCtx!.filter = `blur(${CONCEALING_BLUR}px)`
this.outputCanvasCtx!.drawImage(this.videoElement!, 0, 0)
this.outputCanvasCtx!.filter = 'none'
} }
async process() { async process() {
if (this.destroyed) {
return
}
try {
this._syncOutputCanvasSize()
// No decoded frame available (e.g. right after a device switch): skip
// this tick rather than processing a 0x0 source.
if (
!this.videoElement ||
this.videoElement.videoWidth === 0 ||
this.videoElement.videoHeight === 0
) {
this._scheduleNextFrame()
return
}
if (this.degraded) {
this._drawDegradedFrame()
this._scheduleNextFrame()
return
}
await this.sizeSource() await this.sizeSource()
await this.segment() await this.segment()
if (this.destroyed) {
return
}
if (this.options.type === 'blur') { if (this.options.type === 'blur') {
await this.blur() await this.blur()
} else { } else {
await this.drawVirtualBackground() await this.drawVirtualBackground()
} }
this.timerWorker!.postMessage({ this.consecutiveErrors = 0
} catch (error) {
if (this.destroyed) {
return
}
this.consecutiveErrors += 1
if (this.consecutiveErrors >= MAX_CONSECUTIVE_ERRORS) {
// Degrade to a fully blurred frame: a live camera beats a frozen
// one, and concealment must survive the failure.
this.degraded = true
reportError('effects_processor_failure', error, {
context:
'Background processing failed repeatedly, falling back to fully blurred video',
consecutive_errors: this.consecutiveErrors,
})
this.imageSegmenter?.close()
this.imageSegmenter = undefined
}
}
this._scheduleNextFrame()
}
_scheduleNextFrame() {
if (this.destroyed) {
return
}
this.timerWorker?.postMessage({
id: SET_TIMEOUT, id: SET_TIMEOUT,
timeMs: 1000 / 30, timeMs: FRAME_INTERVAL_MS,
}) })
} }
_createMainCanvas() { /**
this.outputCanvas = document.querySelector( * Keep the output canvas in sync with the actual decoded video dimensions.
'canvas#background-blur-local' * `MediaStreamTrack.getSettings()` can be incomplete or stale on Firefox,
) as HTMLCanvasElement * so the video element is the source of truth.
if (!this.outputCanvas) { */
this.outputCanvas = this._createCanvas( _syncOutputCanvasSize() {
BLUR_CANVAS_ID, const width = this.videoElement?.videoWidth
this.sourceSettings!.width!, const height = this.videoElement?.videoHeight
this.sourceSettings!.height! if (!width || !height || !this.outputCanvas) {
) return
} }
if (
this.outputCanvas.width !== width ||
this.outputCanvas.height !== height
) {
this.outputCanvas.width = width
this.outputCanvas.height = height
}
}
_createMainCanvas() {
const width =
this.sourceSettings?.width || this.videoElement?.videoWidth || 1280
const height =
this.sourceSettings?.height || this.videoElement?.videoHeight || 720
this.outputCanvas = this._createCanvas(BLUR_CANVAS_ID, width, height)
this.outputCanvasCtx = this.outputCanvas.getContext('2d')! this.outputCanvasCtx = this.outputCanvas.getContext('2d')!
} }
_createMaskCanvas() { _createMaskCanvas() {
this.segmentationMaskCanvas = document.querySelector(
`#${SEGMENTATION_MASK_CANVAS_ID}`
) as HTMLCanvasElement
if (!this.segmentationMaskCanvas) {
this.segmentationMaskCanvas = this._createCanvas( this.segmentationMaskCanvas = this._createCanvas(
SEGMENTATION_MASK_CANVAS_ID, SEGMENTATION_MASK_CANVAS_ID,
PROCESSING_WIDTH, PROCESSING_WIDTH,
PROCESSING_HEIGHT PROCESSING_HEIGHT
) )
} // getImageData is called on this canvas 30 times per second: opt out of
this.segmentationMaskCanvasCtx = // GPU backing to avoid a costly readback on every frame.
this.segmentationMaskCanvas.getContext('2d')! this.segmentationMaskCanvasCtx = this.segmentationMaskCanvas.getContext(
'2d',
{ willReadFrequently: true }
)!
} }
_createCanvas(id: string, width: number, height: number) { _createCanvas(id: string, width: number, height: number) {
@@ -348,11 +514,39 @@ export class BackgroundCustomProcessor implements BackgroundProcessorInterface {
} }
async destroy() { async destroy() {
this.destroyed = true
this.timerWorker?.postMessage({ this.timerWorker?.postMessage({
id: CLEAR_TIMEOUT, id: CLEAR_TIMEOUT,
}) })
// Let any in-flight frame finish before releasing the resources it uses,
// so segmentForVideo is never called on a closed segmenter.
try {
await this.processing
} catch {
// Failures are already handled inside process().
}
this.processing = undefined
if (this.onVideoLoaded && this.videoElement) {
this.videoElement.removeEventListener('loadeddata', this.onVideoLoaded)
}
this.onVideoLoaded = undefined
this.timerWorker?.terminate() this.timerWorker?.terminate()
this.timerWorker = undefined
this.imageSegmenter?.close() this.imageSegmenter?.close()
this.imageSegmenter = undefined
this.processedTrack?.stop()
this.processedTrack = undefined
this.outputCanvas = undefined
this.outputCanvasCtx = undefined
this.segmentationMaskCanvas = undefined
this.segmentationMaskCanvasCtx = undefined
this.sourceImageData = undefined
} }
} }
@@ -6,6 +6,7 @@ import type { Track, TrackProcessor } from 'livekit-client'
import { BackgroundCustomProcessor } from './BackgroundCustomProcessor' import { BackgroundCustomProcessor } from './BackgroundCustomProcessor'
import { UnifiedBackgroundTrackProcessor } from './UnifiedBackgroundTrackProcessor' import { UnifiedBackgroundTrackProcessor } from './UnifiedBackgroundTrackProcessor'
import { FaceLandmarksOptions } from './FaceLandmarksProcessor' import { FaceLandmarksOptions } from './FaceLandmarksProcessor'
import { captureEvent } from '@/features/analytics/telemetry'
export const SELFIE_SEGMENTER_MODEL_PATH = export const SELFIE_SEGMENTER_MODEL_PATH =
'/assets/mediapipe/models/selfie_segmenter_landscape.tflite' '/assets/mediapipe/models/selfie_segmenter_landscape.tflite'
@@ -31,15 +32,29 @@ export interface BackgroundProcessorInterface extends TrackProcessor<Track.Kind>
options: ProcessorConfig options: ProcessorConfig
} }
let unsupportedReported = false
export class BackgroundProcessorFactory { export class BackgroundProcessorFactory {
private static _isSupported?: boolean
static hasModernApiSupport() { static hasModernApiSupport() {
return ProcessorWrapper.hasModernApiSupport return ProcessorWrapper.hasModernApiSupport
} }
static isSupported() { static isSupported() {
return ( if (this._isSupported === undefined) {
this._isSupported =
supportsBackgroundProcessors() || BackgroundCustomProcessor.isSupported supportsBackgroundProcessors() || BackgroundCustomProcessor.isSupported
) }
if (!this._isSupported && !unsupportedReported) {
unsupportedReported = true
captureEvent('background-processor-unsupported', {
path: 'isSupported',
})
}
return this._isSupported
} }
static getProcessor( static getProcessor(
@@ -48,6 +63,8 @@ export class BackgroundProcessorFactory {
const isBlur = config.type === ProcessorType.BLUR const isBlur = config.type === ProcessorType.BLUR
const isVirtual = config.type === ProcessorType.VIRTUAL const isVirtual = config.type === ProcessorType.VIRTUAL
return new BackgroundCustomProcessor(config)
if (!isBlur && !isVirtual) return undefined if (!isBlur && !isVirtual) return undefined
if (supportsBackgroundProcessors()) { if (supportsBackgroundProcessors()) {
@@ -58,6 +75,12 @@ export class BackgroundProcessorFactory {
return new BackgroundCustomProcessor(config) return new BackgroundCustomProcessor(config)
} }
if (!unsupportedReported) {
captureEvent('background-processor-unsupported', {
path: 'getProcessor',
})
}
return undefined return undefined
} }
@@ -2,7 +2,7 @@ import { useTranslation } from 'react-i18next'
import { useTrackToggle, UseTrackToggleProps } from '@livekit/components-react' import { useTrackToggle, UseTrackToggleProps } from '@livekit/components-react'
import { Button, Popover } from '@/primitives' import { Button, Popover } from '@/primitives'
import { RiArrowUpSLine, RiImageCircleAiFill } from '@remixicon/react' import { RiArrowUpSLine, RiImageCircleAiFill } from '@remixicon/react'
import { Track, type VideoCaptureOptions } from 'livekit-client' import { Track, type VideoCaptureOptions, VideoPresets } from 'livekit-client'
import { ToggleDevice } from './ToggleDevice' import { ToggleDevice } from './ToggleDevice'
import { css } from '@/styled-system/css' import { css } from '@/styled-system/css'
@@ -57,7 +57,8 @@ export const VideoDeviceControl = ({
}: VideoDeviceControlProps) => { }: VideoDeviceControlProps) => {
const { t } = useTranslation('rooms', { keyPrefix: 'selectDevice' }) const { t } = useTranslation('rooms', { keyPrefix: 'selectDevice' })
const { videoDeviceId, processorConfig } = useSnapshot(userChoicesStore) const { videoDeviceId, processorConfig, videoPublishResolution } =
useSnapshot(userChoicesStore)
const onChange = React.useCallback( const onChange = React.useCallback(
(enabled: boolean, isUserInitiated: boolean) => (enabled: boolean, isUserInitiated: boolean) =>
@@ -97,6 +98,9 @@ export const VideoDeviceControl = ({
await toggle(!trackProps.enabled, { await toggle(!trackProps.enabled, {
processor: processor, processor: processor,
...(videoPublishResolution && {
resolution: VideoPresets[videoPublishResolution].resolution,
}),
} as VideoCaptureOptions) } as VideoCaptureOptions)
} }
@@ -25,7 +25,11 @@ import {
} from '@/features/files/api/listFiles.ts' } from '@/features/files/api/listFiles.ts'
import { useCreateFile } from '@/features/files/api/createFile.ts' import { useCreateFile } from '@/features/files/api/createFile.ts'
import { FileTrigger } from 'react-aria-components' import { FileTrigger } from 'react-aria-components'
import { RiDeleteBinLine, RiImageAddFill } from '@remixicon/react' import {
RiDeleteBinLine,
RiImageAddFill,
RiProhibitedLine,
} from '@remixicon/react'
import { useDeleteFile } from '@/features/files/api/deleteFile.ts' import { useDeleteFile } from '@/features/files/api/deleteFile.ts'
import { useUser } from '@/features/auth/api/useUser' import { useUser } from '@/features/auth/api/useUser'
import { ApiFileItem } from '@/features/files/api/types.ts' import { ApiFileItem } from '@/features/files/api/types.ts'
@@ -197,10 +201,23 @@ export const EffectsConfiguration = ({
* *
* We arrive in this condition when we enter the room with the camera already off. * We arrive in this condition when we enter the room with the camera already off.
*/ */
const newProcessorTmp = BackgroundProcessorFactory.getProcessor(config)! try {
const newProcessorTmp =
BackgroundProcessorFactory.getProcessor(config)!
await toggle(true, { await toggle(true, {
processor: newProcessorTmp, processor: newProcessorTmp,
}) })
} catch (error) {
reportError('effects_processor_failure', error, {
context: 'Error applying effect while enabling camera:',
})
saveProcessorConfig(undefined)
try {
await toggle(true)
} catch {
// Camera errors are handled by the toggle's own error path.
}
}
setTimeout(() => setProcessorPending(false)) setTimeout(() => setProcessorPending(false))
return return
} }
@@ -242,6 +259,14 @@ export const EffectsConfiguration = ({
reportError('effects_processor_failure', error, { reportError('effects_processor_failure', error, {
context: 'Error applying effect:', context: 'Error applying effect:',
}) })
try {
if (videoTrack.getProcessor()) {
await videoTrack.stopProcessor()
}
} catch {
// Best effort: the processor may already be broken.
}
saveProcessorConfig(undefined)
} finally { } finally {
// Without setTimeout the DOM is not refreshing when updating the options. // Without setTimeout the DOM is not refreshing when updating the options.
setTimeout(() => setProcessorPending(false)) setTimeout(() => setProcessorPending(false))
@@ -250,6 +275,24 @@ export const EffectsConfiguration = ({
[enabled, selectedId, toggle, updateEffectStatusMessage, videoTrack] [enabled, selectedId, toggle, updateEffectStatusMessage, videoTrack]
) )
const clearEffect = useCallback(async () => {
if (selectedId === 'none') return
setProcessorPending(true)
try {
if (videoTrack?.getProcessor()) {
await videoTrack.stopProcessor()
}
saveProcessorConfig(undefined)
announceEffectStatusMessage(t('blur.status.none'))
} catch (error) {
reportError('effects_processor_failure', error, {
context: 'Error clearing effect:',
})
} finally {
setTimeout(() => setProcessorPending(false))
}
}, [announceEffectStatusMessage, selectedId, t, videoTrack])
const { data: appConfig } = useConfig() const { data: appConfig } = useConfig()
const { isLoggedIn } = useUser() const { isLoggedIn } = useUser()
const canUploadBackground = const canUploadBackground =
@@ -647,6 +690,17 @@ export const EffectsConfiguration = ({
gap: '1.25rem', gap: '1.25rem',
})} })}
> >
<ToggleButton
variant="bigSquare"
aria-label={t('clear')}
tooltip={t('clear')}
isDisabled={processorOptions.isDisabled}
onChange={clearEffect}
isSelected={selectedId === 'none'}
data-attr="toggle-effect-none"
>
<RiProhibitedLine />
</ToggleButton>
{processorOptions.blurBased.map(({ Icon, ...option }) => ( {processorOptions.blurBased.map(({ Icon, ...option }) => (
<ToggleButton <ToggleButton
key={option.id} key={option.id}
@@ -22,10 +22,12 @@ import { VOICE_AUDIO_CONSTRAINTS } from '../utils/constants'
import { import {
saveAudioInputDeviceId, saveAudioInputDeviceId,
saveAudioInputEnabled, saveAudioInputEnabled,
saveProcessorConfig,
saveVideoInputDeviceId, saveVideoInputDeviceId,
saveVideoInputEnabled, saveVideoInputEnabled,
userChoicesStore, userChoicesStore,
} from '@/stores/userChoices' } from '@/stores/userChoices'
import { reportError } from '@/features/analytics/telemetry'
import { useSyncTrackDeviceId } from './useSyncTrackDeviceId' import { useSyncTrackDeviceId } from './useSyncTrackDeviceId'
// Module-level: effect dependencies, must be referentially stable. // Module-level: effect dependencies, must be referentially stable.
@@ -221,15 +223,32 @@ export function useJoinTracks(): {
[audioDeviceId] [audioDeviceId]
) )
const createVideo = useCallback( const createVideo = useCallback(async () => {
() => const processor =
createLocalVideoTrack({ BackgroundProcessorFactory.fromProcessorConfig(processorConfig)
if (!processor) {
return createLocalVideoTrack({ deviceId: videoDeviceId })
}
try {
return await createLocalVideoTrack({
deviceId: videoDeviceId, deviceId: videoDeviceId,
processor: processor,
BackgroundProcessorFactory.fromProcessorConfig(processorConfig), })
}), } catch (error) {
[videoDeviceId, processorConfig] // A camera problem (permission, device missing/busy) is not the
) // effect's fault: let the normal media error handling deal with it
// without touching the user's saved effect.
const e = getMediaDeviceFailure(error as Error)
if (e !== MediaDeviceFailure.Other && !!e) {
throw error
}
reportError('effects_processor_failure', error, {
context: 'Restoring saved effect failed, retrying without it',
})
saveProcessorConfig(undefined)
return createLocalVideoTrack({ deviceId: videoDeviceId })
}
}, [videoDeviceId, processorConfig])
const audioTrack = useLocalTrack({ const audioTrack = useLocalTrack({
ready: audioReady, ready: audioReady,
@@ -18,6 +18,7 @@ import {
saveVideoPublishResolution, saveVideoPublishResolution,
saveVideoSubscribeQuality, saveVideoSubscribeQuality,
userChoicesStore, userChoicesStore,
VIDEO_RESOLUTIONS,
VideoResolution, VideoResolution,
} from '@/stores/userChoices' } from '@/stores/userChoices'
import { RowWrapper } from './layout/RowWrapper' import { RowWrapper } from './layout/RowWrapper'
@@ -32,7 +33,8 @@ const EMPTY_PROPS = {}
export const VideoTab = ({ id }: VideoTabProps) => { export const VideoTab = ({ id }: VideoTabProps) => {
const { t } = useTranslation('settings', { keyPrefix: 'video' }) const { t } = useTranslation('settings', { keyPrefix: 'video' })
const { localParticipant, remoteParticipants } = useRoomContext() const room = useRoomContext()
const { localParticipant, remoteParticipants } = room
const { const {
videoDeviceId, videoDeviceId,
@@ -69,13 +71,15 @@ export const VideoTab = ({ id }: VideoTabProps) => {
isDisabled: true, isDisabled: true,
} }
const handleVideoResolutionChange = async (key: 'h720' | 'h360' | 'h180') => { const handleVideoResolutionChange = async (key: VideoResolution) => {
const videoPublication = localParticipant.getTrackPublication(
Track.Source.Camera
)
const videoTrack = videoPublication?.track
if (videoTrack) {
saveVideoPublishResolution(key) saveVideoPublishResolution(key)
const videoTrack = localParticipant.getTrackPublication(
Track.Source.Camera
)?.track
if (!videoTrack) {
return
}
await videoTrack.restartTrack({ await videoTrack.restartTrack({
resolution: VideoPresets[key].resolution, resolution: VideoPresets[key].resolution,
deviceId: { exact: videoDeviceId }, deviceId: { exact: videoDeviceId },
@@ -83,7 +87,6 @@ export const VideoTab = ({ id }: VideoTabProps) => {
BackgroundProcessorFactory.fromProcessorConfig(processorConfig), BackgroundProcessorFactory.fromProcessorConfig(processorConfig),
}) })
} }
}
/** /**
* Updates video quality for all existing remote video tracks when user preference changes. * Updates video quality for all existing remote video tracks when user preference changes.
@@ -122,20 +125,13 @@ export const VideoTab = ({ id }: VideoTabProps) => {
}, [videoDeviceId, videoElement]) }, [videoDeviceId, videoElement])
const resolutionItems = useMemo(() => { const resolutionItems = useMemo(() => {
return [ const labels: Record<VideoResolution, string> = {
{ h1080: `${t('resolution.publish.items.veryHigh')} (1080p)`,
value: 'h720', h720: `${t('resolution.publish.items.high')} (720p)`,
label: `${t('resolution.publish.items.high')} (720p)`, h360: `${t('resolution.publish.items.medium')} (360p)`,
}, h180: `${t('resolution.publish.items.low')} (180p)`,
{ }
value: 'h360', return VIDEO_RESOLUTIONS.map((value) => ({ value, label: labels[value] }))
label: `${t('resolution.publish.items.medium')} (360p)`,
},
{
value: 'h180',
label: `${t('resolution.publish.items.low')} (180p)`,
},
]
}, [t]) }, [t])
const videoQualityItems = useMemo(() => { const videoQualityItems = useMemo(() => {
@@ -56,6 +56,7 @@
"publish": { "publish": {
"label": "Wähle die maximale Auflösung beim Senden", "label": "Wähle die maximale Auflösung beim Senden",
"items": { "items": {
"veryHigh": "Sehr hohe Auflösung",
"high": "Hohe Auflösung", "high": "Hohe Auflösung",
"medium": "Mittlere Auflösung", "medium": "Mittlere Auflösung",
"low": "Niedrige Auflösung" "low": "Niedrige Auflösung"
@@ -56,6 +56,7 @@
"publish": { "publish": {
"label": "Select your sending resolution (max.)", "label": "Select your sending resolution (max.)",
"items": { "items": {
"veryHigh": "Very high definition",
"high": "High definition", "high": "High definition",
"medium": "Standard definition", "medium": "Standard definition",
"low": "Low definition" "low": "Low definition"
@@ -56,6 +56,7 @@
"publish": { "publish": {
"label": "Selecciona tu resolución de envío (máx.)", "label": "Selecciona tu resolución de envío (máx.)",
"items": { "items": {
"veryHigh": "Muy alta definición",
"high": "Alta definición", "high": "Alta definición",
"medium": "Definición estándar", "medium": "Definición estándar",
"low": "Baja definición" "low": "Baja definición"
@@ -56,6 +56,7 @@
"publish": { "publish": {
"label": "Sélectionner votre résolution d'envoi (max.)", "label": "Sélectionner votre résolution d'envoi (max.)",
"items": { "items": {
"veryHigh": "Très haute définition",
"high": "Haute définition", "high": "Haute définition",
"medium": "Définition standard", "medium": "Définition standard",
"low": "Basse définition" "low": "Basse définition"
@@ -56,6 +56,7 @@
"publish": { "publish": {
"label": "Selecteer uw verzendresolutie (max.)", "label": "Selecteer uw verzendresolutie (max.)",
"items": { "items": {
"veryHigh": "Zeer hoge definitie",
"high": "Hoge definitie", "high": "Hoge definitie",
"medium": "Standaarddefinitie", "medium": "Standaarddefinitie",
"low": "Lage definitie" "low": "Lage definitie"
+11 -2
View File
@@ -10,7 +10,12 @@ import {
} from '@livekit/components-core' } from '@livekit/components-core'
import { VideoQuality } from 'livekit-client' import { VideoQuality } from 'livekit-client'
export type VideoResolution = 'h720' | 'h360' | 'h180' export const VIDEO_RESOLUTIONS = ['h1080', 'h720', 'h360', 'h180'] as const
export type VideoResolution = (typeof VIDEO_RESOLUTIONS)[number]
const isVideoResolution = (value: unknown): value is VideoResolution =>
VIDEO_RESOLUTIONS.includes(value as VideoResolution)
export type LocalUserChoices = Omit<LocalUserChoicesLK, 'username'> & { export type LocalUserChoices = Omit<LocalUserChoicesLK, 'username'> & {
processorConfig?: ProcessorConfig processorConfig?: ProcessorConfig
@@ -21,13 +26,17 @@ export type LocalUserChoices = Omit<LocalUserChoicesLK, 'username'> & {
} }
function getUserChoicesState(): LocalUserChoices { function getUserChoicesState(): LocalUserChoices {
return { const stored: LocalUserChoices = {
noiseReductionEnabled: false, noiseReductionEnabled: false,
audioOutputDeviceId: 'default', // Use 'default' to match LiveKit's standard device selection behavior audioOutputDeviceId: 'default', // Use 'default' to match LiveKit's standard device selection behavior
videoPublishResolution: 'h720', videoPublishResolution: 'h720',
videoSubscribeQuality: VideoQuality.HIGH, videoSubscribeQuality: VideoQuality.HIGH,
...loadUserChoices(), ...loadUserChoices(),
} }
if (!isVideoResolution(stored.videoPublishResolution)) {
stored.videoPublishResolution = 'h720'
}
return stored
} }
export const userChoicesStore = proxy<LocalUserChoices>(getUserChoicesState()) export const userChoicesStore = proxy<LocalUserChoices>(getUserChoicesState())