mirror of
https://github.com/suitenumerique/meet.git
synced 2026-10-30 16:25:51 +00:00
Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ae4d7b60b9 | |||
| 620602269f | |||
| 71056b6f86 | |||
| 5727d4968d | |||
| 9c75664051 | |||
| ab73621bad | |||
| d8c16c2ffe | |||
| e0dbc487f8 |
@@ -8,10 +8,24 @@ and this project adheres to
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- ✨(frontend) add jittered exponential backoff to LiveKit reconnections
|
||||
- ✨(frontend) tell participants when a recording is interrupted
|
||||
|
||||
### Changed
|
||||
|
||||
- ⬆️(summary) update ffmpeg to 8.1.3-r0
|
||||
- ✨(frontend) replace the Crisp chat with the La Suite support form
|
||||
- ⬆️(frontend) upgrade livekit-client to 2.23.0
|
||||
- ✨(backend) expose LiveKit reconnection settings in the config
|
||||
- ✨(backend) keep recordings cut short by LiveKit egress
|
||||
- ⬆️(dev) upgrade LiveKit server to v1.13.11 and egress to v1.15.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🐛(frontend) show a page when the connection ends unexpectedly
|
||||
- 🐛(frontend) show the transcript mic hint when every mic is muted
|
||||
|
||||
## [1.36.3] - 2026-10-23
|
||||
|
||||
|
||||
+10
@@ -21,6 +21,16 @@ the following command inside your docker container:
|
||||
The Crisp chat is gone, the support button, now open the La Suite support form. They are hidden unless these `FRONTEND_SUPPORT` keys are all set.
|
||||
The former help links for recording/transcription are now redirecting to the adhoc page in the official documentation
|
||||
|
||||
### LiveKit server v1.13.11 and egress v1.15.0
|
||||
|
||||
The development stack now runs LiveKit server v1.13.11 and LiveKit Egress v1.15.0. When upgrading your own deployment:
|
||||
|
||||
- Deploy this release of Meet first, then upgrade egress, then the server. Older Meet releases discard the partial recordings described below.
|
||||
- Transcripts no longer go through Chrome. Since egress v1.13, audio-only recordings always use the egress SDK pipeline.
|
||||
- A transcript now waits for actual audio: it stays "starting", and participants are asked to turn on their microphone, while every microphone is muted.
|
||||
- Recordings cut short are now kept. Since egress v1.14, a transcript that lost the room, or any recording drained because its egress host ran out of CPU, still uploads what it recorded, even though egress may report it as failed. Meet saves and processes it as usual, and tells participants the recording was interrupted. A screen recording that loses the room is still reported as complete, and saved without a warning.
|
||||
- Telephony: since server v1.13.8, a call that matches no dispatch rule, for example while no room is active, is rejected with a SIP 404 instead of being silently dropped.
|
||||
|
||||
## [v1.35.0]
|
||||
|
||||
### Purging deleted rooms
|
||||
|
||||
+2
-2
@@ -218,7 +218,7 @@ services:
|
||||
- kc_postgresql
|
||||
|
||||
livekit:
|
||||
image: livekit/livekit-server:v1.13.6
|
||||
image: livekit/livekit-server:v1.13.11
|
||||
entrypoint: /livekit-server --dev --bind 0.0.0.0 --config ./config.yaml
|
||||
ports:
|
||||
- "7880:7880"
|
||||
@@ -233,7 +233,7 @@ services:
|
||||
- livekit-egress
|
||||
|
||||
livekit-egress:
|
||||
image: livekit/egress:v1.11.0
|
||||
image: livekit/egress:v1.15.0
|
||||
environment:
|
||||
EGRESS_CONFIG_FILE: ./livekit-egress.yaml
|
||||
volumes:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
FROM livekit/livekit-server:v1.13.6
|
||||
FROM livekit/livekit-server:v1.13.11
|
||||
|
||||
# We inject the nip.io certificate manually because the livekit chart doesn't support volume mounting
|
||||
COPY rootCA.pem /etc/ssl/certs/
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
log_level: debug
|
||||
logging:
|
||||
level: debug
|
||||
api_key: devkey
|
||||
api_secret: secret
|
||||
ws_url: "ws://livekit:7880"
|
||||
|
||||
@@ -11,14 +11,23 @@ We do not cover internal signaling behavior. For full reference, see the [LiveKi
|
||||
> [!IMPORTANT]
|
||||
> The WebSocket is the backbone of LiveKit signaling. All signaling messages rely on it, and without it, ICE candidate exchange and peer connection setup cannot occur. If the WebSocket connection is lost, the client automatically attempts to resume the RTC session once connectivity is restored.
|
||||
|
||||
## Reconnection
|
||||
|
||||
When the WebSocket drops, for instance after a LiveKit node crash or during a Redis outage, every client in every room loses it at the same time. To keep them from all reconnecting at once, each client waits a random delay before every attempt. The first delay is at most `LIVEKIT_RECONNECT_BASE_DELAY_MS`. That ceiling then doubles after each failed attempt, capped at `LIVEKIT_RECONNECT_MAX_DELAY_MS`, and each later delay falls between half the ceiling and the ceiling. The first attempt always starts within 5 seconds of the drop.
|
||||
|
||||
A client gives up once its reconnection has lasted longer than `LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS`, and then shows a "connection lost" page. While only signaling is down and its media still flows, it keeps trying instead. A client that loses the connection again less than `LIVEKIT_RECONNECT_BACKOFF_RESET_MS` after reconnecting resumes the backoff one step above where it stopped.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Type | Default | Purpose |
|
||||
| ----------------------------------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `LIVEKIT_FORCE_WSS_PROTOCOL` | Boolean | `True` | Forces the WebSocket URL to use `wss://`. Required for legacy browsers (Firefox <124, Chrome <125, Edge <125) where HTTPS URLs in `WebSocket()` may fail. |
|
||||
| `LIVEKIT_ENABLE_FIREFOX_PROXY_WORKAROUND` | Boolean | `True` | Workaround for Firefox clients behind proxies that fail to establish WebSocket connections. Pre-establishes a dummy connection to “prime” the WebSocket. |
|
||||
| Variable | Type | Default | Purpose |
|
||||
| ----------------------------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `LIVEKIT_FORCE_WSS_PROTOCOL` | Boolean | `False` | Forces the WebSocket URL to use `wss://`. Required for legacy browsers (Firefox <124, Chrome <125, Edge <125) where HTTPS URLs in `WebSocket()` may fail. |
|
||||
| `LIVEKIT_ENABLE_FIREFOX_PROXY_WORKAROUND` | Boolean | `False` | Workaround for Firefox clients behind proxies that fail to establish WebSocket connections. Pre-establishes a dummy connection to “prime” the WebSocket. |
|
||||
| `LIVEKIT_RECONNECT_BACKOFF_ENABLED` | Boolean | `True` | Enables the reconnection backoff described above. When `False`, clients use LiveKit's default reconnection policy. |
|
||||
| `LIVEKIT_RECONNECT_BASE_DELAY_MS` | Integer | `1000` | Longest wait before the first reconnection attempt, in milliseconds. It doubles after each failed attempt. Must be greater than 0. |
|
||||
| `LIVEKIT_RECONNECT_MAX_DELAY_MS` | Integer | `10000` | Longest wait between two reconnection attempts, in milliseconds. Must be greater than 0. |
|
||||
| `LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS` | Integer | `300000` | How long a client keeps trying to reconnect, in milliseconds. Not applied while its media still flows. Must be greater than 0. |
|
||||
| `LIVEKIT_RECONNECT_BACKOFF_RESET_MS` | Integer | `120000` | How long a connection must stay up, in milliseconds, for the next reconnection to start from the shortest delay again. With `0`, every reconnection does. |
|
||||
|
||||
> [!NOTE]
|
||||
> Questions? Open an issue on [GitHub](https://github.com/suitenumerique/meet/issues/new?assignees=&labels=bug&template=Bug_report.md) or join our [Matrix community](https://matrix.to/#/#meet-official:matrix.org).
|
||||
|
||||
@@ -413,6 +413,11 @@ These are the environmental options available on meet backend.
|
||||
| LIVEKIT_API_TIMEOUT_SECONDS | Seconds before a call to the LiveKit API gives up | 10 |
|
||||
| LIVEKIT_FORCE_WSS_PROTOCOL | Enables WSS protocol conversion for legacy browser compatibility (Firefox <124, Chrome <125, Edge <125) where HTTPS URLs fail in WebSocket() constructor. | false |
|
||||
| LIVEKIT_ENABLE_FIREFOX_PROXY_WORKAROUND | Firefox-only connection warmup: pre-calls WebSocket endpoint (expecting 401) to initialize cache, resolving proxy/network connectivity issues. | false |
|
||||
| LIVEKIT_RECONNECT_BACKOFF_ENABLED | Spread client reconnections with a random exponential backoff. When false, clients use LiveKit's default reconnection policy | true |
|
||||
| LIVEKIT_RECONNECT_BASE_DELAY_MS | Longest wait before the first reconnection attempt, in milliseconds. It doubles after each failed attempt. Must be greater than 0 | 1000 |
|
||||
| LIVEKIT_RECONNECT_MAX_DELAY_MS | Longest wait between two reconnection attempts, in milliseconds. Must be greater than 0 | 10000 |
|
||||
| LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS | How long a client keeps trying to reconnect, in milliseconds. Not applied while its media still flows. Must be greater than 0 | 300000 |
|
||||
| LIVEKIT_RECONNECT_BACKOFF_RESET_MS | How long a connection must stay up, in milliseconds, for the next reconnection to start from the shortest delay again. With 0, every reconnection does | 120000 |
|
||||
| RESOURCE_DEFAULT_ACCESS_LEVEL | Default resource access level for rooms | public |
|
||||
| ALLOW_UNREGISTERED_ROOMS | Allow usage of unregistered rooms | true |
|
||||
| ROOM_INACTIVITY_DELETION_DAYS | Days without being started after which a room is purged. Unset to never purge | |
|
||||
|
||||
@@ -86,6 +86,13 @@ def get_frontend_configuration(request):
|
||||
"enable_firefox_proxy_workaround": settings.LIVEKIT_ENABLE_FIREFOX_PROXY_WORKAROUND,
|
||||
"default_sources": settings.LIVEKIT_DEFAULT_SOURCES,
|
||||
"default_video_codec": settings.LIVEKIT_DEFAULT_VIDEO_CODEC,
|
||||
"reconnect": {
|
||||
"backoff_enabled": settings.LIVEKIT_RECONNECT_BACKOFF_ENABLED,
|
||||
"base_delay_ms": settings.LIVEKIT_RECONNECT_BASE_DELAY_MS,
|
||||
"max_delay_ms": settings.LIVEKIT_RECONNECT_MAX_DELAY_MS,
|
||||
"give_up_after_ms": settings.LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS,
|
||||
"backoff_reset_ms": settings.LIVEKIT_RECONNECT_BACKOFF_RESET_MS,
|
||||
},
|
||||
},
|
||||
"allow_unregistered_rooms": settings.ALLOW_UNREGISTERED_ROOMS,
|
||||
"authenticated_users_can_edit_display_name": (
|
||||
|
||||
@@ -27,6 +27,9 @@ class RecordingWorkerEvent(Enum):
|
||||
COMPLETED = "completed"
|
||||
# The recording ended on its configured limit, its media file is available.
|
||||
LIMIT_REACHED = "limit reached"
|
||||
# The recording ended early without being stopped, its partial media file
|
||||
# is available.
|
||||
INTERRUPTED = "interrupted"
|
||||
# The worker stopped before it ever started recording, there is no media file.
|
||||
ABORTED = "aborted"
|
||||
# The worker hit a runtime error once recording had started; its media file
|
||||
@@ -44,6 +47,7 @@ SUCCESSFUL_EVENTS = frozenset(
|
||||
{
|
||||
RecordingWorkerEvent.COMPLETED,
|
||||
RecordingWorkerEvent.LIMIT_REACHED,
|
||||
RecordingWorkerEvent.INTERRUPTED,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ NOTIFICATION_PREFIXES = {
|
||||
}
|
||||
NOTIFICATION_SUFFIXES = {
|
||||
RecordingWorkerEvent.LIMIT_REACHED: "LimitReached",
|
||||
RecordingWorkerEvent.INTERRUPTED: "Interrupted",
|
||||
RecordingWorkerEvent.FAILED: "Failed",
|
||||
RecordingWorkerEvent.ABORTED: "Aborted",
|
||||
}
|
||||
@@ -61,24 +62,29 @@ class RecordingEventsService:
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def log_worker_error(recording, event, error=None, error_code=None):
|
||||
"""Log FAILED at error level and expected ABORTED outcomes at info level."""
|
||||
def log_worker_error(recording, event, error=None, error_code=None, details=None):
|
||||
"""Log FAILED at error level, INTERRUPTED at warning level and expected
|
||||
ABORTED outcomes at info level."""
|
||||
|
||||
if event == RecordingWorkerEvent.FAILED:
|
||||
log = logger.error
|
||||
elif event == RecordingWorkerEvent.INTERRUPTED:
|
||||
log = logger.warning
|
||||
elif event == RecordingWorkerEvent.ABORTED:
|
||||
log = logger.info
|
||||
else:
|
||||
return
|
||||
|
||||
log(
|
||||
"Recording worker reported %s for recording %s (room=%s, mode=%s): %s (error_code=%s)",
|
||||
"Recording worker reported %s for recording %s (room=%s, mode=%s): %s "
|
||||
"(error_code=%s, details=%s)",
|
||||
event.value,
|
||||
recording.id,
|
||||
recording.room.id,
|
||||
recording.mode,
|
||||
error or "no error reported",
|
||||
error_code or "no error_code reported",
|
||||
details or "no details reported",
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
@@ -196,6 +202,13 @@ class RecordingEventsService:
|
||||
):
|
||||
self._apply_outcome(recording, event, self._handle_limit_reached)
|
||||
|
||||
# Interrupted
|
||||
if (
|
||||
event == RecordingWorkerEvent.INTERRUPTED
|
||||
and recording.status == models.RecordingStatusChoices.ACTIVE
|
||||
):
|
||||
self._apply_outcome(recording, event, self._handle_interrupted)
|
||||
|
||||
try:
|
||||
self._handle_successful(recording)
|
||||
except RecordingNotSavableError:
|
||||
@@ -224,6 +237,20 @@ class RecordingEventsService:
|
||||
|
||||
cls._notify_participants(recording, RecordingWorkerEvent.LIMIT_REACHED)
|
||||
|
||||
@classmethod
|
||||
def _handle_interrupted(cls, recording: Recording):
|
||||
"""Stop recording and notify participants that only part of it is saved.
|
||||
|
||||
INTERRUPTED: used when the worker ended the recording early on its own
|
||||
because it lost the room or ran out of resources but kept what it had
|
||||
recorded so far
|
||||
"""
|
||||
|
||||
recording.status = models.RecordingStatusChoices.STOPPED
|
||||
recording.save()
|
||||
|
||||
cls._notify_participants(recording, RecordingWorkerEvent.INTERRUPTED)
|
||||
|
||||
@classmethod
|
||||
def _handle_failed(cls, recording: Recording):
|
||||
"""Set recording status to failed, matching the worker event, and notify participants.
|
||||
|
||||
@@ -95,8 +95,6 @@ class BaseEgressService:
|
||||
# Cases below should be very infrequent as status changes should be
|
||||
# received and processed by `handle_ended`, thus `stop` would not
|
||||
# be called (unless failure and stop are very close in time).
|
||||
# We therefore accept not to notify the user in this code branch.
|
||||
# This could be fixed in a future refactoring.
|
||||
if response.status == livekit_api.EgressStatus.EGRESS_ABORTED:
|
||||
self._log_egress_error(response, "aborted")
|
||||
return "ABORTED"
|
||||
|
||||
@@ -96,6 +96,9 @@ EGRESS_STATUS_TO_RECORDING_EVENT = {
|
||||
api.EgressStatus.EGRESS_FAILED: RecordingWorkerEvent.FAILED,
|
||||
}
|
||||
|
||||
EGRESS_ROOM_CONNECTION_FAILED_ERROR = "recording ended early: connection to room failed"
|
||||
EGRESS_CPU_EXHAUSTED_DETAILS = "End reason: CPU exhausted"
|
||||
|
||||
|
||||
def to_recording_event(egress_status):
|
||||
"""Translate a LiveKit egress status into a recording worker event."""
|
||||
@@ -109,6 +112,34 @@ def to_recording_event(egress_status):
|
||||
return event
|
||||
|
||||
|
||||
def to_ended_recording_event(egress_info):
|
||||
"""Translate an ended LiveKit egress into a recording worker event.
|
||||
|
||||
This handles partial media file upload where an egress was interrupted
|
||||
which can happen when:
|
||||
- an transcription egress that lost its room connection uploads an audio file, then ends
|
||||
FAILED with the EGRESS_ROOM_CONNECTION_FAILED_ERROR
|
||||
- an egress drained because its host ran out of CPU which can transition to COMPLETE state
|
||||
with EGRESS_CPU_EXHAUSTED_DETAILS or FAILED state if the drain outlasted its grace period
|
||||
and the egress got killed, possibly before the upload.
|
||||
"""
|
||||
|
||||
status = egress_info.status
|
||||
is_cut_short = (
|
||||
egress_info.error == EGRESS_ROOM_CONNECTION_FAILED_ERROR
|
||||
or egress_info.details == EGRESS_CPU_EXHAUSTED_DETAILS
|
||||
)
|
||||
has_uploaded_file = any(file.location for file in egress_info.file_results)
|
||||
|
||||
if is_cut_short and (
|
||||
status == api.EgressStatus.EGRESS_COMPLETE
|
||||
or (status == api.EgressStatus.EGRESS_FAILED and has_uploaded_file)
|
||||
):
|
||||
return RecordingWorkerEvent.INTERRUPTED
|
||||
|
||||
return to_recording_event(status)
|
||||
|
||||
|
||||
class LiveKitEventsService:
|
||||
"""Service for processing and handling LiveKit webhook events and notifications."""
|
||||
|
||||
@@ -221,7 +252,7 @@ class LiveKitEventsService:
|
||||
f"Recording with worker ID {data.egress_info.egress_id} does not exist"
|
||||
) from err
|
||||
|
||||
event = to_recording_event(data.egress_info.status)
|
||||
event = to_ended_recording_event(data.egress_info)
|
||||
self._audit_recording_end(recording, event, data.egress_info.error_code)
|
||||
|
||||
# Log if/why the recording failed
|
||||
@@ -230,6 +261,7 @@ class LiveKitEventsService:
|
||||
event,
|
||||
error=data.egress_info.error,
|
||||
error_code=data.egress_info.error_code,
|
||||
details=data.egress_info.details,
|
||||
)
|
||||
|
||||
# Update room
|
||||
|
||||
@@ -50,6 +50,26 @@ def test_handle_limit_reached_success(mock_notify, mode, notification_type, serv
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mode", "notification_type"),
|
||||
(
|
||||
("screen_recording", "screenRecordingInterrupted"),
|
||||
("transcript", "transcriptionInterrupted"),
|
||||
),
|
||||
)
|
||||
@mock.patch("core.utils.notify_participants")
|
||||
def test_handle_interrupted_success(mock_notify, mode, notification_type, service):
|
||||
"""Test _handle_interrupted stops recording and notifies participants."""
|
||||
|
||||
recording = RecordingFactory(status="active", mode=mode)
|
||||
service._handle_interrupted(recording)
|
||||
|
||||
assert recording.status == "stopped"
|
||||
mock_notify.assert_called_once_with(
|
||||
room_name=str(recording.room.id), notification_data={"type": notification_type}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mode", "notification_type"),
|
||||
(
|
||||
@@ -222,6 +242,7 @@ def test_handle_update_syncs_room_metadata(
|
||||
RecordingWorkerEvent.STARTING,
|
||||
RecordingWorkerEvent.COMPLETED,
|
||||
RecordingWorkerEvent.LIMIT_REACHED,
|
||||
RecordingWorkerEvent.INTERRUPTED,
|
||||
RecordingWorkerEvent.ABORTED,
|
||||
RecordingWorkerEvent.FAILED,
|
||||
),
|
||||
@@ -244,6 +265,10 @@ def test_handle_update_ignores_events_without_a_metadata_status(
|
||||
(RecordingWorkerEvent.LIMIT_REACHED, "active", "saved", "LimitReached"),
|
||||
(RecordingWorkerEvent.LIMIT_REACHED, "stopped", "saved", None),
|
||||
(RecordingWorkerEvent.LIMIT_REACHED, "saved", "saved", None),
|
||||
(RecordingWorkerEvent.INTERRUPTED, "active", "saved", "Interrupted"),
|
||||
(RecordingWorkerEvent.INTERRUPTED, "stopped", "saved", None),
|
||||
(RecordingWorkerEvent.INTERRUPTED, "saved", "saved", None),
|
||||
(RecordingWorkerEvent.INTERRUPTED, "failed_to_stop", "failed_to_stop", None),
|
||||
(RecordingWorkerEvent.ABORTED, "active", "aborted", "Aborted"),
|
||||
(RecordingWorkerEvent.ABORTED, "failed_to_stop", "failed_to_stop", None),
|
||||
(RecordingWorkerEvent.FAILED, "active", "failed", "Failed"),
|
||||
@@ -321,6 +346,7 @@ def test_handle_terminal_event_ignores_non_terminal_events(
|
||||
("event", "expected_status"),
|
||||
(
|
||||
(RecordingWorkerEvent.LIMIT_REACHED, "saved"),
|
||||
(RecordingWorkerEvent.INTERRUPTED, "saved"),
|
||||
(RecordingWorkerEvent.ABORTED, "aborted"),
|
||||
(RecordingWorkerEvent.FAILED, "failed"),
|
||||
),
|
||||
@@ -407,6 +433,7 @@ def test_handle_update_survives_a_metadata_failure(
|
||||
("event", "expected_level"),
|
||||
(
|
||||
(RecordingWorkerEvent.ABORTED, logging.INFO),
|
||||
(RecordingWorkerEvent.INTERRUPTED, logging.WARNING),
|
||||
(RecordingWorkerEvent.FAILED, logging.ERROR),
|
||||
),
|
||||
)
|
||||
@@ -419,7 +446,11 @@ def test_log_worker_error_reports_an_unsuccessful_event(
|
||||
|
||||
with caplog.at_level(logging.INFO):
|
||||
service.log_worker_error(
|
||||
recording, event, error="could not connect to the room", error_code=500
|
||||
recording,
|
||||
event,
|
||||
error="could not connect to the room",
|
||||
error_code=500,
|
||||
details="End reason: Failure",
|
||||
)
|
||||
|
||||
assert (
|
||||
@@ -428,6 +459,7 @@ def test_log_worker_error_reports_an_unsuccessful_event(
|
||||
)
|
||||
assert "could not connect to the room" in caplog.text
|
||||
assert "error_code=500" in caplog.text
|
||||
assert "details=End reason: Failure" in caplog.text
|
||||
worker_logs = [
|
||||
record
|
||||
for record in caplog.records
|
||||
@@ -448,7 +480,7 @@ def test_log_worker_error_reports_an_unsuccessful_event(
|
||||
),
|
||||
)
|
||||
def test_log_worker_error_stays_quiet_on_anything_else(event, service, caplog):
|
||||
"""Test log_worker_error ignores events other than FAILED and ABORTED."""
|
||||
"""Test log_worker_error ignores events other than FAILED, INTERRUPTED and ABORTED."""
|
||||
|
||||
recording = RecordingFactory(status="active")
|
||||
|
||||
|
||||
@@ -8,9 +8,12 @@ import hashlib
|
||||
import json
|
||||
from unittest import mock
|
||||
|
||||
from django.utils import timezone
|
||||
|
||||
import pytest
|
||||
from livekit import api
|
||||
|
||||
from ...factories import RecordingFactory
|
||||
from ...services.livekit_events import ActionFailedError, LiveKitEventsService
|
||||
|
||||
|
||||
@@ -184,3 +187,75 @@ def test_action_error(client, mock_livekit_config):
|
||||
content_type="application/json",
|
||||
HTTP_AUTHORIZATION=auth_token,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
@mock.patch(
|
||||
"core.recording.services.recording_events.notification_service."
|
||||
"notify_external_services",
|
||||
return_value=False,
|
||||
)
|
||||
@mock.patch("core.utils.notify_participants")
|
||||
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
|
||||
def test_egress_ended_keeps_a_partial_transcript(
|
||||
mock_update_metadata,
|
||||
mock_notify,
|
||||
mock_notify_external_services,
|
||||
client,
|
||||
mock_livekit_config,
|
||||
):
|
||||
"""Should save a transcript whose egress lost the room but uploaded its file.
|
||||
|
||||
LiveKit Egress v1.14 reports it as failed. The payload mirrors the one it
|
||||
sends, including a field unknown to the pinned livekit-protocol, which
|
||||
must be ignored rather than rejected.
|
||||
"""
|
||||
recording = RecordingFactory(
|
||||
worker_id="EG_hycBMAjmt6Ub", mode="transcript", status="active"
|
||||
)
|
||||
event_data = json.dumps(
|
||||
{
|
||||
"event": "egress_ended",
|
||||
"egressInfo": {
|
||||
"egressId": "EG_hycBMAjmt6Ub",
|
||||
"roomId": "RM_hycBMAjmt6Ub",
|
||||
"roomName": str(recording.room.id),
|
||||
"status": "EGRESS_FAILED",
|
||||
"error": "recording ended early: connection to room failed",
|
||||
"errorCode": 503,
|
||||
"details": "End reason: Source closed",
|
||||
"fileResults": [
|
||||
{
|
||||
"filename": f"recordings/{recording.id}.ogg",
|
||||
"location": f"https://s3.example.com/recordings/{recording.id}.ogg",
|
||||
"size": "123456",
|
||||
}
|
||||
],
|
||||
},
|
||||
"id": "EV_eugWmGhovZmm",
|
||||
"createdAt": str(int(timezone.now().timestamp())),
|
||||
}
|
||||
)
|
||||
hash64 = base64.b64encode(hashlib.sha256(event_data.encode()).digest()).decode()
|
||||
token = api.AccessToken(
|
||||
mock_livekit_config["api_key"], mock_livekit_config["api_secret"]
|
||||
)
|
||||
token.claims.sha256 = hash64
|
||||
|
||||
response = client.post(
|
||||
"/api/v1.0/rooms/webhooks-livekit/",
|
||||
data=event_data,
|
||||
content_type="application/json",
|
||||
HTTP_AUTHORIZATION=token.to_jwt(),
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
mock_notify.assert_called_once_with(
|
||||
room_name=str(recording.room.id),
|
||||
notification_data={"type": "transcriptionInterrupted"},
|
||||
)
|
||||
mock_notify_external_services.assert_called_once_with(recording)
|
||||
|
||||
recording.refresh_from_db()
|
||||
|
||||
assert recording.status == "saved"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Test LiveKitEvents service.
|
||||
"""
|
||||
# pylint: disable=W0621,W0613, W0212, E0611, C0302
|
||||
# pylint: disable=W0621,W0613, W0212, E0611, E1101, C0302
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
@@ -11,6 +11,7 @@ from unittest import mock
|
||||
from django.utils import timezone
|
||||
|
||||
import pytest
|
||||
from faker import Faker
|
||||
from livekit.api import EgressStatus
|
||||
|
||||
from core.audit.testing import find_events
|
||||
@@ -19,12 +20,15 @@ from core.models import Room
|
||||
from core.recording.enums import RecordingWorkerEvent
|
||||
from core.recording.services.recording_events import RecordingEventsService
|
||||
from core.services.livekit_events import (
|
||||
EGRESS_CPU_EXHAUSTED_DETAILS,
|
||||
EGRESS_ROOM_CONNECTION_FAILED_ERROR,
|
||||
EGRESS_STATUS_TO_RECORDING_EVENT,
|
||||
ActionFailedError,
|
||||
AuthenticationError,
|
||||
InvalidPayloadError,
|
||||
LiveKitEventsService,
|
||||
api,
|
||||
to_ended_recording_event,
|
||||
to_recording_event,
|
||||
)
|
||||
from core.services.lobby import LobbyService
|
||||
@@ -39,6 +43,7 @@ from core.services.sip_management import (
|
||||
from core.utils import NotificationError
|
||||
|
||||
pytestmark = pytest.mark.django_db
|
||||
fake = Faker()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -494,6 +499,228 @@ def test_to_recording_event_returns_none_on_unmapped_status(caplog):
|
||||
assert "Unmapped LiveKit egress status" in caplog.text
|
||||
|
||||
|
||||
recording_filename = f"recordings/{fake.file_name(extension='ogg')}"
|
||||
|
||||
UPLOADED_FILE = api.FileInfo(
|
||||
filename=recording_filename,
|
||||
location=f"https://storage.example.com/{recording_filename}",
|
||||
size=1048576,
|
||||
)
|
||||
NOT_UPLOADED_FILE = api.FileInfo(filename=recording_filename)
|
||||
|
||||
ROOM_CONNECTION_FAILED = {
|
||||
"status": EgressStatus.EGRESS_FAILED,
|
||||
"error": EGRESS_ROOM_CONNECTION_FAILED_ERROR,
|
||||
"error_code": 503,
|
||||
"details": "End reason: Source closed",
|
||||
}
|
||||
CPU_EXHAUSTED_DRAINED = {
|
||||
"status": EgressStatus.EGRESS_COMPLETE,
|
||||
"details": EGRESS_CPU_EXHAUSTED_DETAILS,
|
||||
}
|
||||
CPU_EXHAUSTED_KILLED = {
|
||||
"status": EgressStatus.EGRESS_FAILED,
|
||||
"error": "CPU exhausted: 8.5 cores used",
|
||||
"error_code": 403,
|
||||
"details": EGRESS_CPU_EXHAUSTED_DETAILS,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("egress_info", "expected_event"),
|
||||
(
|
||||
(
|
||||
api.EgressInfo(**ROOM_CONNECTION_FAILED, file_results=[UPLOADED_FILE]),
|
||||
RecordingWorkerEvent.INTERRUPTED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(**ROOM_CONNECTION_FAILED, file_results=[NOT_UPLOADED_FILE]),
|
||||
RecordingWorkerEvent.FAILED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(**CPU_EXHAUSTED_DRAINED, file_results=[UPLOADED_FILE]),
|
||||
RecordingWorkerEvent.INTERRUPTED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(**CPU_EXHAUSTED_KILLED, file_results=[UPLOADED_FILE]),
|
||||
RecordingWorkerEvent.INTERRUPTED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(**CPU_EXHAUSTED_KILLED, file_results=[NOT_UPLOADED_FILE]),
|
||||
RecordingWorkerEvent.FAILED,
|
||||
),
|
||||
# Other failures stay failures, even with an uploaded file.
|
||||
(
|
||||
api.EgressInfo(
|
||||
status=EgressStatus.EGRESS_FAILED,
|
||||
error="pipeline frozen",
|
||||
error_code=500,
|
||||
details="End reason: Source closed",
|
||||
file_results=[UPLOADED_FILE],
|
||||
),
|
||||
RecordingWorkerEvent.FAILED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(
|
||||
status=EgressStatus.EGRESS_COMPLETE,
|
||||
details="End reason: StopEgress API",
|
||||
file_results=[UPLOADED_FILE],
|
||||
),
|
||||
RecordingWorkerEvent.COMPLETED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(
|
||||
status=EgressStatus.EGRESS_ABORTED,
|
||||
error="Stop called before pipeline could start",
|
||||
error_code=412,
|
||||
details="End reason: StopEgress API",
|
||||
),
|
||||
RecordingWorkerEvent.ABORTED,
|
||||
),
|
||||
(
|
||||
api.EgressInfo(
|
||||
status=EgressStatus.EGRESS_LIMIT_REACHED,
|
||||
error="Session limit reached",
|
||||
error_code=413,
|
||||
details="End reason: Session limit reached",
|
||||
file_results=[UPLOADED_FILE],
|
||||
),
|
||||
RecordingWorkerEvent.LIMIT_REACHED,
|
||||
),
|
||||
),
|
||||
)
|
||||
def test_to_ended_recording_event_translates_egress_info(egress_info, expected_event):
|
||||
"""Should tell an egress cut short with a partial file apart from the others."""
|
||||
|
||||
assert to_ended_recording_event(egress_info) == expected_event
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mode", "egress_info_kwargs", "notification_type"),
|
||||
(
|
||||
("transcript", ROOM_CONNECTION_FAILED, "transcriptionInterrupted"),
|
||||
("transcript", CPU_EXHAUSTED_DRAINED, "transcriptionInterrupted"),
|
||||
("screen_recording", CPU_EXHAUSTED_DRAINED, "screenRecordingInterrupted"),
|
||||
("screen_recording", CPU_EXHAUSTED_KILLED, "screenRecordingInterrupted"),
|
||||
),
|
||||
)
|
||||
@mock.patch(
|
||||
"core.recording.services.recording_events.notification_service."
|
||||
"notify_external_services"
|
||||
)
|
||||
@mock.patch("core.utils.notify_participants")
|
||||
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
|
||||
def test_handle_egress_ended_interrupted_saves_partial_recording( # noqa: PLR0913, PLR0917
|
||||
mock_update_metadata,
|
||||
mock_notify,
|
||||
mock_notify_external_services,
|
||||
mode,
|
||||
egress_info_kwargs,
|
||||
notification_type,
|
||||
service,
|
||||
caplog,
|
||||
): # pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||
"""Should save an interrupted recording and tell participants it is partial."""
|
||||
|
||||
mock_notify_external_services.return_value = True
|
||||
|
||||
recording = RecordingFactory(worker_id="worker-1", mode=mode, status="active")
|
||||
data = api.WebhookEvent(
|
||||
event="egress_ended",
|
||||
egress_info=api.EgressInfo(
|
||||
egress_id=recording.worker_id,
|
||||
file_results=[UPLOADED_FILE],
|
||||
**egress_info_kwargs,
|
||||
),
|
||||
)
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
service._handle_egress_ended(data)
|
||||
|
||||
mock_notify.assert_called_once_with(
|
||||
room_name=str(recording.room.id), notification_data={"type": notification_type}
|
||||
)
|
||||
mock_notify_external_services.assert_called_once_with(recording)
|
||||
mock_update_metadata.assert_called_once_with(
|
||||
str(recording.room.id), remove_keys=["recording_mode", "recording_status"]
|
||||
)
|
||||
assert (
|
||||
f"Recording worker reported interrupted for recording {recording.id}"
|
||||
in caplog.text
|
||||
)
|
||||
|
||||
recording.refresh_from_db()
|
||||
|
||||
assert recording.status == "notification_succeeded"
|
||||
|
||||
|
||||
@mock.patch(
|
||||
"core.recording.services.recording_events.notification_service."
|
||||
"notify_external_services"
|
||||
)
|
||||
@mock.patch("core.utils.notify_participants")
|
||||
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
|
||||
def test_handle_egress_ended_interrupted_after_stop_does_not_notify_participants(
|
||||
mock_update_metadata, mock_notify, mock_notify_external_services, service
|
||||
):
|
||||
"""Shouldn't notify participants about a recording they already stopped."""
|
||||
|
||||
mock_notify_external_services.return_value = False
|
||||
|
||||
recording = RecordingFactory(worker_id="worker-1", status="stopped")
|
||||
data = api.WebhookEvent(
|
||||
event="egress_ended",
|
||||
egress_info=api.EgressInfo(
|
||||
egress_id=recording.worker_id,
|
||||
file_results=[UPLOADED_FILE],
|
||||
**CPU_EXHAUSTED_DRAINED,
|
||||
),
|
||||
)
|
||||
|
||||
service._handle_egress_ended(data)
|
||||
|
||||
mock_notify.assert_not_called()
|
||||
mock_notify_external_services.assert_called_once_with(recording)
|
||||
|
||||
recording.refresh_from_db()
|
||||
|
||||
assert recording.status == "saved"
|
||||
|
||||
|
||||
@mock.patch("core.utils.notify_participants")
|
||||
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
|
||||
def test_handle_egress_ended_failed_on_another_error_discards_recording(
|
||||
mock_update_metadata, mock_notify, service
|
||||
):
|
||||
"""Should still flag as failed an egress that failed for another reason."""
|
||||
|
||||
recording = RecordingFactory(
|
||||
worker_id="worker-1", mode="transcript", status="active"
|
||||
)
|
||||
data = api.WebhookEvent(
|
||||
event="egress_ended",
|
||||
egress_info=api.EgressInfo(
|
||||
egress_id=recording.worker_id,
|
||||
status=EgressStatus.EGRESS_FAILED,
|
||||
error="pipeline frozen",
|
||||
error_code=500,
|
||||
details="End reason: Source closed",
|
||||
file_results=[UPLOADED_FILE],
|
||||
),
|
||||
)
|
||||
|
||||
service._handle_egress_ended(data)
|
||||
|
||||
mock_notify.assert_called_once_with(
|
||||
room_name=str(recording.room.id),
|
||||
notification_data={"type": "transcriptionFailed"},
|
||||
)
|
||||
|
||||
recording.refresh_from_db()
|
||||
|
||||
assert recording.status == "failed"
|
||||
|
||||
|
||||
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
|
||||
def test_handle_egress_updated_ignores_unmapped_status(mock_update_metadata, service):
|
||||
"""Shouldn't touch the room's metadata when the egress status is unknown."""
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Test the LiveKit reconnection settings exposed by the config endpoint."""
|
||||
|
||||
import pytest
|
||||
from rest_framework.test import APIClient
|
||||
|
||||
pytestmark = pytest.mark.django_db
|
||||
|
||||
|
||||
def test_api_config_retrieve_livekit_reconnect_defaults():
|
||||
"""Without any override, the frontend gets the default backoff."""
|
||||
response = APIClient().get("/api/v1.0/config/")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["livekit"]["reconnect"] == {
|
||||
"backoff_enabled": True,
|
||||
"base_delay_ms": 1000,
|
||||
"max_delay_ms": 10000,
|
||||
"give_up_after_ms": 300000,
|
||||
"backoff_reset_ms": 120000,
|
||||
}
|
||||
|
||||
|
||||
def test_api_config_retrieve_livekit_reconnect_settings(settings):
|
||||
"""Each reconnection setting reaches the frontend unchanged."""
|
||||
settings.LIVEKIT_RECONNECT_BACKOFF_ENABLED = False
|
||||
settings.LIVEKIT_RECONNECT_BASE_DELAY_MS = 500
|
||||
settings.LIVEKIT_RECONNECT_MAX_DELAY_MS = 30000
|
||||
settings.LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS = 600000
|
||||
settings.LIVEKIT_RECONNECT_BACKOFF_RESET_MS = 60000
|
||||
|
||||
response = APIClient().get("/api/v1.0/config/")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["livekit"]["reconnect"] == {
|
||||
"backoff_enabled": False,
|
||||
"base_delay_ms": 500,
|
||||
"max_delay_ms": 30000,
|
||||
"give_up_after_ms": 600000,
|
||||
"backoff_reset_ms": 60000,
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Unit tests for the value of the LiveKit reconnection delay settings."""
|
||||
|
||||
import pytest
|
||||
|
||||
from meet.settings import StrictlyPositiveIntegerValue
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,expected", [("1", 1), (" 15000 ", 15000)])
|
||||
def test_strictly_positive_integer_value_parses(raw, expected):
|
||||
"""A delay above 0 is read as an integer."""
|
||||
assert StrictlyPositiveIntegerValue(environ=False).to_python(raw) == expected
|
||||
|
||||
|
||||
def test_strictly_positive_integer_value_rejects_zero():
|
||||
"""0 stops the settings module loading, unlike with PositiveIntegerValue."""
|
||||
with pytest.raises(ValueError, match="greater than 0"):
|
||||
StrictlyPositiveIntegerValue(environ=False).to_python("0")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", ["-1", "1e3", "1000ms", ""])
|
||||
def test_strictly_positive_integer_value_rejects_invalid(raw):
|
||||
"""A negative or non-integer value stops the settings module loading."""
|
||||
with pytest.raises(ValueError):
|
||||
StrictlyPositiveIntegerValue(environ=False).to_python(raw)
|
||||
@@ -68,6 +68,17 @@ class VideoCodecValue(values.Value):
|
||||
return codec
|
||||
|
||||
|
||||
class StrictlyPositiveIntegerValue(values.PositiveIntegerValue):
|
||||
"""An integer greater than 0"""
|
||||
|
||||
def to_python(self, value):
|
||||
"""Ensure the integer is greater than 0."""
|
||||
int_value = super().to_python(value)
|
||||
if int_value == 0:
|
||||
raise ValueError(f"Expected an integer greater than 0, got {value!r}.")
|
||||
return int_value
|
||||
|
||||
|
||||
class ResolutionSpec(BaseModel):
|
||||
"""An value of RECORDING_ENCODING_AVAILABLE_RESOLUTIONS."""
|
||||
|
||||
@@ -787,6 +798,28 @@ class Base(Configuration):
|
||||
LIVEKIT_DEFAULT_VIDEO_CODEC = VideoCodecValue(
|
||||
"vp9", environ_name="LIVEKIT_DEFAULT_VIDEO_CODEC", environ_prefix=None
|
||||
)
|
||||
# Livekit websocket reconnection exponential backoff config sent to the frontend.
|
||||
# Before its first reconnection attempt, a client waits a random delay up to the base
|
||||
# delay. That ceiling doubles after every failed attempt, capped at the max delay, and
|
||||
# later delays fall between half the ceiling and the ceiling.
|
||||
# It gives up after the give-up delay unless the media still flows.
|
||||
# After a successful reconnection, a new disconnection starting sooner than the reset
|
||||
# delay after the last one resumes the backoff one step above where that one stopped.
|
||||
LIVEKIT_RECONNECT_BACKOFF_ENABLED = values.BooleanValue(
|
||||
True, environ_name="LIVEKIT_RECONNECT_BACKOFF_ENABLED", environ_prefix=None
|
||||
)
|
||||
LIVEKIT_RECONNECT_BASE_DELAY_MS = StrictlyPositiveIntegerValue(
|
||||
1000, environ_name="LIVEKIT_RECONNECT_BASE_DELAY_MS", environ_prefix=None
|
||||
)
|
||||
LIVEKIT_RECONNECT_MAX_DELAY_MS = StrictlyPositiveIntegerValue(
|
||||
10000, environ_name="LIVEKIT_RECONNECT_MAX_DELAY_MS", environ_prefix=None
|
||||
)
|
||||
LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS = StrictlyPositiveIntegerValue(
|
||||
300000, environ_name="LIVEKIT_RECONNECT_GIVE_UP_AFTER_MS", environ_prefix=None
|
||||
)
|
||||
LIVEKIT_RECONNECT_BACKOFF_RESET_MS = values.PositiveIntegerValue(
|
||||
120000, environ_name="LIVEKIT_RECONNECT_BACKOFF_RESET_MS", environ_prefix=None
|
||||
)
|
||||
CONNECTION_TEST_ENABLED = values.BooleanValue(
|
||||
environ_name="CONNECTION_TEST_ENABLED",
|
||||
environ_prefix=None,
|
||||
|
||||
Generated
+18
-8
@@ -27,7 +27,7 @@
|
||||
"i18next-browser-languagedetector": "8.2.1",
|
||||
"i18next-parser": "9.4.0",
|
||||
"i18next-resources-to-backend": "1.2.3",
|
||||
"livekit-client": "2.21.0",
|
||||
"livekit-client": "2.23.0",
|
||||
"posthog-js": "1.418.10",
|
||||
"react": "18.3.1",
|
||||
"react-aria": "3.51.0",
|
||||
@@ -1027,9 +1027,9 @@
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@livekit/protocol": {
|
||||
"version": "1.50.4",
|
||||
"resolved": "https://registry.npmjs.org/@livekit/protocol/-/protocol-1.50.4.tgz",
|
||||
"integrity": "sha512-L1uggNQAqyY21smQY8AllyOYbcv9Me9TaxwuLytL1R8ck9nbYPmQLNwEDi3pOFGAMa5F8I2nUi2Jc59W5awxlA==",
|
||||
"version": "1.52.1",
|
||||
"resolved": "https://registry.npmjs.org/@livekit/protocol/-/protocol-1.52.1.tgz",
|
||||
"integrity": "sha512-X8DxKlNbzk6Lh+v630+mjdFHSotdAFwIyzxSRSUql2DbTxCnlpa7JsvFGc6Na3mXiex2OFh0iiQtWwOBP0rIYA==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@bufbuild/protobuf": "^1.10.0"
|
||||
@@ -8228,16 +8228,17 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/livekit-client": {
|
||||
"version": "2.21.0",
|
||||
"resolved": "https://registry.npmjs.org/livekit-client/-/livekit-client-2.21.0.tgz",
|
||||
"integrity": "sha512-RBUhPkV/sl1nzl8lokVlK5uATPwn0AlsudCBZXissw/kDl9yz8ac4pNJ43iPpMVoHOeBYH/BZ3vUC1adqa/zFQ==",
|
||||
"version": "2.23.0",
|
||||
"resolved": "https://registry.npmjs.org/livekit-client/-/livekit-client-2.23.0.tgz",
|
||||
"integrity": "sha512-j9Kg86nJjBjNFUD+ukp2jwMCGa0k+YD5gU//RzAM+ZY3pbG631QCte7fQjwqWMmtBzhDJL9nKYuOLMv/9IbAxg==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@livekit/mutex": "1.1.1",
|
||||
"@livekit/protocol": "1.50.4",
|
||||
"@livekit/protocol": "1.52.1",
|
||||
"events": "^3.3.0",
|
||||
"jose": "^6.1.0",
|
||||
"loglevel": "^1.9.2",
|
||||
"machina": "^7.0.0",
|
||||
"sdp-transform": "^2.15.0",
|
||||
"tslib": "2.8.1",
|
||||
"typed-emitter": "^2.1.0",
|
||||
@@ -8346,6 +8347,15 @@
|
||||
"yallist": "^3.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/machina": {
|
||||
"version": "7.0.1",
|
||||
"resolved": "https://registry.npmjs.org/machina/-/machina-7.0.1.tgz",
|
||||
"integrity": "sha512-0Xkgs8NeBvO2hW6bjJT8ewLWrMfh1haMPstaghpASWjXkLjRhtDiaJG1Kb7mslaOHxZYxpH7PIjpmirRVYbMMA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=22.22"
|
||||
}
|
||||
},
|
||||
"node_modules/magic-string": {
|
||||
"version": "0.30.21",
|
||||
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
"i18next-browser-languagedetector": "8.2.1",
|
||||
"i18next-parser": "9.4.0",
|
||||
"i18next-resources-to-backend": "1.2.3",
|
||||
"livekit-client": "2.21.0",
|
||||
"livekit-client": "2.23.0",
|
||||
"posthog-js": "1.418.10",
|
||||
"react": "18.3.1",
|
||||
"react-aria": "3.51.0",
|
||||
|
||||
@@ -75,6 +75,13 @@ export interface ApiConfig {
|
||||
enable_firefox_proxy_workaround: boolean
|
||||
default_sources: Source[]
|
||||
default_video_codec: VideoCodec
|
||||
reconnect?: {
|
||||
backoff_enabled: boolean
|
||||
base_delay_ms: number
|
||||
max_delay_ms: number
|
||||
give_up_after_ms: number
|
||||
backoff_reset_ms: number
|
||||
}
|
||||
}
|
||||
transcription_destination?: string
|
||||
max_participants_for_sound: number
|
||||
|
||||
@@ -2,12 +2,6 @@ import type { CaptureResult } from 'posthog-js'
|
||||
|
||||
const IGNORED_EXCEPTION_PATTERNS = [
|
||||
/ResizeObserver loop (completed with undelivered notifications|limit exceeded)/,
|
||||
// livekit-client leaks the raw WebSocket error Event as an unhandled
|
||||
// rejection when the signal ws errors after connect (Firefox-heavy,
|
||||
// coincides with signal reconnects). Carries zero diagnostic content —
|
||||
// the close reason is already logged by the SDK.
|
||||
// See: https://github.com/livekit/client-sdk-js/issues/2062
|
||||
/^Event captured as exception with keys: isTrusted$/,
|
||||
]
|
||||
|
||||
const shouldIgnoreException = (value: unknown): boolean =>
|
||||
|
||||
@@ -96,6 +96,8 @@ export const MainNotificationToast = () => {
|
||||
case NotificationType.ScreenRecordingStopped:
|
||||
case NotificationType.TranscriptionLimitReached:
|
||||
case NotificationType.ScreenRecordingLimitReached:
|
||||
case NotificationType.TranscriptionInterrupted:
|
||||
case NotificationType.ScreenRecordingInterrupted:
|
||||
case NotificationType.TranscriptionFailed:
|
||||
case NotificationType.ScreenRecordingFailed:
|
||||
case NotificationType.TranscriptionAborted:
|
||||
|
||||
@@ -10,6 +10,7 @@ export enum NotificationType {
|
||||
TranscriptionStarted = 'transcriptionStarted',
|
||||
TranscriptionStopped = 'transcriptionStopped',
|
||||
TranscriptionLimitReached = 'transcriptionLimitReached',
|
||||
TranscriptionInterrupted = 'transcriptionInterrupted',
|
||||
TranscriptionFailed = 'transcriptionFailed',
|
||||
TranscriptionAborted = 'transcriptionAborted',
|
||||
TranscriptionRequested = 'transcriptionRequested',
|
||||
@@ -17,6 +18,7 @@ export enum NotificationType {
|
||||
ScreenRecordingStopped = 'screenRecordingStopped',
|
||||
ScreenRecordingRequested = 'screenRecordingRequested',
|
||||
ScreenRecordingLimitReached = 'screenRecordingLimitReached',
|
||||
ScreenRecordingInterrupted = 'screenRecordingInterrupted',
|
||||
ScreenRecordingFailed = 'screenRecordingFailed',
|
||||
ScreenRecordingAborted = 'screenRecordingAborted',
|
||||
RecordingSaving = 'recordingSaving',
|
||||
|
||||
@@ -22,6 +22,8 @@ export function ToastAnyRecording({ state, ...props }: Readonly<ToastProps>) {
|
||||
return 'transcript.stopped'
|
||||
case NotificationType.TranscriptionLimitReached:
|
||||
return 'transcript.limitReached'
|
||||
case NotificationType.TranscriptionInterrupted:
|
||||
return 'transcript.interrupted'
|
||||
case NotificationType.TranscriptionFailed:
|
||||
return 'transcript.failed'
|
||||
case NotificationType.TranscriptionAborted:
|
||||
@@ -32,6 +34,8 @@ export function ToastAnyRecording({ state, ...props }: Readonly<ToastProps>) {
|
||||
return 'screenRecording.stopped'
|
||||
case NotificationType.ScreenRecordingLimitReached:
|
||||
return 'screenRecording.limitReached'
|
||||
case NotificationType.ScreenRecordingInterrupted:
|
||||
return 'screenRecording.interrupted'
|
||||
case NotificationType.ScreenRecordingFailed:
|
||||
return 'screenRecording.failed'
|
||||
case NotificationType.ScreenRecordingAborted:
|
||||
|
||||
@@ -58,6 +58,8 @@ const renderToast = (
|
||||
case NotificationType.ScreenRecordingStarted:
|
||||
case NotificationType.ScreenRecordingStopped:
|
||||
case NotificationType.ScreenRecordingLimitReached:
|
||||
case NotificationType.TranscriptionInterrupted:
|
||||
case NotificationType.ScreenRecordingInterrupted:
|
||||
case NotificationType.TranscriptionFailed:
|
||||
case NotificationType.ScreenRecordingFailed:
|
||||
case NotificationType.TranscriptionAborted:
|
||||
|
||||
@@ -12,6 +12,10 @@ export const useRecordingWaitingForTracks = (mode: RecordingMode) => {
|
||||
mode === RecordingMode.Transcript ? [Track.Source.Microphone] : undefined,
|
||||
{ onlySubscribed: false }
|
||||
)
|
||||
const hasTracks =
|
||||
mode === RecordingMode.Transcript
|
||||
? tracks.some((track) => !track.publication.isMuted)
|
||||
: tracks.length > 0
|
||||
const [delayElapsed, setDelayElapsed] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
@@ -25,5 +29,5 @@ export const useRecordingWaitingForTracks = (mode: RecordingMode) => {
|
||||
return () => window.clearTimeout(timeout)
|
||||
}, [isStarting, mode])
|
||||
|
||||
return isStarting && delayElapsed && tracks.length === 0
|
||||
return isStarting && delayElapsed && !hasTracks
|
||||
}
|
||||
|
||||
@@ -10,7 +10,6 @@ import {
|
||||
ConnectionErrorReason,
|
||||
DisconnectReason,
|
||||
MediaDeviceFailure,
|
||||
Room,
|
||||
type RoomOptions,
|
||||
VideoPresets,
|
||||
} from 'livekit-client'
|
||||
@@ -46,6 +45,26 @@ import { WatchMediaDeviceErrors } from './WatchMediaDeviceErrors'
|
||||
import { MeetDevtools } from '@/features/devtools'
|
||||
import { VOICE_AUDIO_CONSTRAINTS } from '@/features/rooms/livekit/utils/constants'
|
||||
import { resolveAudioOutputDeviceId } from '@/features/rooms/utils/resolveAudioOutputDeviceId'
|
||||
import { ConnectionEndReason } from '@/features/rooms/utils/connectionEndReason'
|
||||
import { createLiveKitRoom } from '@/features/rooms/livekit/utils/reconnectPolicy'
|
||||
|
||||
const handleError = (e: Error) => {
|
||||
const failure = getMediaDeviceFailure(e)
|
||||
if (failure && failure !== MediaDeviceFailure.Other) return
|
||||
|
||||
// connect() was aborted by a disconnect() before the join completed
|
||||
if (
|
||||
e instanceof ConnectionError &&
|
||||
e.reason === ConnectionErrorReason.Cancelled
|
||||
) {
|
||||
void captureEvent('connection-cancelled')
|
||||
return
|
||||
}
|
||||
|
||||
reportError('livekit_room_error', e, {
|
||||
path: 'connect_publish',
|
||||
})
|
||||
}
|
||||
|
||||
export const Conference = ({
|
||||
roomId,
|
||||
@@ -136,7 +155,11 @@ export const Conference = ({
|
||||
apiConfig?.livekit.default_video_codec,
|
||||
])
|
||||
|
||||
const room = useMemo(() => new Room(roomOptions), [roomOptions])
|
||||
const reconnect = apiConfig?.livekit.reconnect
|
||||
const room = useMemo(
|
||||
() => createLiveKitRoom(roomOptions, reconnect),
|
||||
[roomOptions, reconnect]
|
||||
)
|
||||
|
||||
useEffect(() => {
|
||||
/**
|
||||
@@ -186,6 +209,7 @@ export const Conference = ({
|
||||
const isMobile = useIsMobile()
|
||||
|
||||
const hasAutoMutedRef = useRef(false)
|
||||
const hasConnectedRef = useRef(false)
|
||||
|
||||
/*
|
||||
* Ensure stable WebSocket connection URL. This is critical for legacy browser compatibility
|
||||
@@ -224,7 +248,6 @@ export const Conference = ({
|
||||
// Some clients (like DINUM) operate in bandwidth-constrained environments
|
||||
// These settings help ensure successful connections in poor network conditions
|
||||
const connectOptions = {
|
||||
maxRetries: 5, // Default: 1. Only for unreachable server scenarios
|
||||
peerConnectionTimeout: 60000, // Default: 15s. Extended for slow TURN/TLS negotiation
|
||||
}
|
||||
|
||||
@@ -248,24 +271,9 @@ export const Conference = ({
|
||||
className={css({
|
||||
backgroundColor: 'primaryDark.50 !important',
|
||||
})}
|
||||
onError={(e) => {
|
||||
const failure = getMediaDeviceFailure(e)
|
||||
if (failure && failure !== MediaDeviceFailure.Other) return
|
||||
|
||||
// connect() was aborted by a disconnect() before the join completed
|
||||
if (
|
||||
e instanceof ConnectionError &&
|
||||
e.reason === ConnectionErrorReason.Cancelled
|
||||
) {
|
||||
void captureEvent('connection-cancelled')
|
||||
return
|
||||
}
|
||||
|
||||
reportError('livekit_room_error', e, {
|
||||
path: 'connect_publish',
|
||||
})
|
||||
}}
|
||||
onError={handleError}
|
||||
onConnected={async () => {
|
||||
hasConnectedRef.current = true
|
||||
if (!apiConfig) return
|
||||
if (
|
||||
userPreferencesSnap.is_auto_mute_large_room_enabled &&
|
||||
@@ -307,6 +315,20 @@ export const Conference = ({
|
||||
}
|
||||
)
|
||||
return
|
||||
default:
|
||||
// Reconnecting gave up, or the join itself failed
|
||||
navigateTo(
|
||||
'feedback',
|
||||
{},
|
||||
{
|
||||
state: {
|
||||
reason: hasConnectedRef.current
|
||||
? ConnectionEndReason.Lost
|
||||
: ConnectionEndReason.Failed,
|
||||
...metadata,
|
||||
},
|
||||
}
|
||||
)
|
||||
}
|
||||
}}
|
||||
>
|
||||
|
||||
@@ -1,17 +1,29 @@
|
||||
import { useConfig } from '@/api/useConfig'
|
||||
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
|
||||
import {
|
||||
useConnectionState,
|
||||
useRemoteParticipants,
|
||||
useRoomContext,
|
||||
} from '@livekit/components-react'
|
||||
import { useEffect, useRef } from 'react'
|
||||
import { useSnapshot } from 'valtio'
|
||||
import { DisconnectReason, RoomEvent } from 'livekit-client'
|
||||
import {
|
||||
ConnectionState,
|
||||
DisconnectReason,
|
||||
type Room,
|
||||
RoomEvent,
|
||||
} from 'livekit-client'
|
||||
|
||||
import { userPreferencesStore } from '@/stores/userPreferences'
|
||||
|
||||
import { captureEvent, captureMediaEvent } from '@/features/analytics/telemetry'
|
||||
import { connectionObserverStore } from '@/stores/connectionObserver'
|
||||
import { BackoffReconnectPolicy } from '@/features/rooms/livekit/utils/reconnectPolicy'
|
||||
|
||||
const getBackoffPolicy = (room: Room) => {
|
||||
const policy = room.options.reconnectPolicy
|
||||
return policy instanceof BackoffReconnectPolicy ? policy : undefined
|
||||
}
|
||||
|
||||
export const ConnectionObserver = () => {
|
||||
const room = useRoomContext()
|
||||
@@ -31,6 +43,7 @@ export const ConnectionObserver = () => {
|
||||
RoomEvent.ParticipantDisconnected,
|
||||
],
|
||||
})
|
||||
const connectionState = useConnectionState(room)
|
||||
|
||||
useEffect(() => {
|
||||
// Always clear existing timer on dependency change
|
||||
@@ -48,6 +61,10 @@ export const ConnectionObserver = () => {
|
||||
return
|
||||
}
|
||||
|
||||
// A full reconnection removes every remote participant until it
|
||||
// succeeds: being alone then doesn't mean being idle
|
||||
if (connectionState !== ConnectionState.Connected) return
|
||||
|
||||
if (remoteParticipants.length === 0) {
|
||||
idleDisconnectModalTimeoutRef.current = setTimeout(() => {
|
||||
connectionObserverStore.isIdleDisconnectModalOpen = true
|
||||
@@ -64,6 +81,7 @@ export const ConnectionObserver = () => {
|
||||
}
|
||||
}, [
|
||||
remoteParticipants.length,
|
||||
connectionState,
|
||||
data?.idle_disconnect_warning_delay,
|
||||
userPreferencesSnap.is_idle_disconnect_modal_enabled,
|
||||
])
|
||||
@@ -89,11 +107,13 @@ export const ConnectionObserver = () => {
|
||||
}
|
||||
|
||||
const handleReconnected = async () => {
|
||||
const durationMs = getBackoffPolicy(room)?.lastReconnectionDurationMs
|
||||
const participantSid = room.localParticipant.sid
|
||||
const roomSid = await room.getSid().catch(() => undefined)
|
||||
captureEvent('reconnected-event', {
|
||||
livekit_room_sid: roomSid,
|
||||
livekit_participant_sid: participantSid,
|
||||
durationMs,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -116,9 +136,12 @@ export const ConnectionObserver = () => {
|
||||
sessionDuration: connectionStartTimeRef.current
|
||||
? connectionEndTime - connectionStartTimeRef.current
|
||||
: -1,
|
||||
reason: disconnectReason
|
||||
? DisconnectReason[disconnectReason]
|
||||
: 'UNKNOWN',
|
||||
// LiveKit reports no reason when the reconnect policy gives up
|
||||
reason: getBackoffPolicy(room)?.gaveUp
|
||||
? 'RECONNECT_GAVE_UP'
|
||||
: disconnectReason
|
||||
? DisconnectReason[disconnectReason]
|
||||
: 'UNKNOWN',
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
import { ConnectionState, RoomEvent, type Room } from 'livekit-client'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
BackoffReconnectPolicy,
|
||||
createLiveKitRoom,
|
||||
MAX_FIRST_DELAY_MS,
|
||||
} from './reconnectPolicy'
|
||||
|
||||
const CONFIG = {
|
||||
backoff_enabled: true,
|
||||
base_delay_ms: 1_000,
|
||||
max_delay_ms: 15_000,
|
||||
give_up_after_ms: 300_000,
|
||||
backoff_reset_ms: 120_000,
|
||||
}
|
||||
|
||||
const createRoom = () => {
|
||||
const listeners = new Map<RoomEvent, Array<(...args: unknown[]) => void>>()
|
||||
const room = {
|
||||
on(event: RoomEvent, listener: (...args: unknown[]) => void) {
|
||||
listeners.set(event, [...(listeners.get(event) ?? []), listener])
|
||||
return room
|
||||
},
|
||||
emit(event: RoomEvent, ...args: unknown[]) {
|
||||
listeners.get(event)?.forEach((listener) => listener(...args))
|
||||
},
|
||||
setState(state: ConnectionState, event?: RoomEvent) {
|
||||
room.emit(RoomEvent.ConnectionStateChanged, state)
|
||||
if (event) room.emit(event)
|
||||
},
|
||||
}
|
||||
return room
|
||||
}
|
||||
|
||||
const setup = ({ config = CONFIG, random = 0.5 } = {}) => {
|
||||
const clock = { now: 0 }
|
||||
|
||||
vi.spyOn(Math, 'random').mockReturnValue(random)
|
||||
vi.spyOn(performance, 'now').mockImplementation(() => clock.now)
|
||||
|
||||
const room = createRoom()
|
||||
const policy = new BackoffReconnectPolicy(config)
|
||||
policy.attach(room as unknown as Room)
|
||||
|
||||
const delay = (retryCount: number, elapsedMs = 0) =>
|
||||
policy.nextRetryDelayInMs({ retryCount, elapsedMs })
|
||||
|
||||
return { clock, room, policy, delay }
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
describe('BackoffReconnectPolicy', () => {
|
||||
it('doubles the delay ceiling after each failed attempt, up to the cap', () => {
|
||||
const { delay } = setup()
|
||||
|
||||
expect([0, 1, 2, 3, 4, 10].map((retryCount) => delay(retryCount))).toEqual([
|
||||
500, 1_500, 3_000, 6_000, 11_250, 11_250,
|
||||
])
|
||||
})
|
||||
|
||||
it('draws the first delay anywhere below its ceiling (check two extremes)', () => {
|
||||
expect(setup({ random: 0 }).delay(0)).toBe(0)
|
||||
expect(setup({ random: 0.9999 }).delay(0)).toBe(999)
|
||||
})
|
||||
|
||||
it('draws the next delays between half their ceiling and their ceiling (check two extremes)', () => {
|
||||
expect(setup({ random: 0 }).delay(3)).toBe(4_000)
|
||||
expect(setup({ random: 0.9999 }).delay(3)).toBe(7_999)
|
||||
})
|
||||
|
||||
it('keeps the first delay of a reconnection under 5 s', () => {
|
||||
const config = { ...CONFIG, base_delay_ms: 8_000 }
|
||||
const { delay } = setup({ config, random: 0.9999 })
|
||||
|
||||
expect(delay(0)).toBeLessThan(MAX_FIRST_DELAY_MS)
|
||||
expect(delay(1)).toBe(14_999)
|
||||
})
|
||||
|
||||
it('starts the first attempt within 5 s of the drop', () => {
|
||||
const config = { ...CONFIG, base_delay_ms: 8_000 }
|
||||
const { clock, delay } = setup({ config })
|
||||
expect(delay(0)).toBe(2_500)
|
||||
|
||||
clock.now = 4_000
|
||||
expect(delay(0)).toBe(500)
|
||||
|
||||
clock.now = 6_000
|
||||
expect(delay(0)).toBe(0)
|
||||
})
|
||||
|
||||
it('never steps the backoff down within a reconnection', () => {
|
||||
const { delay } = setup()
|
||||
|
||||
expect(delay(3)).toBe(6_000)
|
||||
expect(delay(0)).toBe(6_000)
|
||||
})
|
||||
|
||||
it('ignores the elapsed time reported by LiveKit', () => {
|
||||
const { delay } = setup()
|
||||
|
||||
expect(delay(1, Date.now())).toBe(1_500)
|
||||
})
|
||||
|
||||
it('picks up the backoff of a reconnection that ended shortly before', () => {
|
||||
const { clock, room, delay } = setup()
|
||||
delay(0)
|
||||
delay(1)
|
||||
delay(2)
|
||||
room.emit(RoomEvent.Reconnected)
|
||||
|
||||
clock.now += 60_000
|
||||
// One step above where the last reconnection ended (capped)
|
||||
expect(delay(0)).toBe(2_500)
|
||||
expect(delay(1)).toBe(11_250)
|
||||
})
|
||||
|
||||
it('starts the backoff over once the connection held for the reset delay', () => {
|
||||
const { clock, room, delay } = setup()
|
||||
delay(0)
|
||||
delay(1)
|
||||
delay(2)
|
||||
room.emit(RoomEvent.Reconnected)
|
||||
|
||||
clock.now += 120_000
|
||||
|
||||
expect(delay(0)).toBe(500)
|
||||
})
|
||||
|
||||
it('gives up once the reconnection lasts longer than the give-up delay', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
delay(0)
|
||||
room.setState(ConnectionState.Reconnecting, RoomEvent.Reconnecting)
|
||||
|
||||
clock.now = 299_999
|
||||
|
||||
expect(delay(20)).toBe(11_250)
|
||||
expect(policy.gaveUp).toBe(false)
|
||||
|
||||
clock.now = 300_000
|
||||
|
||||
expect(delay(21)).toBeNull()
|
||||
expect(policy.gaveUp).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps trying while only signaling is down', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
delay(0)
|
||||
room.setState(
|
||||
ConnectionState.SignalReconnecting,
|
||||
RoomEvent.SignalReconnecting
|
||||
)
|
||||
|
||||
clock.now = 600_000
|
||||
|
||||
expect(delay(40)).toBe(11_250)
|
||||
expect(policy.gaveUp).toBe(false)
|
||||
|
||||
// The media connection failed as well
|
||||
room.setState(ConnectionState.Reconnecting, RoomEvent.Reconnecting)
|
||||
|
||||
expect(delay(41)).toBeNull()
|
||||
expect(policy.gaveUp).toBe(true)
|
||||
})
|
||||
|
||||
it('gives up once the media connection failed, even back to resuming', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
delay(0)
|
||||
room.setState(
|
||||
ConnectionState.SignalReconnecting,
|
||||
RoomEvent.SignalReconnecting
|
||||
)
|
||||
delay(1)
|
||||
|
||||
// The peer connections fail, then LiveKit tries to resume again
|
||||
clock.now = 30_000
|
||||
room.setState(ConnectionState.Reconnecting, RoomEvent.Reconnecting)
|
||||
room.setState(
|
||||
ConnectionState.SignalReconnecting,
|
||||
RoomEvent.SignalReconnecting
|
||||
)
|
||||
|
||||
clock.now = 299_999
|
||||
|
||||
expect(delay(20)).toBe(11_250)
|
||||
expect(policy.gaveUp).toBe(false)
|
||||
|
||||
clock.now = 300_000
|
||||
|
||||
expect(delay(21)).toBeNull()
|
||||
expect(policy.gaveUp).toBe(true)
|
||||
})
|
||||
|
||||
it('starts a new reconnection once the room connects again', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
delay(0)
|
||||
room.setState(ConnectionState.Reconnecting, RoomEvent.Reconnecting)
|
||||
clock.now = 300_000
|
||||
|
||||
expect(delay(21)).toBeNull()
|
||||
|
||||
room.setState(ConnectionState.Disconnected, RoomEvent.Disconnected)
|
||||
expect(policy.gaveUp).toBe(true)
|
||||
|
||||
room.setState(ConnectionState.Connecting)
|
||||
room.setState(ConnectionState.Connected, RoomEvent.Connected)
|
||||
expect(policy.gaveUp).toBe(false)
|
||||
|
||||
clock.now += 600_000
|
||||
|
||||
expect(delay(0)).toBe(500)
|
||||
|
||||
// The media connection of the previous reconnection is not lost anymore
|
||||
room.setState(
|
||||
ConnectionState.SignalReconnecting,
|
||||
RoomEvent.SignalReconnecting
|
||||
)
|
||||
clock.now += 300_000
|
||||
|
||||
expect(delay(20)).toBe(11_250)
|
||||
})
|
||||
|
||||
it('starts over when a new join replaces the reconnection', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
delay(0)
|
||||
delay(3)
|
||||
room.setState(ConnectionState.Reconnecting, RoomEvent.Reconnecting)
|
||||
|
||||
// LiveKitRoom calls room.connect() again in the middle of the reconnection
|
||||
clock.now = 400_000
|
||||
room.setState(ConnectionState.Connecting)
|
||||
|
||||
expect(delay(0)).toBe(500)
|
||||
expect(policy.gaveUp).toBe(false)
|
||||
})
|
||||
|
||||
it('reports how long a reconnection lasted, once', () => {
|
||||
const { clock, room, policy, delay } = setup()
|
||||
clock.now = 1_000
|
||||
delay(0)
|
||||
|
||||
clock.now = 4_000
|
||||
room.emit(RoomEvent.Reconnected)
|
||||
expect(policy.lastReconnectionDurationMs).toBe(3_000)
|
||||
|
||||
// LiveKit can emit Reconnected several times for one recovery
|
||||
room.emit(RoomEvent.Reconnected)
|
||||
expect(policy.lastReconnectionDurationMs).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('createLiveKitRoom', () => {
|
||||
it('builds the policy from the config endpoint values', () => {
|
||||
vi.spyOn(Math, 'random').mockReturnValue(0.5)
|
||||
|
||||
const { reconnectPolicy } = createLiveKitRoom({}, CONFIG).options
|
||||
|
||||
expect(reconnectPolicy).toBeInstanceOf(BackoffReconnectPolicy)
|
||||
expect(
|
||||
reconnectPolicy.nextRetryDelayInMs({ retryCount: 2, elapsedMs: 0 })
|
||||
).toBe(3_000)
|
||||
})
|
||||
|
||||
it('keeps the default policy when the backoff is missing or disabled', () => {
|
||||
for (const config of [undefined, { ...CONFIG, backoff_enabled: false }]) {
|
||||
expect(
|
||||
createLiveKitRoom({}, config).options.reconnectPolicy
|
||||
).not.toBeInstanceOf(BackoffReconnectPolicy)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,146 @@
|
||||
import {
|
||||
ConnectionState,
|
||||
Room,
|
||||
RoomEvent,
|
||||
type ReconnectContext,
|
||||
type ReconnectPolicy,
|
||||
type RoomOptions,
|
||||
} from 'livekit-client'
|
||||
import type { ApiConfig } from '@/api/useConfig'
|
||||
|
||||
/**
|
||||
* Until the first attempt of a reconnection starts, the room still reports
|
||||
* Connected and LiveKit keeps checking its transport for a while.
|
||||
* Around 10s after the initial drop, it will try a full websocket reconnection.
|
||||
* Hence, the first attempt should start well before that.
|
||||
*/
|
||||
export const MAX_FIRST_DELAY_MS = 5_000
|
||||
|
||||
type ReconnectConfig = NonNullable<ApiConfig['livekit']['reconnect']>
|
||||
|
||||
type Reconnection = {
|
||||
startedAt: number
|
||||
// Higher than 0 when the previous reconnection ended shortly before
|
||||
firstStep: number
|
||||
failedAttempts: number
|
||||
isMediaLost: boolean
|
||||
gaveUp: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* When a LiveKit node crashes or Redis stalls, every client loses its signal
|
||||
* connection at the same time. This policy implements an exponential backoff
|
||||
* with some randomness to spread the reconnections out.
|
||||
*
|
||||
* The policy keeps its own clock since LiveKit calls the policy again on every
|
||||
* new disconnection trigger
|
||||
*/
|
||||
export class BackoffReconnectPolicy implements ReconnectPolicy {
|
||||
lastReconnectionDurationMs: number | undefined
|
||||
private readonly config: ReconnectConfig
|
||||
private reconnection: Reconnection | undefined
|
||||
private lastRecovery: { at: number; step: number } | undefined
|
||||
|
||||
constructor(config: ReconnectConfig) {
|
||||
this.config = config
|
||||
}
|
||||
|
||||
attach(room: Room) {
|
||||
room.on(RoomEvent.Connected, this.handleRecovery)
|
||||
room.on(RoomEvent.Reconnected, this.handleRecovery)
|
||||
room.on(RoomEvent.Reconnecting, this.handleMediaLost)
|
||||
room.on(RoomEvent.ConnectionStateChanged, this.handleStateChange)
|
||||
}
|
||||
|
||||
get gaveUp() {
|
||||
return this.reconnection?.gaveUp ?? false
|
||||
}
|
||||
|
||||
nextRetryDelayInMs({ retryCount }: ReconnectContext): number | null {
|
||||
const now = performance.now()
|
||||
this.reconnection ??= this.startReconnection(now)
|
||||
const reconnection = this.reconnection
|
||||
|
||||
reconnection.failedAttempts = Math.max(
|
||||
reconnection.failedAttempts,
|
||||
retryCount
|
||||
)
|
||||
|
||||
if (
|
||||
now - reconnection.startedAt >= this.config.give_up_after_ms &&
|
||||
reconnection.isMediaLost
|
||||
) {
|
||||
reconnection.gaveUp = true
|
||||
return null
|
||||
}
|
||||
|
||||
const ceiling = Math.min(
|
||||
this.config.max_delay_ms,
|
||||
this.config.base_delay_ms *
|
||||
2 ** (reconnection.firstStep + reconnection.failedAttempts)
|
||||
)
|
||||
|
||||
if (reconnection.failedAttempts === 0) {
|
||||
// LiveKit re-arms its timer on each trigger: count from the first drop
|
||||
const timeLeft = reconnection.startedAt + MAX_FIRST_DELAY_MS - now
|
||||
return Math.floor(
|
||||
Math.random() * Math.min(ceiling, Math.max(timeLeft, 0))
|
||||
)
|
||||
}
|
||||
return Math.floor((ceiling / 2) * (1 + Math.random()))
|
||||
}
|
||||
|
||||
private startReconnection(now: number): Reconnection {
|
||||
const recovery = this.lastRecovery
|
||||
|
||||
const isLastDisconnectionRecent =
|
||||
recovery !== undefined && now - recovery.at < this.config.backoff_reset_ms
|
||||
|
||||
return {
|
||||
startedAt: now,
|
||||
firstStep: isLastDisconnectionRecent ? recovery.step + 1 : 0,
|
||||
failedAttempts: 0,
|
||||
isMediaLost: false,
|
||||
gaveUp: false,
|
||||
}
|
||||
}
|
||||
|
||||
private handleMediaLost = () => {
|
||||
this.reconnection ??= this.startReconnection(performance.now())
|
||||
this.reconnection.isMediaLost = true
|
||||
}
|
||||
|
||||
private handleStateChange = (state: ConnectionState) => {
|
||||
if (state === ConnectionState.Connecting) this.reconnection = undefined
|
||||
}
|
||||
|
||||
private handleRecovery = () => {
|
||||
const now = performance.now()
|
||||
if (this.reconnection) {
|
||||
const { startedAt, firstStep, failedAttempts } = this.reconnection
|
||||
this.lastRecovery = { at: now, step: firstStep + failedAttempts }
|
||||
this.lastReconnectionDurationMs = now - startedAt
|
||||
this.reconnection = undefined
|
||||
} else {
|
||||
// LiveKit can emit Reconnected several times for a full reconnection
|
||||
// We want to avoid duplicating the reconnection duration in PostHog
|
||||
this.lastReconnectionDurationMs = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a room with the reconnection policy from the config endpoint.
|
||||
* Without it, the room keeps the LiveKit default policy
|
||||
*/
|
||||
export const createLiveKitRoom = (
|
||||
options: RoomOptions,
|
||||
config?: ReconnectConfig
|
||||
): Room => {
|
||||
if (!config?.backoff_enabled) return new Room(options)
|
||||
|
||||
const reconnectPolicy = new BackoffReconnectPolicy(config)
|
||||
const room = new Room({ ...options, reconnectPolicy })
|
||||
reconnectPolicy.attach(room)
|
||||
return room
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { Rating } from '@/features/rooms/components/Rating.tsx'
|
||||
import { useLocation } from 'wouter'
|
||||
import { useMemo } from 'react'
|
||||
import { DisconnectReason } from 'livekit-client'
|
||||
import { ConnectionEndReason } from '@/features/rooms/utils/connectionEndReason'
|
||||
|
||||
// fixme - duplicated with home, refactor in a proper style
|
||||
const Heading = styled('h1', {
|
||||
@@ -32,6 +33,8 @@ enum DisconnectReasonKey {
|
||||
DuplicateIdentity = 'duplicateIdentity',
|
||||
ParticipantRemoved = 'participantRemoved',
|
||||
RoomDeleted = 'roomDeleted',
|
||||
ConnectionLost = 'connectionLost',
|
||||
ConnectionFailed = 'connectionFailed',
|
||||
}
|
||||
|
||||
const FeedbackRoute = () => {
|
||||
@@ -49,6 +52,10 @@ const FeedbackRoute = () => {
|
||||
return DisconnectReasonKey.ParticipantRemoved
|
||||
case DisconnectReason.ROOM_DELETED:
|
||||
return DisconnectReasonKey.RoomDeleted
|
||||
case ConnectionEndReason.Lost:
|
||||
return DisconnectReasonKey.ConnectionLost
|
||||
case ConnectionEndReason.Failed:
|
||||
return DisconnectReasonKey.ConnectionFailed
|
||||
}
|
||||
}, [])
|
||||
|
||||
@@ -91,7 +98,9 @@ const FeedbackRoute = () => {
|
||||
{t('feedback.home')}
|
||||
</Button>
|
||||
</Stack>
|
||||
<Rating metadata={metadata} />
|
||||
{reasonKey !== DisconnectReasonKey.ConnectionFailed && (
|
||||
<Rating metadata={metadata} />
|
||||
)}
|
||||
</VStack>
|
||||
</Center>
|
||||
</Screen>
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
/**
|
||||
* Why a room ended when no more specific reason applies
|
||||
*/
|
||||
export enum ConnectionEndReason {
|
||||
Failed = 'connectionFailed',
|
||||
Lost = 'connectionLost',
|
||||
}
|
||||
@@ -37,6 +37,7 @@
|
||||
"started": "{{name}} hat die Meeting-Transkription gestartet.",
|
||||
"stopped": "{{name}} hat die Meeting-Transkription gestoppt.",
|
||||
"limitReached": "Die Transkription hat die maximal zulässige Dauer überschritten und wird automatisch gespeichert.",
|
||||
"interrupted": "Die Transkription wurde unerwartet beendet, der bereits aufgezeichnete Teil wird gespeichert.",
|
||||
"failed": "Die Transkription wurde unerwartet beendet und konnte nicht gespeichert werden.",
|
||||
"aborted": "Die Transkription konnte nicht gestartet werden.",
|
||||
"requested": "{{name}} möchte die Meeting-Transkription starten."
|
||||
@@ -46,6 +47,7 @@
|
||||
"started": "{{name}} hat die Meeting-Aufzeichnung gestartet.",
|
||||
"stopped": "{{name}} hat die Meeting-Aufzeichnung gestoppt.",
|
||||
"limitReached": "Die Aufzeichnung hat die maximal zulässige Dauer überschritten und wird automatisch gespeichert.",
|
||||
"interrupted": "Die Aufzeichnung wurde unerwartet beendet, der bereits aufgezeichnete Teil wird gespeichert.",
|
||||
"failed": "Die Aufzeichnung wurde unerwartet beendet und konnte nicht gespeichert werden.",
|
||||
"aborted": "Die Aufzeichnung konnte nicht gestartet werden.",
|
||||
"requested": "{{name}} möchte die Meeting-Aufzeichnung starten."
|
||||
|
||||
@@ -4,7 +4,9 @@
|
||||
"normal": "Du hast das Meeting verlassen",
|
||||
"duplicateIdentity": "Du bist dem Meeting von einem anderen Gerät aus beigetreten",
|
||||
"participantRemoved": "Du wurdest vom Host aus dem Meeting entfernt",
|
||||
"roomDeleted": "Dieses Meeting wurde gelöscht"
|
||||
"roomDeleted": "Dieses Meeting wurde gelöscht",
|
||||
"connectionLost": "Die Verbindung zum Meeting wurde unterbrochen",
|
||||
"connectionFailed": "Beitritt zum Meeting nicht möglich"
|
||||
},
|
||||
"home": "Zur Startseite zurückkehren",
|
||||
"back": "Dem Meeting erneut beitreten"
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"started": "{{name}} started the meeting transcription.",
|
||||
"stopped": "{{name}} stopped the meeting transcription.",
|
||||
"limitReached": "The transcription has exceeded the maximum allowed duration and will be automatically saved.",
|
||||
"interrupted": "The transcription stopped unexpectedly, the part already recorded will be saved.",
|
||||
"failed": "The transcription stopped unexpectedly and could not be saved.",
|
||||
"aborted": "The transcription could not be started.",
|
||||
"requested": "{{name}} wants to start the meeting transcription."
|
||||
@@ -46,6 +47,7 @@
|
||||
"started": "{{name}} started the meeting recording.",
|
||||
"stopped": "{{name}} stopped the meeting recording.",
|
||||
"limitReached": "The recording has exceeded the maximum allowed duration and will be automatically saved.",
|
||||
"interrupted": "The recording stopped unexpectedly, the part already recorded will be saved.",
|
||||
"failed": "The recording stopped unexpectedly and could not be saved.",
|
||||
"aborted": "The recording could not be started.",
|
||||
"requested": "{{name}} wants to start the meeting recording."
|
||||
|
||||
@@ -4,7 +4,9 @@
|
||||
"normal": "You have left the meeting",
|
||||
"duplicateIdentity": "You have joined the meeting from another device",
|
||||
"participantRemoved": "You have been removed from the meeting by a host",
|
||||
"roomDeleted": "This meeting has been deleted"
|
||||
"roomDeleted": "This meeting has been deleted",
|
||||
"connectionLost": "The connection to the meeting was lost",
|
||||
"connectionFailed": "Unable to join the meeting"
|
||||
},
|
||||
"home": "Return to home",
|
||||
"back": "Rejoin the meeting"
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"started": "{{name}} ha iniciado la transcripción de la reunión.",
|
||||
"stopped": "{{name}} ha detenido la transcripción de la reunión.",
|
||||
"limitReached": "La transcripción ha superado la duración máxima permitida, se va a guardar automáticamente.",
|
||||
"interrupted": "La transcripción se ha interrumpido de forma inesperada, se va a guardar la parte ya grabada.",
|
||||
"failed": "La transcripción se ha interrumpido de forma inesperada y no se ha podido guardar.",
|
||||
"aborted": "No se ha podido iniciar la transcripción.",
|
||||
"requested": "{{name}} ha solicitado iniciar la transcripción."
|
||||
@@ -46,6 +47,7 @@
|
||||
"started": "{{name}} ha iniciado la grabación de la reunión.",
|
||||
"stopped": "{{name}} ha detenido la grabación de la reunión.",
|
||||
"limitReached": "La grabación ha superado la duración máxima permitida, se va a guardar automáticamente.",
|
||||
"interrupted": "La grabación se ha interrumpido de forma inesperada, se va a guardar la parte ya grabada.",
|
||||
"failed": "La grabación se ha interrumpido de forma inesperada y no se ha podido guardar.",
|
||||
"aborted": "No se ha podido iniciar la grabación.",
|
||||
"requested": "{{name}} ha solicitado iniciar la grabación."
|
||||
|
||||
@@ -4,7 +4,9 @@
|
||||
"normal": "Has salido de la reunión",
|
||||
"duplicateIdentity": "Te has unido a la reunión desde otro dispositivo",
|
||||
"participantRemoved": "Un administrador te ha expulsado de la llamada",
|
||||
"roomDeleted": "Esta reunión ha sido eliminada"
|
||||
"roomDeleted": "Esta reunión ha sido eliminada",
|
||||
"connectionLost": "Se ha perdido la conexión con la reunión",
|
||||
"connectionFailed": "No se ha podido acceder a la reunión"
|
||||
},
|
||||
"home": "Volver al inicio",
|
||||
"back": "Volver a la reunión"
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"started": "{{name}} a démarré la transcription de la réunion.",
|
||||
"stopped": "{{name}} a arrêté la transcription de la réunion.",
|
||||
"limitReached": "La transcription a dépassé la durée maximale autorisée, elle va être automatiquement sauvegardée.",
|
||||
"interrupted": "La transcription s'est interrompue de façon inattendue, la partie déjà enregistrée va être sauvegardée.",
|
||||
"failed": "La transcription s'est interrompue de façon inattendue et n'a pas pu être sauvegardée.",
|
||||
"aborted": "La transcription n'a pas pu être démarrée.",
|
||||
"requested": "{{name}} a demandé à démarrer la transcription."
|
||||
@@ -46,6 +47,7 @@
|
||||
"started": "{{name}} a démarré l'enregistrement de la réunion.",
|
||||
"stopped": "{{name}} a arrêté l'enregistrement de la réunion.",
|
||||
"limitReached": "L'enregistrement a dépassé la durée maximale autorisée, il va être automatiquement sauvegardé.",
|
||||
"interrupted": "L'enregistrement s'est interrompu de façon inattendue, la partie déjà enregistrée va être sauvegardée.",
|
||||
"failed": "L'enregistrement s'est interrompu de façon inattendue et n'a pas pu être sauvegardé.",
|
||||
"aborted": "L'enregistrement n'a pas pu être démarré.",
|
||||
"requested": "{{name}} a demandé à démarrer l'enregistrement."
|
||||
|
||||
@@ -4,7 +4,9 @@
|
||||
"normal": "Vous avez quitté la réunion",
|
||||
"duplicateIdentity": "Vous avez rejoint la réunion depuis un autre appareil",
|
||||
"participantRemoved": "Vous avez été exclu de l'appel par un administrateur",
|
||||
"roomDeleted": "Cette réunion a été supprimée"
|
||||
"roomDeleted": "Cette réunion a été supprimée",
|
||||
"connectionLost": "La connexion à la réunion a été perdue",
|
||||
"connectionFailed": "Impossible de rejoindre la réunion"
|
||||
},
|
||||
"home": "Retourner à l'accueil",
|
||||
"back": "Réintégrer la réunion"
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
"started": "{{name}} is de transcriptie van de vergadering gestart.",
|
||||
"stopped": "{{name}} heeft de transcriptie van de vergadering gestopt.",
|
||||
"limitReached": "De transcriptie heeft de maximaal toegestane duur overschreden en wordt automatisch opgeslagen.",
|
||||
"interrupted": "De transcriptie is onverwacht gestopt, het al opgenomen deel wordt opgeslagen.",
|
||||
"failed": "De transcriptie is onverwacht gestopt en kon niet worden opgeslagen.",
|
||||
"aborted": "De transcriptie kon niet worden gestart.",
|
||||
"requested": "{{name}} wil graag de transcriptie starten."
|
||||
@@ -46,6 +47,7 @@
|
||||
"started": "{{name}} is begonnen met het opnemen van de vergadering.",
|
||||
"stopped": "{{name}} is gestopt met het opnemen van de vergadering.",
|
||||
"limitReached": "De opname heeft de maximaal toegestane duur overschreden en wordt automatisch opgeslagen.",
|
||||
"interrupted": "De opname is onverwacht gestopt, het al opgenomen deel wordt opgeslagen.",
|
||||
"failed": "De opname is onverwacht gestopt en kon niet worden opgeslagen.",
|
||||
"aborted": "De opname kon niet worden gestart.",
|
||||
"requested": "{{name}} wil graag de opname starten."
|
||||
|
||||
@@ -4,7 +4,9 @@
|
||||
"normal": "Je hebt de vergadering verlaten",
|
||||
"duplicateIdentity": "U heeft de vergadering via een ander apparaat geopend",
|
||||
"participantRemoved": "U bent door een beheerder uit het gesprek verwijderd",
|
||||
"roomDeleted": "Deze vergadering is verwijderd"
|
||||
"roomDeleted": "Deze vergadering is verwijderd",
|
||||
"connectionLost": "De verbinding met de vergadering is verbroken",
|
||||
"connectionFailed": "Deelnemen aan de vergadering is niet gelukt"
|
||||
},
|
||||
"home": "Keer terug naar het hoofdscherm",
|
||||
"back": "Sluit weer bij de vergadering aan"
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
replicaCount: 1
|
||||
terminationGracePeriodSeconds: 18000
|
||||
|
||||
image:
|
||||
tag: v1.15.0
|
||||
|
||||
egress:
|
||||
log_level: debug
|
||||
logging:
|
||||
level: debug
|
||||
ws_url: ws://livekit-livekit-server:80
|
||||
insecure: true
|
||||
enable_chrome_sandbox: true
|
||||
|
||||
@@ -2,10 +2,11 @@ replicaCount: 1
|
||||
terminationGracePeriodSeconds: 18000
|
||||
|
||||
image:
|
||||
tag: v1.11.0
|
||||
tag: v1.15.0
|
||||
|
||||
egress:
|
||||
log_level: debug
|
||||
logging:
|
||||
level: debug
|
||||
ws_url: ws://livekit-livekit-server:80
|
||||
insecure: true
|
||||
enable_chrome_sandbox: true
|
||||
|
||||
Reference in New Issue
Block a user