Compare commits

..

1 Commits

Author SHA1 Message Date
lebaudantoine f4c315bdbf wip connection tester updated 2026-08-25 23:14:05 +02:00
40 changed files with 1377 additions and 141 deletions
+1 -8
View File
@@ -8,13 +8,6 @@ and this project adheres to
## [Unreleased]
### Changed
- 🔥(backend) remove the S3 storage-event webhook for recordings
- ♻️(backend) always finalize recordings using the LiveKit egress_ended webhook
## [1.29.0] - 2026-08-25
### Added
- ✨(any) let any authenticated user manage the lobby on trusted rooms
@@ -31,7 +24,6 @@ and this project adheres to
- ⬆️(frontend) upgrade posthog-js from 1.404.1 to 1.409.5
- ⚡️(frontend) apply frugal constraint to the active meeting audio track
- ⚡️(backend) replace blocking Redis KEYS with cursor-based SCAN
- ✨(summary) add hostname to analytics properties
## [1.28.0] - 2026-08-24
@@ -70,6 +62,7 @@ and this project adheres to
- 🔥(frontend) drop unused vendored ConnectionObserver
- 🐛(frontend) vendor formatChatMessageLinks and trim surrounding newlines
- ✨(summary) add hostname to analytics properties
### Fixed
-17
View File
@@ -16,23 +16,6 @@ the following command inside your docker container:
## [Unreleased]
### Removing S3 storage-event webhooks for recordings
Recordings were previously confirmed as saved by an S3 storage-event webhook posting to `/api/v1.0/recordings/storage-hook/`. That endpoint has been removed: recordings are now always finalized from LiveKit's own `egress_ended` webhook, which has been the default path since v1.22.0.
**Required for every deployment:** LiveKit must be able to deliver webhooks to the backend at `/api/v1.0/rooms/webhooks-livekit/`. This is now the only way a recording reaches a saved state; if `egress_ended` is never delivered, recordings stay in the `active` state.
For hosters who had configured storage-event webhooks:
- Recordings reach the same final state, but they are now finalized when LiveKit reports the egress as ended rather than when the storage backend reports the upload.
- Remove the event notification from your bucket configuration: it now targets a non-existent endpoint and will fail on every delivery.
For hosters who had **not** configured storage-event webhooks:
- Nothing changes. Recordings have been finalized from the `egress_ended` webhook since v1.22.0.
In both cases, the following settings are no longer used and can be removed from your env: `RECORDING_EVENT_PARSER_CLASS`, `RECORDING_ENABLE_STORAGE_EVENT_AUTH`, `RECORDING_STORAGE_EVENT_ENABLE`, `RECORDING_STORAGE_EVENT_TOKEN`.
On completion of the egress, a recording moves to `notification_succeeded`, or to `saved` if notifying external services failed.
## v1.23.0
As part of the 1.23.0 release, the legacy `api/v1` implementation has been removed from the _experimental_ Summary service and Meet has been migrated to the new `api/v2`.
+16
View File
@@ -46,6 +46,21 @@ services:
/usr/bin/mc mb meet/meet-media-storage && \
exit 0;"
createwebhook:
image: minio/mc
depends_on:
minio:
condition: service_healthy
restart: true
entrypoint: >
sh -c "
/usr/bin/mc alias set meet http://minio:9000 meet password &&
/usr/bin/mc admin config set meet notify_webhook:meet-webhook endpoint='http://app-dev:8000/api/v1.0/recordings/storage-hook/' auth_token='Bearer password' &&
/usr/bin/mc admin service restart meet --wait --json &&
sleep 15 &&
/usr/bin/mc event add meet/meet-media-storage arn:minio:sqs::meet-webhook:webhook --event put --prefix "recordings" &&
exit 0;"
app-dev:
build:
context: .
@@ -71,6 +86,7 @@ services:
- mailcatcher
- redis
- createbuckets
- createwebhook
extra_hosts:
- "127.0.0.1.nip.io:host-gateway"
networks:
+22 -2
View File
@@ -23,11 +23,14 @@ It uses LiveKit Egress to record room sessions. For reference, see the [LiveKit
To use the room recording feature, the following components are required:
- A running [LiveKit Egress](https://github.com/livekit/egress) server capable of handling room composite recordings.
- A S3-compatible object storage where the egress uploads the recorded files.
- A S3-compatible object storage that supports webhook events to notify the backend when recordings are uploaded.
- An email service to notify room owners when a recording is available for download.
- Webhook events configured between LiveKit Server and the backend.
> [!CAUTION]
> Minio supports lifecycle events; other providers may not work out of the box. There is currently a dependency on Minio, which is planned to be refactored in the future.
> [!NOTE]
> Celery isnt in use for these async tasks yet. Its something wed like to add, but its not planned at this stage.
@@ -72,7 +75,7 @@ sequenceDiagram
LiveKit->>Egress: Stop recording
Egress->>Storage: Upload recorded file
LiveKit->>Backend: POST /api/v1.0/rooms/webhooks-livekit/ (egress_ended)
Storage->>Backend: Storage event notification
Backend->>Backend: Update Recording status to SAVED
Backend->>Email: Send notification to room owner
@@ -91,6 +94,10 @@ sequenceDiagram
| **RECORDING_ENABLE** | Boolean | `False` | Enable or disable the room recording feature. |
| **RECORDING_OUTPUT_FOLDER** | String | `"recordings"` | Folder/prefix where recordings are stored in the object storage. |
| **RECORDING_WORKER_CLASSES** | Dict | `{ "screen_recording": "core.recording.worker.services.VideoCompositeEgressService", "transcript": "core.recording.worker.services.AudioCompositeEgressService" }` | Maps recording types to their worker service classes. |
| **RECORDING_EVENT_PARSER_CLASS** | String | `"core.recording.event.parsers.MinioParser"` | Class responsible for parsing storage events and updating the backend. |
| **RECORDING_ENABLE_STORAGE_EVENT_AUTH** | Boolean | `True` | Enable authentication for storage event webhook requests. |
| **RECORDING_STORAGE_EVENT_ENABLE** | Boolean | `False` | Enable handling of storage events (must configure webhook in storage). If `False`, fallback to LiveKit egress complete webhook. |
| **RECORDING_STORAGE_EVENT_TOKEN** | Secret/File | `None` | Token used to authenticate storage webhook requests, if `RECORDING_ENABLE_STORAGE_EVENT_AUTH` is enabled. |
| **RECORDING_EXPIRATION_DAYS** | Integer | `None` | Number of days before recordings expire. Should match bucket lifecycle policy. Set to `None` for no expiration. |
| **RECORDING_MAX_DURATION** | Integer | `None` | Maximum duration of a recording in milliseconds. Must be synced with the LiveKit Egress configuration. Set to None for unlimited duration. When the maximum duration is reached, the recording is automatically stopped and saved, and the user is prompted in the frontend with an alert message. |
| **RECORDING_ENCODING_ENABLED** | Boolean | `False` | When `False`, LiveKit Egress uses its built-in `H264_720P_30` preset. When `True`, the `RECORDING_ENCODING_*` values below are sent to LiveKit as advanced `EncodingOptions`. See [Tuning recording encoding](#tuning-recording-encoding). |
@@ -102,6 +109,19 @@ sequenceDiagram
| **RECORDING_ENCODING_KEY_FRAME_INTERVAL_S** | Float | `4.0` | Keyframe interval in seconds. Drives seek granularity in the recorded MP4 (a player can only seek to keyframe boundaries). Larger values give the encoder slightly more bits for non-keyframe content at a fixed bitrate. `4.0` is a standard VOD value. Only applied when `RECORDING_ENCODING_ENABLED` is `True`. |
### Manual Storage Webhook
Storage events must be configured manually; the Kubernetes chart does not do this automatically.
1. Configure your S3 bucket to send file creation events to the backend webhook.
2. Enable events and token in settings:
```python
RECORDING_STORAGE_EVENT_ENABLE = True
RECORDING_ENABLE_STORAGE_EVENT_AUTH = True
RECORDING_STORAGE_EVENT_TOKEN = <token>
```
> [!NOTE]
> Questions? Open an issue on [GitHub](https://github.com/suitenumerique/meet/issues/new?assignees=&labels=bug&template=Bug_report.md) or join our [Matrix community](https://matrix.to/#/#meet-official:matrix.org).
+4
View File
@@ -406,6 +406,10 @@ These are the environmental options available on meet backend.
| RECORDING_ENABLE | Record meeting option | false |
| RECORDING_OUTPUT_FOLDER | Folder to store meetings | recordings |
| RECORDING_WORKER_CLASSES | Worker classes for recording | {"screen_recording": "core.recording.worker.services.VideoCompositeEgressService","transcript": "core.recording.worker.services.AudioCompositeEgressService"} |
| RECORDING_EVENT_PARSER_CLASS | Storage event engine for recording | core.recording.event.parsers.MinioParser |
| RECORDING_ENABLE_STORAGE_EVENT_AUTH | Enable storage event authorization | true |
| RECORDING_STORAGE_EVENT_ENABLE | Enable recording storage events. If false, fallback to egress webhook. | false |
| RECORDING_STORAGE_EVENT_TOKEN | Recording storage event token | |
| RECORDING_EXPIRATION_DAYS | Recording expiration in days | |
| RECORDING_MAX_DURATION | Maximum recording duration in milliseconds. Must match LiveKit Egress configuration exactly. | |
| SCREEN_RECORDING_BASE_URL | Screen recording base URL | |
+2
View File
@@ -63,6 +63,8 @@ ALLOW_UNREGISTERED_ROOMS=False
# Recording
RECORDING_ENABLE=True
RECORDING_STORAGE_EVENT_ENABLE=False
RECORDING_STORAGE_EVENT_TOKEN=password
SUMMARY_SERVICE_ENDPOINT=http://app-summary-dev:8000/api/v2/async-jobs/transcribe/
SUMMARY_SERVICE_API_TOKEN=password
SUMMARY_SERVICE_WEBHOOK_API_TOKEN=webhook-password
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "agents"
version = "1.29.0"
version = "1.28.0"
requires-python = ">=3.12"
dependencies = [
"livekit-agents==1.6.7",
+1 -1
View File
@@ -9,7 +9,7 @@ resolution-markers = [
[[package]]
name = "agents"
version = "1.29.0"
version = "1.28.0"
source = { virtual = "." }
dependencies = [
{ name = "livekit-agents" },
+1
View File
@@ -11,6 +11,7 @@ class FeatureFlag:
FLAGS = {
"recording": "RECORDING_ENABLE",
"storage_event": "RECORDING_STORAGE_EVENT_ENABLE",
"subtitle": "ROOM_SUBTITLE_ENABLED",
"file_upload": "FILE_UPLOAD_ENABLED",
"addons": "ADDONS_ENABLED",
+65 -1
View File
@@ -45,11 +45,25 @@ from core.api import throttling
from core.api.filters import ListFileFilter
from core.enums import MEDIA_STORAGE_URL_PATTERN
from core.recording.enums import FileExtension
from core.recording.event.authentication import RecordingProcessWebhookAuthentication
from core.recording.event.authentication import (
RecordingProcessWebhookAuthentication,
StorageEventAuthentication,
)
from core.recording.event.exceptions import (
InvalidBucketError,
InvalidFilepathError,
InvalidFileTypeError,
ParsingEventDataError,
)
from core.recording.event.parsers import get_parser
from core.recording.services.metadata_collector import (
MetadataCollectorException,
MetadataCollectorService,
)
from core.recording.services.recording_events import (
RecordingEventsService,
RecordingNotSavableError,
)
from core.recording.worker.exceptions import (
RecordingStartError,
RecordingStopError,
@@ -1020,6 +1034,56 @@ class RecordingViewSet(
.filter(Q(accesses__user=user) | Q(accesses__team__in=user.get_teams()))
)
@decorators.action(
detail=False,
methods=["post"],
url_path="storage-hook",
authentication_classes=[StorageEventAuthentication],
)
@FeatureFlag.require("storage_event")
def on_storage_event_received(self, request, pk=None): # pylint: disable=unused-argument
"""Handle incoming storage hook events for recordings."""
parser = get_parser()
try:
recording_id = parser.get_recording_id(request.data)
except ParsingEventDataError as e:
raise drf_exceptions.PermissionDenied("Invalid request data.") from e
except InvalidBucketError as e:
raise drf_exceptions.PermissionDenied("Invalid bucket specified.") from e
except InvalidFilepathError:
return drf_response.Response(
{"message": "Notification ignored."},
)
except InvalidFileTypeError:
return drf_response.Response(
{"message": "Notification ignored."},
)
try:
recording = models.Recording.objects.get(id=recording_id)
except models.Recording.DoesNotExist as e:
raise drf_exceptions.NotFound("No recording found for this event.") from e
# Save recording
recording_events_service = RecordingEventsService()
try:
recording_events_service.handle_complete(recording)
except RecordingNotSavableError:
raise drf_exceptions.PermissionDenied(
f"Recording with ID {recording_id} cannot be saved because it is either,"
" in an error state or has already been saved."
) from None
return drf_response.Response(
{"message": "Event processed."},
)
@decorators.action(
detail=False,
methods=["post"],
@@ -1,4 +1,4 @@
"""Authentication classes for server-to-server webhook token validation."""
"""Authentication class for storage event token validation."""
import logging
import secrets
@@ -12,9 +12,9 @@ logger = logging.getLogger(__name__)
class MachineUser:
"""Represent a non-interactive system user for automated operations."""
"""Represent a non-interactive system user for automated storage operations."""
def __init__(self, username: str = "machine_user") -> None:
def __init__(self, username: str = "storage_event_user") -> None:
self.pk = None
self.username = username
self.is_active = True
@@ -41,17 +41,24 @@ class HeaderBasedAuthentication(BaseAuthentication):
TOKEN_TYPE = "Bearer" # noqa S105
REALM = ""
IS_ENFORCED_SETTINGS_KEY = None
EXPECTED_TOKEN_SETTINGS_KEY = None
def authenticate(self, request):
"""Validate the Bearer token from the Authorization header."""
if self.IS_ENFORCED_SETTINGS_KEY is not None:
if not getattr(settings, self.IS_ENFORCED_SETTINGS_KEY):
return MachineUser(), None
if (
self.EXPECTED_TOKEN_SETTINGS_KEY is None
or (required_token := getattr(settings, self.EXPECTED_TOKEN_SETTINGS_KEY))
is None
):
raise AuthenticationFailed("Authentication token is not configured.")
raise AuthenticationFailed(
"Authentication is enabled but token is not configured."
)
auth_header = request.headers.get(self.AUTH_HEADER)
if not auth_header:
@@ -81,6 +88,18 @@ class HeaderBasedAuthentication(BaseAuthentication):
return f"{self.TOKEN_TYPE} realm='{self.REALM}'"
class StorageEventAuthentication(HeaderBasedAuthentication):
"""Authenticate requests using a Bearer token for storage event integration.
This class validates Bearer tokens for storage events that don't map to database users.
It's designed for S3-compatible storage integrations and similar use cases.
Events are submitted when a webhook is configured on some bucket's events.
"""
REALM = "Storage event API"
IS_ENFORCED_SETTINGS_KEY = "RECORDING_ENABLE_STORAGE_EVENT_AUTH"
EXPECTED_TOKEN_SETTINGS_KEY = "RECORDING_STORAGE_EVENT_TOKEN" # noqa S105
class RecordingProcessWebhookAuthentication(HeaderBasedAuthentication):
"""
Custom authentication class for recording process webhook requests.
@@ -0,0 +1,17 @@
"""Storage parsers specific exceptions."""
class ParsingEventDataError(Exception):
"""Raised when the request data is malformed, incomplete, or missing."""
class InvalidBucketError(Exception):
"""Raised when the bucket name in the request does not match the expected one."""
class InvalidFileTypeError(Exception):
"""Raised when the file type in the request is not supported."""
class InvalidFilepathError(Exception):
"""Raised when the filepath in the request is invalid."""
+178
View File
@@ -0,0 +1,178 @@
"""Meet storage event parser classes."""
import logging
import mimetypes
import re
from dataclasses import dataclass
from functools import lru_cache
from typing import Any, Dict, Optional, Protocol
from urllib.parse import quote
from django.conf import settings
from django.utils.module_loading import import_string
from core.enums import FILE_EXT_REGEX, UUID_REGEX
from .exceptions import (
InvalidBucketError,
InvalidFilepathError,
InvalidFileTypeError,
ParsingEventDataError,
)
# Additional MIME type mapping
mimetypes.add_type("audio/ogg", ".ogg")
logger = logging.getLogger(__name__)
@dataclass
class StorageEvent:
"""Represents a storage event with relevant metadata.
Attributes:
filepath: Identifier for the affected recording
filetype: Type of storage event
bucket_name: When the event occurred
metadata: Additional event data
"""
filepath: str
filetype: str
bucket_name: str
metadata: Optional[Dict[str, Any]]
def __post_init__(self):
if self.filepath is None:
raise TypeError("filepath cannot be None")
if self.filetype is None:
raise TypeError("filetype cannot be None")
if self.bucket_name is None:
raise TypeError("bucket_name cannot be None")
class EventParser(Protocol):
"""Interface for parsing storage events."""
def __init__(self, bucket_name, allowed_filetypes=None):
"""Initialize parser with bucket name and optional allowed filetypes."""
def parse(self, data: Dict) -> StorageEvent:
"""Extract storage event data from raw dictionary input."""
def validate(self, data: StorageEvent) -> str:
"""Verify storage event data meets all requirements."""
def get_recording_id(self, data: Dict) -> str:
"""Extract recording ID from event dictionary."""
@lru_cache(maxsize=1)
def get_parser() -> EventParser:
"""Return cached instance of configured event parser.
Uses function memoization instead of a factory class since the only
varying parameter is the parser class from settings. A factory class
would add unnecessary complexity when a cached function provides the
same singleton behavior with simpler code.
"""
event_parser_cls = import_string(settings.RECORDING_EVENT_PARSER_CLASS)
return event_parser_cls(bucket_name=settings.AWS_STORAGE_BUCKET_NAME)
class BaseS3Parser:
"""Base class for handling parsing and validation of S3-compatible storage events."""
def __init__(self, bucket_name: str, allowed_filetypes=None):
"""Initialize parser with target bucket name and accepted filetypes."""
if not bucket_name:
raise ValueError("Bucket name cannot be None or empty")
self._bucket_name = bucket_name
self._allowed_filetypes = allowed_filetypes or {"audio/ogg", "video/mp4"}
# pylint: disable=line-too-long
self._filepath_regex = re.compile(
rf"(?P<url_encoded_folder_path>(?:[^%]+%2F)+)?{settings.RECORDING_OUTPUT_FOLDER}%2F(?P<recording_id>{UUID_REGEX})\.(?P<extension>{FILE_EXT_REGEX})"
)
def validate(self, event_data: StorageEvent) -> str:
"""Verify StorageEvent matches bucket, filetype and filepath requirements."""
if event_data.bucket_name != self._bucket_name:
raise InvalidBucketError(
f"Invalid bucket: expected {self._bucket_name}, got {event_data.bucket_name}"
)
if event_data.filetype not in self._allowed_filetypes:
raise InvalidFileTypeError(
f"Invalid file type, expected {self._allowed_filetypes},"
f"got '{event_data.filetype}'"
)
match = self._filepath_regex.match(event_data.filepath)
if not match:
raise InvalidFilepathError(
f"Invalid filepath structure: {event_data.filepath}"
)
recording_id = match.group("recording_id")
return recording_id
def get_recording_id(self, data):
"""Extract recording ID from S3 event through parsing and validation."""
event_data = self.parse(data)
return self.validate(event_data)
def parse(self, data: Dict) -> StorageEvent:
"""To be implemented by subclasses."""
raise NotImplementedError("Subclasses must implement parse()")
class MinioParser(BaseS3Parser):
"""Minio specific event parsing."""
def parse(self, data: Dict) -> StorageEvent:
if not data:
raise ParsingEventDataError("Received empty data.")
try:
record = data["Records"][0]
s3 = record["s3"]
return StorageEvent(
filepath=s3["object"]["key"],
filetype=s3["object"]["contentType"], # Minio-specific field
bucket_name=s3["bucket"]["name"],
metadata=None,
)
except (KeyError, IndexError) as e:
raise ParsingEventDataError(f"Malformed Minio event: {e}") from e
except TypeError as e:
raise ParsingEventDataError(f"Missing essential data fields: {e}") from e
class S3Parser(BaseS3Parser):
"""AWS S3 specific event parsing."""
def parse(self, data: Dict) -> StorageEvent:
if not data:
raise ParsingEventDataError("Received empty data.")
try:
# AWS S3 structure can slightly differ from Minio implementation
record = data["Records"][0]
s3 = record["s3"]
filepath = s3["object"]["key"]
if not filepath:
raise ParsingEventDataError("Missing object key name")
filetype, _ = mimetypes.guess_type(filepath)
# Normalize raw S3-compatible object keys without re-encoding
# already encoded AWS S3 notification keys.
filepath = quote(filepath, safe="%+")
return StorageEvent(
filepath=filepath,
filetype=filetype,
bucket_name=s3["bucket"]["name"],
metadata=None,
)
except (KeyError, IndexError) as e:
raise ParsingEventDataError(f"Malformed S3 event: {e}") from e
+4 -2
View File
@@ -220,8 +220,10 @@ class LiveKitEventsService:
f"Failed to process limit reached event for recording {recording}"
) from e
# Finalize the recording, the egress has uploaded the file to the storage
if data.egress_info.status in [
# Fallback for completion when no MinIO/S3 webhooks are configured
if (
not settings.RECORDING_STORAGE_EVENT_ENABLE
) and data.egress_info.status in [
api.EgressStatus.EGRESS_COMPLETE,
api.EgressStatus.EGRESS_LIMIT_REACHED,
]:
@@ -11,98 +11,135 @@ from rest_framework.exceptions import AuthenticationFailed
from core.recording.event.authentication import (
MachineUser,
RecordingProcessWebhookAuthentication,
StorageEventAuthentication,
)
def test_successful_authentication(settings):
"""Test successful authentication with valid token."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "Bearer valid-test-token"}
user, token = RecordingProcessWebhookAuthentication().authenticate(request)
user, token = StorageEventAuthentication().authenticate(request)
assert token == "valid-test-token"
assert isinstance(user, MachineUser)
def test_authentication_fails_when_token_not_configured(settings):
"""Authentication should fail when no token is configured."""
def test_disabled_authentication_with_header(settings):
"""Authentication should pass when no auth is configured, and header is present."""
settings.RECORDING_STORAGE_EVENT_TOKEN = None
settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = None
request = RequestFactory().get("/")
request.headers = {"Authorization": "Bearer some-token"}
user, token = StorageEventAuthentication().authenticate(request)
assert token is None
assert isinstance(user, MachineUser)
def test_disabled_authentication_without_header(settings):
"""Authentication should pass when no auth is configured, and no header is present."""
settings.RECORDING_STORAGE_EVENT_TOKEN = None
settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False
request = RequestFactory().get("/")
user, token = StorageEventAuthentication().authenticate(request)
assert token is None
assert isinstance(user, MachineUser)
def test_authentication_when_disabled(settings):
"""Authentication should pass when disabled, regardless of token configuration."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "some-token"
settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False
request = RequestFactory().get("/")
user, token = StorageEventAuthentication().authenticate(request)
assert token is None
assert isinstance(user, MachineUser)
def test_authentication_fails_when_token_not_configured(settings):
"""Authentication should fail when authentication is enabled but no token is configured."""
# By default RECORDING_ENABLE_STORAGE_EVENT_AUTH should be True
settings.RECORDING_STORAGE_EVENT_TOKEN = None
request = RequestFactory().get("/")
with pytest.raises(
AuthenticationFailed,
match="Authentication token is not configured",
match="Authentication is enabled but token is not configured",
):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_missing_auth_header(settings):
"""Test failure when Authorization header is missing."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {}
with pytest.raises(AuthenticationFailed, match="Authorization header is required"):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_invalid_auth_header_format(settings):
"""Test failure when Authorization header has invalid format."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "InvalidFormat"}
with pytest.raises(AuthenticationFailed, match="Invalid authorization header"):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_invalid_token_type(settings):
"""Test failure when token type is not Bearer."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "Basic some-token"}
with pytest.raises(AuthenticationFailed, match="Invalid authorization header"):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_invalid_token(settings):
"""Test failure when token is invalid."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "Bearer wrong-token"}
with pytest.raises(AuthenticationFailed, match="Invalid token"):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_malformed_auth_header(settings):
"""Test failure when Authorization header is malformed."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token"
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "Bearer"} # Missing token part
with pytest.raises(AuthenticationFailed, match="Invalid authorization header"):
RecordingProcessWebhookAuthentication().authenticate(request)
StorageEventAuthentication().authenticate(request)
def test_authenticate_header():
"""Test the WWW-Authenticate header value."""
request = RequestFactory().get("/")
header = RecordingProcessWebhookAuthentication().authenticate_header(request)
assert header == "Bearer realm='External process webhook API'"
header = StorageEventAuthentication().authenticate_header(request)
assert header == "Bearer realm='Storage event API'"
def test_multiple_spaces_in_auth_header(settings):
"""Extra spaces between the scheme and the token should be tolerated."""
settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "extra-spaces-token"
"""Test success when Authorization header contains multiple spaces."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token"
request = RequestFactory().get("/")
request.headers = {"Authorization": "Bearer extra-spaces-token"}
user, token = RecordingProcessWebhookAuthentication().authenticate(request)
assert token == "extra-spaces-token"
assert isinstance(user, MachineUser)
header = StorageEventAuthentication().authenticate_header(request)
assert header == "Bearer realm='Storage event API'"
@@ -0,0 +1,512 @@
"""
Test event parsers.
"""
# pylint: disable=protected-access,redefined-outer-name,unused-argument
from unittest import mock
from django.conf import settings
import pytest
from core.recording.event.exceptions import (
InvalidBucketError,
InvalidFilepathError,
InvalidFileTypeError,
ParsingEventDataError,
)
from core.recording.event.parsers import (
MinioParser,
S3Parser,
StorageEvent,
get_parser,
)
# MinioParser
@pytest.fixture
def valid_minio_event():
"""Mock a valid Minio event."""
return {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
"contentType": "audio/ogg",
},
}
}
]
}
@pytest.fixture
def minio_parser():
"""Mock a Minio parser."""
return MinioParser(bucket_name="test-bucket")
def test_minio_parse_valid_event(minio_parser, valid_minio_event):
"""Test parsing a valid Minio event."""
event = minio_parser.parse(valid_minio_event)
assert isinstance(event, StorageEvent)
assert event.filepath == "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg"
assert event.filetype == "audio/ogg"
assert event.bucket_name == "test-bucket"
assert event.metadata is None
def test_minio_parse_with_video_type(minio_parser):
"""Test parsing event with video file type."""
video_event = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "46d1a121-2426-484d-8fb3-09b5d886f7a8.mp4",
"contentType": "video/mp4",
},
}
}
]
}
event = minio_parser.parse(video_event)
assert event.filetype == "video/mp4"
assert event.filepath.endswith(".mp4")
def test_minio_parse_empty_data(minio_parser):
"""Test parsing empty event data raises error."""
with pytest.raises(ParsingEventDataError, match="Received empty data."):
minio_parser.parse({})
def test_minio_parse_missing_keys(minio_parser):
"""Test parsing event with missing key."""
invalid_minio_event = {
"Records": [
{
"s3": {
"bucket": {"name": None},
# Missing 'object' key
}
}
]
}
with pytest.raises(ParsingEventDataError, match="Malformed Minio event:"):
minio_parser.parse(invalid_minio_event)
def test_minio_parse_none_key(minio_parser):
"""Test parsing event with None field."""
invalid_minio_event = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
"contentType": None, # 'contentType' should not be None
},
}
}
]
}
with pytest.raises(ParsingEventDataError, match="Missing essential data fields"):
minio_parser.parse(invalid_minio_event)
def test_minio_validate_invalid_bucket(minio_parser):
"""Test validation with wrong bucket name."""
event = StorageEvent(
filepath="recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
filetype="audio/ogg",
bucket_name="wrong-bucket",
metadata=None,
)
with pytest.raises(InvalidBucketError):
minio_parser.validate(event)
def test_minio_validate_invalid_filetype(minio_parser):
"""Test validation with unsupported file type."""
event = StorageEvent(
filepath="recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.txt",
filetype="text/plain", # Not included in the default allowed filetypes
bucket_name="test-bucket",
metadata=None,
)
with pytest.raises(InvalidFileTypeError):
minio_parser.validate(event)
@pytest.mark.parametrize(
"invalid_filepath",
[
"invalid_filepath", # totally invalid string
"recordings/46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
"recordings/46d1a121-2426-484d-8fb3-09b5d886f7a8", # missing extension
"46d1a121-2426-484d-8fb3-09b5d886f7a8", # missing url_encoded_folder_path and extension
"", # empty string
"46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # no folder at all
"uploads%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # wrong folder name
"folder%2Fuploads%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # nested but no recordings/
],
)
def test_minio_validate_invalid_filepath(invalid_filepath, minio_parser):
"""Test validation with malformed filepath."""
event = StorageEvent(
filepath=invalid_filepath,
filetype="audio/ogg",
bucket_name="test-bucket",
metadata=None,
)
with pytest.raises(InvalidFilepathError):
minio_parser.validate(event)
def test_minio_validate_valid_event(minio_parser):
"""Test validation with valid event data."""
event = StorageEvent(
filepath="recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
filetype="audio/ogg",
bucket_name="test-bucket",
metadata=None,
)
recording_id = minio_parser.validate(event)
assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8"
def test_minio_get_recording_id_success(minio_parser, valid_minio_event):
"""Test successful extraction of recording ID."""
recording_id = minio_parser.get_recording_id(valid_minio_event)
assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8"
def test_minio_validate_filepath_with_folder(minio_parser):
"""Test validation of filepath with folder structure."""
event = StorageEvent(
filepath="parent_folder%2Frecordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
filetype="audio/ogg",
bucket_name="test-bucket",
metadata=None,
)
recording_id = minio_parser.validate(event)
assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8"
def test_minio_empty_allowed_filetypes():
"""Test MinioParser with empty allowed_filetypes."""
empty_types = set()
parser = MinioParser(bucket_name="test-bucket", allowed_filetypes=empty_types)
assert parser._allowed_filetypes == {"audio/ogg", "video/mp4"}
def test_minio_custom_allowed_filetypes():
"""Test MinioParser with empty allowed_filetypes."""
custom_types = {"audio/mp3", "video/mov"}
parser = MinioParser(bucket_name="test-bucket", allowed_filetypes=custom_types)
assert parser._allowed_filetypes == {"audio/mp3", "video/mov"}
def test_minio_validate_custom_filetypes():
"""Test validation of filepath with folder structure."""
parser = MinioParser(bucket_name="test-bucket", allowed_filetypes={"audio/mp3"})
event = StorageEvent(
filepath="parent_folder%2Frecordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
filetype="audio/mp3",
bucket_name="test-bucket",
metadata=None,
)
parser.validate(event)
def test_minio_constructor_none_bucket():
"""Test MinioParser constructor with None bucket name."""
with pytest.raises(ValueError, match="Bucket name cannot be None or empty"):
MinioParser(bucket_name=None)
def test_minio_constructor_empty_bucket():
"""Test MinioParser constructor with empty bucket name."""
with pytest.raises(ValueError, match="Bucket name cannot be None or empty"):
MinioParser(bucket_name="")
# S3Parser
@pytest.fixture
def valid_s3_event():
"""Mock a valid S3 event."""
return {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg",
},
}
}
]
}
@pytest.fixture
def s3_parser():
"""Mock an S3 parser."""
return S3Parser(bucket_name="test-bucket")
def test_s3_parse_valid_event(s3_parser, valid_s3_event):
"""Test parsing a valid S3 event."""
event = s3_parser.parse(valid_s3_event)
assert isinstance(event, StorageEvent)
assert event.filepath == "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg"
assert event.filetype == "audio/ogg"
assert event.bucket_name == "test-bucket"
assert event.metadata is None
def test_s3_parse_empty_data(s3_parser):
"""Test parsing empty S3 event data raises error."""
with pytest.raises(ParsingEventDataError, match="Received empty data."):
s3_parser.parse({})
def test_s3_parse_missing_keys(s3_parser):
"""Test parsing S3 event with missing key."""
invalid_s3_event = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
# Missing 'object' key
}
}
]
}
with pytest.raises(ParsingEventDataError, match="Malformed S3 event:"):
s3_parser.parse(invalid_s3_event)
def test_s3_parse_none_key(s3_parser):
"""Test parsing S3 event with None field."""
invalid_s3_event = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": None,
},
}
}
]
}
with pytest.raises(ParsingEventDataError, match="Missing object key name"):
s3_parser.parse(invalid_s3_event)
def test_s3_parse_with_video_type(s3_parser):
"""Test parsing S3 event with mp4 file extension."""
video_event = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.mp4",
},
}
}
]
}
event = s3_parser.parse(video_event)
assert event.filetype == "video/mp4"
assert event.filepath.endswith(".mp4")
def test_s3_parse_unrecognized_extension(s3_parser):
"""Test parsing S3 event with unrecognized file extension."""
event_with_unknown_ext = {
"Records": [
{
"s3": {
"bucket": {"name": "test-bucket"},
"object": {
"key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.zzunknown999",
},
}
}
]
}
with pytest.raises(TypeError, match="filetype cannot be None"):
s3_parser.parse(event_with_unknown_ext)
def test_s3_parser_keeps_encoded_filepath_compatible(settings):
"""Test S3 parser keeps already encoded object keys compatible."""
settings.RECORDING_OUTPUT_FOLDER = "recordings"
recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d"
parser = S3Parser(bucket_name="recordings-bucket")
data = {
"Records": [
{
"s3": {
"bucket": {"name": "recordings-bucket"},
"object": {
"key": f"recordings%2F{recording_id}.mp4",
},
}
}
]
}
assert parser.get_recording_id(data) == recording_id
def test_s3_parser_accepts_unencoded_filepath(settings):
"""Test S3 parser accepts raw object keys with slash separators."""
settings.RECORDING_OUTPUT_FOLDER = "recordings"
recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d"
parser = S3Parser(bucket_name="recordings-bucket")
data = {
"Records": [
{
"s3": {
"bucket": {"name": "recordings-bucket"},
"object": {
"key": f"recordings/{recording_id}.mp4",
},
}
}
]
}
assert parser.get_recording_id(data) == recording_id
def test_s3_parser_preserves_plus_signs_in_encoded_filepath(settings):
"""Test S3 parser preserves plus signs in already encoded object keys."""
settings.RECORDING_OUTPUT_FOLDER = "recordings"
recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d"
parser = S3Parser(bucket_name="recordings-bucket")
data = {
"Records": [
{
"s3": {
"bucket": {"name": "recordings-bucket"},
"object": {
"key": f"folder+name%2Frecordings%2F{recording_id}.mp4",
},
}
}
]
}
assert parser.get_recording_id(data) == recording_id
def test_s3_get_recording_id_success(s3_parser, valid_s3_event):
"""Test successful extraction of recording ID from S3 event."""
recording_id = s3_parser.get_recording_id(valid_s3_event)
assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8"
# get_parser
@pytest.fixture
def clear_lru_cache():
"""Fixture to clear the LRU cache between tests."""
get_parser.cache_clear()
yield
get_parser.cache_clear()
def test_returns_correct_instance(clear_lru_cache):
"""Test if get_parser returns the correct parser instance."""
settings.AWS_STORAGE_BUCKET_NAME = "test-bucket"
parser = get_parser()
assert isinstance(parser, MinioParser)
assert parser._bucket_name == "test-bucket"
def test_caching_behavior(clear_lru_cache):
"""Test if the function properly caches the parser instance."""
settings.AWS_STORAGE_BUCKET_NAME = "test-bucket"
parser1 = get_parser()
parser2 = get_parser()
assert parser1 is parser2 # Check object identity
def test_different_settings_new_instance():
"""Test if changing settings creates a new instance."""
settings.AWS_STORAGE_BUCKET_NAME = "different-bucket"
parser = get_parser()
assert parser._bucket_name == "different-bucket"
def test_import_error_handling(clear_lru_cache):
"""Test handling of import errors for invalid parser class."""
settings.RECORDING_EVENT_PARSER_CLASS = "invalid.parser.path"
with pytest.raises(ImportError):
get_parser()
@mock.patch("core.recording.event.parsers.import_string")
def test_parser_instantiation_called_once(mock_import_string, clear_lru_cache):
"""Test that parser class is instantiated only once due to caching."""
mock_parser_cls = type(
"MockParser",
(),
{
"__init__": lambda self, bucket_name: setattr(
self, "_bucket_name", bucket_name
)
},
)
mock_import_string.return_value = mock_parser_cls
# First call
parser1 = get_parser()
# Second call
parser2 = get_parser()
# Verify import_string was called only once
mock_import_string.assert_called_once_with(settings.RECORDING_EVENT_PARSER_CLASS)
assert parser1 is parser2
def test_cache_clear_behavior(clear_lru_cache, settings):
"""Test that cache clearing creates new instance."""
settings.RECORDING_EVENT_PARSER_CLASS = "core.recording.event.parsers.MinioParser"
parser1 = get_parser()
get_parser.cache_clear()
parser2 = get_parser()
assert parser1 is not parser2 # Should be different instances after cache clear
@@ -12,7 +12,6 @@ from core.factories import RecordingFactory
from core.recording.services.recording_events import (
RecordingEventsError,
RecordingEventsService,
RecordingNotSavableError,
)
from core.utils import NotificationError
@@ -71,56 +70,3 @@ def test_handle_limit_reached_error(mock_notify, mode, notification_type, servic
mock_notify.assert_called_once_with(
room_name=str(recording.room.id), notification_data={"type": notification_type}
)
@pytest.mark.parametrize("status", ["active", "stopped"])
@pytest.mark.parametrize(
("notify_return_value", "expected_status"),
((True, "notification_succeeded"), (False, "saved")),
)
@mock.patch(
"core.recording.services.recording_events.notification_service."
"notify_external_services"
)
def test_handle_complete_saves_recording( # pylint: disable=too-many-arguments, too-many-positional-arguments
mock_notify_external_services,
notify_return_value,
expected_status,
status,
service,
):
"""Test handle_complete notifies external services and saves a savable recording."""
mock_notify_external_services.return_value = notify_return_value
recording = RecordingFactory(status=status)
service.handle_complete(recording)
mock_notify_external_services.assert_called_once_with(recording)
recording.refresh_from_db()
assert recording.status == expected_status
@pytest.mark.parametrize(
"status",
["initiated", "saved", "notification_succeeded", "aborted", "failed_to_start"],
)
@mock.patch(
"core.recording.services.recording_events.notification_service."
"notify_external_services"
)
def test_handle_complete_non_savable_recording(
mock_notify_external_services, status, service
):
"""Test handle_complete refuses recordings that are already saved or in error."""
recording = RecordingFactory(status=status)
with pytest.raises(RecordingNotSavableError):
service.handle_complete(recording)
mock_notify_external_services.assert_not_called()
recording.refresh_from_db()
assert recording.status == status
@@ -0,0 +1,267 @@
"""
Test recordings API endpoints in the Meet core app: save recording.
"""
# pylint: disable=redefined-outer-name,unused-argument
import uuid
from unittest import mock
import pytest
from rest_framework.test import APIClient
from ...factories import RecordingFactory
from ...models import Recording, RecordingStatusChoices
from ...recording.event.exceptions import (
InvalidBucketError,
InvalidFilepathError,
InvalidFileTypeError,
ParsingEventDataError,
)
pytestmark = pytest.mark.django_db
@pytest.fixture
def recording_settings(settings):
"""Configure recording-related and storage event Django settings."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken"
settings.RECORDING_STORAGE_EVENT_ENABLE = True
return settings
@pytest.fixture
def mock_get_parser():
"""Mock 'get_parser' factory function."""
with mock.patch("core.api.viewsets.get_parser") as mock_parser:
yield mock_parser
def test_save_recording_anonymous(settings, client):
"""Anonymous users should not be allowed to save room recordings."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken"
RecordingFactory(status="active")
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
)
assert response.status_code == 401
assert Recording.objects.count() == 1
def test_save_recording_wrong_bearer(settings, client):
"""Requests with incorrect bearer token should be rejected when auth is required."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken"
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer wrongAuthToken",
)
assert response.status_code == 401
def test_save_recording_permission_needed(settings, client):
"""Recordings should not be saved when feature is disabled."""
settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken"
settings.RECORDING_STORAGE_EVENT_ENABLE = False
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 404
assert response.json() == {"detail": "Not found."}
def test_save_recording_parsing_error(recording_settings, mock_get_parser, client):
"""Test handling of parsing errors in recording event data."""
mock_parser = mock.Mock()
mock_parser.get_recording_id.side_effect = ParsingEventDataError("Error message")
mock_get_parser.return_value = mock_parser
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 403
assert response.json() == {"detail": "Invalid request data."}
def test_save_recording_bucket_error(recording_settings, mock_get_parser, client):
"""Test handling of invalid storage bucket errors in recording event data."""
mock_parser = mock.Mock()
mock_parser.get_recording_id.side_effect = InvalidBucketError("Error message")
mock_get_parser.return_value = mock_parser
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 403
assert response.json() == {"detail": "Invalid bucket specified."}
def test_save_recording_filetype_error(recording_settings, mock_get_parser):
"""Test handling of unsupported file types in recording event data."""
mock_parser = mock.Mock()
mock_parser.get_recording_id.side_effect = InvalidFileTypeError(
"unsupported '.json'"
)
mock_get_parser.return_value = mock_parser
client = APIClient()
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 200
assert response.json() == {"message": "Notification ignored."}
def test_save_recording_filepath_error(recording_settings, mock_get_parser):
"""Test handling of unsupported filepath in recording event data."""
mock_parser = mock.Mock()
mock_parser.get_recording_id.side_effect = InvalidFilepathError(
"Invalid filepath structure: parent/folder/recording.jpeg"
)
mock_get_parser.return_value = mock_parser
client = APIClient()
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 200
assert response.json() == {"message": "Notification ignored."}
def test_save_recording_unknown_recording(recording_settings, mock_get_parser, client):
"""Test handling of events for non-existent recordings."""
RecordingFactory(status="active")
mock_parser = mock.Mock()
mock_parser.get_recording_id.return_value = uuid.uuid4()
mock_get_parser.return_value = mock_parser
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 404
assert response.json() == {"detail": "No recording found for this event."}
@pytest.mark.parametrize(
"status", ["failed_to_start", "aborted", "failed_to_stop", "saved", "initiated"]
)
def test_save_recording_non_savable_recording(
recording_settings, mock_get_parser, client, status
):
"""Test that recordings in non-savable states cannot be saved."""
recording = RecordingFactory(status=status)
mock_parser = mock.Mock()
mock_parser.get_recording_id.return_value = recording.id
mock_get_parser.return_value = mock_parser
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 403
assert response.json() == {
"detail": f"Recording with ID {recording.id} cannot be saved because it is either,"
" in an error state or has already been saved."
}
@pytest.mark.parametrize("status", ["active", "stopped"])
def test_save_recording_success(recording_settings, mock_get_parser, client, status):
"""Test successful saving of recordings in valid states."""
recording = RecordingFactory(status=status)
mock_parser = mock.Mock()
mock_parser.get_recording_id.return_value = recording.id
mock_get_parser.return_value = mock_parser
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 200
assert response.json() == {"message": "Event processed."}
recording.refresh_from_db()
assert recording.status == RecordingStatusChoices.SAVED
@mock.patch(
"core.recording.services.recording_events.notification_service."
"notify_external_services"
)
@pytest.mark.parametrize("notification_succeeded", [True, False])
def test_save_recording_notifies_external_services(
mock_notify_external_services,
recording_settings,
mock_get_parser,
client,
notification_succeeded,
):
"""External services should be notified when a recording is saved."""
recording = RecordingFactory(status="active")
mock_parser = mock.Mock()
mock_parser.get_recording_id.return_value = recording.id
mock_get_parser.return_value = mock_parser
mock_notify_external_services.return_value = notification_succeeded
response = client.post(
"/api/v1.0/recordings/storage-hook/",
{"recording_data": "valid-data"},
HTTP_AUTHORIZATION="Bearer testAuthToken",
)
assert response.status_code == 200
assert response.json() == {"message": "Event processed."}
mock_notify_external_services.assert_called_once_with(recording)
recording.refresh_from_db()
assert recording.status == (
RecordingStatusChoices.NOTIFICATION_SUCCEEDED
if notification_succeeded
else RecordingStatusChoices.SAVED
)
@@ -75,11 +75,12 @@ def test_initialization(
)
@mock.patch("core.utils.notify_participants")
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_handle_egress_ended_success( # pylint: disable=too-many-arguments, too-many-positional-arguments
mock_update_metadata, mock_notify, mode, notification_type, service
def test_handle_egress_ended_success( # noqa: PLR0913, PLR0917 # pylint: disable=too-many-arguments, too-many-positional-arguments
mock_update_metadata, mock_notify, mode, notification_type, service, settings
):
"""Should successfully stop recording and notifies all participant."""
settings.RECORDING_STORAGE_EVENT_ENABLE = False
recording = RecordingFactory(worker_id="worker-1", mode=mode, status="active")
mock_data = mock.MagicMock()
mock_data.egress_info.egress_id = recording.worker_id
@@ -158,11 +159,12 @@ def test_handle_egress_updated_non_handled(
)
@mock.patch("core.utils.notify_participants")
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_handle_egress_ended_metadata_update_fails( # pylint: disable=too-many-arguments, too-many-positional-arguments
mock_update_metadata, mock_notify, mode, notification_type, service
def test_handle_egress_ended_metadata_update_fails( # noqa: PLR0913, PLR0917 # pylint: disable=too-many-arguments, too-many-positional-arguments
mock_update_metadata, mock_notify, mode, notification_type, service, settings
):
"""Should successfully stop and save recording when metadata's update fails."""
settings.RECORDING_STORAGE_EVENT_ENABLE = False
recording = RecordingFactory(worker_id="worker-1", mode=mode, status="active")
mock_data = mock.MagicMock()
mock_data.egress_info.egress_id = recording.worker_id
@@ -356,10 +358,12 @@ def test_handle_egress_ended_finalizes_recording( # noqa: PLR0913, PLR0917
recording_status,
egress_status,
service,
settings,
): # pylint: disable=too-many-arguments,too-many-positional-arguments
"""Should notify external services and save the recording on egress completion
(EGRESS_COMPLETE or EGRESS_LIMIT_REACHED).
(EGRESS_COMPLETE or EGRESS_LIMIT_REACHED) when RECORDING_STORAGE_EVENT_ENABLE is False.
"""
settings.RECORDING_STORAGE_EVENT_ENABLE = False
mock_notify_external_services.return_value = notify_return_value
recording = RecordingFactory(worker_id="worker-1", status="active")
@@ -375,6 +379,47 @@ def test_handle_egress_ended_finalizes_recording( # noqa: PLR0913, PLR0917
assert recording.status == recording_status
@mock.patch(
"core.recording.services.recording_events.notification_service."
"notify_external_services"
)
@mock.patch("core.utils.notify_participants")
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
@pytest.mark.parametrize(
"egress_status, expected_status",
[
(EgressStatus.EGRESS_COMPLETE, "active"),
(EgressStatus.EGRESS_LIMIT_REACHED, "stopped"),
],
)
def test_handle_egress_ended_does_not_finalize_when_webhooks_enabled( # noqa: PLR0913, PLR0917
mock_update_metadata,
mock_notify,
mock_notify_external_services,
egress_status,
expected_status,
service,
settings,
): # pylint: disable=too-many-arguments,too-many-positional-arguments
"""When storage event webhooks are enabled, egress_ended must not finalize the
recording: external services are never notified. EGRESS_LIMIT_REACHED still stops
the recording, EGRESS_COMPLETE leaves it active.
"""
settings.RECORDING_STORAGE_EVENT_ENABLE = True
recording = RecordingFactory(worker_id="worker-1", status="active")
mock_data = mock.MagicMock()
mock_data.egress_info.egress_id = recording.worker_id
mock_data.egress_info.status = egress_status
service._handle_egress_ended(mock_data)
mock_notify_external_services.assert_not_called()
recording.refresh_from_db()
assert recording.status == expected_status
@pytest.mark.parametrize(
"egress_status",
[
@@ -387,9 +432,10 @@ def test_handle_egress_ended_finalizes_recording( # noqa: PLR0913, PLR0917
)
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_handle_egress_ended_does_not_save_on_wrong_status(
mock_update_metadata, egress_status, service
mock_update_metadata, egress_status, service, settings
):
"""Shouldn't save on invalid status."""
settings.RECORDING_STORAGE_EVENT_ENABLE = False
recording = RecordingFactory(worker_id="worker-1", status="active")
mock_data = mock.MagicMock()
@@ -407,13 +453,14 @@ def test_handle_egress_ended_does_not_save_on_wrong_status(
)
@mock.patch("core.services.room_management.RoomManagement.update_metadata")
def test_handle_egress_ended_ignores_non_savable_recording(
mock_update_metadata, status, service
mock_update_metadata, status, service, settings
):
"""Should handle non-savable recordings idempotently without raising.
'egress_ended' may be redelivered (e.g. for an already-saved recording);
this must not raise, otherwise the webhook would 500 and LiveKit would retry.
"""
settings.RECORDING_STORAGE_EVENT_ENABLE = False
recording = RecordingFactory(worker_id="worker-1", status=status)
mock_data = mock.MagicMock()
+14
View File
@@ -729,6 +729,20 @@ class Base(Configuration):
environ_name="RECORDING_WORKER_CLASSES",
environ_prefix=None,
)
RECORDING_EVENT_PARSER_CLASS = values.Value(
"core.recording.event.parsers.MinioParser",
environ_name="RECORDING_EVENT_PARSER_CLASS",
environ_prefix=None,
)
RECORDING_ENABLE_STORAGE_EVENT_AUTH = values.BooleanValue(
True, environ_name="RECORDING_ENABLE_STORAGE_EVENT_AUTH", environ_prefix=None
)
RECORDING_STORAGE_EVENT_ENABLE = values.BooleanValue(
False, environ_name="RECORDING_STORAGE_EVENT_ENABLE", environ_prefix=None
)
RECORDING_STORAGE_EVENT_TOKEN = SecretFileValue(
None, environ_name="RECORDING_STORAGE_EVENT_TOKEN", environ_prefix=None
)
# Number of days before recordings expire - must be synced with bucket lifecycle policy
# Set to None for no expiration
RECORDING_EXPIRATION_DAYS = values.IntegerValue(
+1 -1
View File
@@ -7,7 +7,7 @@ build-backend = "uv_build"
[project]
name = "meet"
version = "1.29.0"
version = "1.28.0"
authors = [{ "name" = "DINUM", "email" = "dev@mail.numerique.gouv.fr" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
+1 -1
View File
@@ -1187,7 +1187,7 @@ wheels = [
[[package]]
name = "meet"
version = "1.29.0"
version = "1.28.0"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "meet",
"version": "1.29.0",
"version": "1.28.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "meet",
"version": "1.29.0",
"version": "1.28.0",
"dependencies": {
"@fontsource-variable/atkinson-hyperlegible-next": "5.3.0",
"@fontsource-variable/lexend": "5.2.11",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "meet",
"private": true,
"version": "1.29.0",
"version": "1.28.0",
"type": "module",
"scripts": {
"dev": "panda codegen && vite",
@@ -42,6 +42,23 @@ export type IceCandidateReport = {
working: IceCandidatePair[]
}
/**
* True when the selected pair does not carry media over plain UDP: the browser
* fell back to a TURN relay over TCP or TLS, or even the direct route is not
* UDP. Media still flows, but quality usually degrades under load.
*
* Accepts the loosely typed `data` stored on the step result; anything that is
* not an IceCandidateReport simply yields false.
*/
export const isSuboptimalRoute = (data: unknown): boolean => {
const selected = (data as IceCandidateReport | null | undefined)?.selected
if (!selected) return false
const transport = selected.local.relayProtocol ?? selected.local.protocol
// An unreported transport is not evidence of a bad route.
if (!transport) return false
return transport.toLowerCase() !== 'udp'
}
const PROBE_WIDTH = 320
const PROBE_HEIGHT = 180
const PROBE_FPS = 15
@@ -2,18 +2,33 @@ import type { ReactNode } from 'react'
import { useTranslation } from 'react-i18next'
import { ProgressBar } from 'react-aria-components'
import { css, cx } from '@/styled-system/css'
import { A } from '@/primitives'
import type { ConnectionTestStats } from '../types'
import { statusSquareClass } from './stepAppearance'
type SummaryState = 'idle' | 'running' | 'passed' | 'partial' | 'failed'
/**
* Network prerequisites (protocols, ports, TURN fallbacks), written for the
* reader's IT department rather than for the end user.
*/
const NETWORK_REQUIREMENTS_DOC_URL =
'https://docs.numerique.gouv.fr/docs/f2baa1b9-f29e-4d58-959d-65d4376fc6b8/'
/** Only a failure earns a colour: everything else stays near-black. */
type SummaryState =
| 'idle'
| 'running'
| 'passed'
| 'partial'
| 'failed'
| 'warning'
/** Only a failure or a degraded route earns a colour: everything else stays near-black. */
const stateColorClass: Record<SummaryState, string> = {
idle: css({ color: 'greyscale.1000' }),
running: css({ color: 'greyscale.1000' }),
passed: css({ color: 'greyscale.1000' }),
partial: css({ color: 'greyscale.1000' }),
failed: css({ color: 'danger.600' }),
warning: css({ color: 'warning' }),
}
const cardClass = css({
@@ -176,23 +191,31 @@ const Counter = ({
export const ConnectionTestSummary = ({
stats,
isRunning,
routeWarning = false,
children,
}: {
stats: ConnectionTestStats
isRunning: boolean
/** The selected ICE route is usable but not UDP (e.g. TURN over TCP/TLS). */
routeWarning?: boolean
children?: ReactNode
}) => {
const { t } = useTranslation('connectionTest')
// A hard failure still outranks the route warning; the warning outranks
// 'partial' because a measured degraded route matters more than skipped
// camera or microphone checks.
const state: SummaryState = isRunning
? 'running'
: !stats.hasStarted
? 'idle'
: stats.failed > 0
? 'failed'
: stats.skipped > 0
? 'partial'
: 'passed'
: routeWarning
? 'warning'
: stats.skipped > 0
? 'partial'
: 'passed'
return (
<section className={cardClass}>
@@ -206,7 +229,22 @@ export const ConnectionTestSummary = ({
: t(`summary.${state}`)}
</p>
<p className={hintClass}>{t(`summary.${state}Hint`)}</p>
<p className={hintClass}>
{t(`summary.${state}Hint`)}
{state === 'warning' && (
<>
{' '}
<A
href={NETWORK_REQUIREMENTS_DOC_URL}
target="_blank"
rel="noreferrer"
size="sm"
>
{t('summary.warningDocLink')}
</A>
</>
)}
</p>
</div>
{stats.hasStarted && (
@@ -53,6 +53,9 @@ const fromCheckInfo = (info: CheckInfo): Partial<ConnectionTestStepResult> => ({
status: CHECK_STATUS_TO_STEP[info.status] ?? 'failed',
summary: info.description,
logs: info.logs,
// Only SelectedCandidateCheck sets `data` (the ICE candidate report); for
// the stock LiveKit checks it is undefined and this is a no-op.
data: info.data as Record<string, unknown> | undefined,
})
const groupDevicesByKind = (devices: MediaDeviceInfo[]) => {
@@ -13,6 +13,7 @@ import { css } from '@/styled-system/css'
import { Center, VStack } from '@/styled-system/jsx'
import { Permissions } from '@/features/rooms/components/Permissions'
import { useConnectionTestRunner } from '../hooks/useConnectionTestRunner'
import { isSuboptimalRoute } from '../checks/selectedCandidate'
import { ConnectionTestStepRow } from '../components/ConnectionTestStepRow'
import { ConnectionTestSummary } from '../components/ConnectionTestSummary'
import { CONNECTION_TEST_GROUPS, summarizeSteps } from '../types'
@@ -70,6 +71,13 @@ const ConnectionTest = () => {
() => new Map(steps.map((step) => [step.id, step] as const)),
[steps]
)
// The summary headline downgrades to a warning when media flows but over a
// fallback route: TURN over TCP/TLS, or anything else that is not UDP.
const routeWarning = useMemo(() => {
const step = stepsById.get('selectedCandidate')
return step?.status === 'success' && isSuboptimalRoute(step.data)
}, [stepsById])
const isPublishVideoRunning =
stepsById.get('publishVideo')?.status === 'running'
@@ -99,7 +107,11 @@ const ConnectionTest = () => {
<CenteredContent withBackButton>
<Center>
<VStack gap="1.5rem" maxWidth="40rem" width="100%">
<ConnectionTestSummary stats={stats} isRunning={isRunning}>
<ConnectionTestSummary
stats={stats}
isRunning={isRunning}
routeWarning={routeWarning}
>
{isRunning ? (
// A disabled "run" button while the test runs is dead weight:
// cancelling is the only thing left to do.
@@ -46,6 +46,9 @@
"passedHint": "Ihr Browser, Ihre Geräte und Ihr Netzwerk sind für eine Besprechung bereit.",
"partial": "Teilweiser Test",
"partialHint": "Einige Prüfungen wurden übersprungen. Erlauben Sie den Zugriff auf Ihre Kamera und Ihr Mikrofon, um diese zu testen.",
"warning": "Verbindung nicht optimal",
"warningHint": "Achtung: Sie befinden sich nicht in optimalen Bedingungen für die Nutzung des Tools. Ihre Medien werden nicht über UDP übertragen (Rückgriff auf ein TURN-Relay über TCP oder TLS), was die Audio- und Videoqualität beeinträchtigen kann.",
"warningDocLink": "Technische Dokumentation für Ihre IT-Abteilung",
"failed_one": "{{count}} Prüfung fehlgeschlagen",
"failed_other": "{{count}} Prüfungen fehlgeschlagen",
"failedHint": "Öffnen Sie die fehlgeschlagenen Prüfungen für weitere Details und senden Sie den Bericht an Ihre IT-Abteilung."
@@ -46,6 +46,9 @@
"passedHint": "Your browser, your devices and your network are ready for a meeting.",
"partial": "Partially tested",
"partialHint": "Some checks were skipped. Allow access to your camera and microphone to test them.",
"warning": "Suboptimal connection",
"warningHint": "Warning: you are not in optimal conditions to use the tool. Your media is not carried over UDP (it falls back to a TURN relay over TCP or TLS), which can degrade audio and video quality.",
"warningDocLink": "Technical documentation for your IT department",
"failed_one": "{{count}} check failed",
"failed_other": "{{count}} checks failed",
"failedHint": "Open the failed checks below for details, then send the report to your IT department."
@@ -46,6 +46,9 @@
"passedHint": "Votre navigateur, vos périphériques et votre réseau sont prêts pour une réunion.",
"partial": "Test partiel",
"partialHint": "Certaines vérifications ont été ignorées. Autorisez l'accès à votre caméra et à votre microphone pour les tester.",
"warning": "Connexion non optimale",
"warningHint": "Attention : vous n'êtes pas dans les conditions optimales pour accéder à l'outil. Vos flux ne transitent pas en UDP (repli sur un relais TURN en TCP ou TLS), ce qui peut dégrader la qualité audio et vidéo.",
"warningDocLink": "Documentation technique à destination de votre service informatique",
"failed_one": "{{count}} vérification en échec",
"failed_other": "{{count}} vérifications en échec",
"failedHint": "Ouvrez les vérifications en échec pour voir le détail, puis transmettez le rapport à votre service informatique."
@@ -46,6 +46,9 @@
"passedHint": "Je browser, apparaten en netwerk zijn klaar voor een vergadering.",
"partial": "Gedeeltelijke test",
"partialHint": "Sommige controles zijn overgeslagen. Geef toegang tot je camera en microfoon om deze te testen.",
"warning": "Verbinding niet optimaal",
"warningHint": "Let op: je bevindt je niet in optimale omstandigheden om de tool te gebruiken. Je media loopt niet via UDP (terugval op een TURN-relay via TCP of TLS), wat de audio- en videokwaliteit kan verminderen.",
"warningDocLink": "Technische documentatie voor je IT-afdeling",
"failed_one": "{{count}} controle mislukt",
"failed_other": "{{count}} controles mislukt",
"failedHint": "Open de mislukte controles voor meer details en stuur het rapport door naar je IT-afdeling."
+2
View File
@@ -186,6 +186,8 @@ backend:
CELERY_BROKER_URL: redis://default:pass@redis-master:6379/1
# Recording & Transcription
RECORDING_ENABLE: True
RECORDING_STORAGE_EVENT_ENABLE: True
RECORDING_STORAGE_EVENT_TOKEN: password
SUMMARY_SERVICE_ENDPOINT: http://meet-summary:80/api/v2/async-jobs/transcribe/
SUMMARY_SERVICE_API_TOKEN: password
SUMMARY_SERVICE_WEBHOOK_API_TOKEN: webhook-password
+23
View File
@@ -113,3 +113,26 @@ spec:
exit 0
restartPolicy: Never
backoffLimit: 3
---
apiVersion: batch/v1
kind: Job
metadata:
name: minio-webhook
spec:
template:
spec:
containers:
- name: mc
image: minio/mc
command:
- /bin/sh
- -c
- |
/usr/bin/mc alias set meet http://minio:9000 meet password && \
/usr/bin/mc admin config set meet notify_webhook:meet-webhook endpoint="https://meet.127.0.0.1.nip.io/api/v1.0/recordings/storage-hook/" auth_token="Bearer password" && \
/usr/bin/mc admin service restart meet --wait --json && \
sleep 15 && \
/usr/bin/mc event add meet/meet-media-storage arn:minio:sqs::meet-webhook:webhook --event put --prefix "recordings" && \
exit 0
restartPolicy: Never
backoffLimit: 3
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "mail_mjml",
"version": "1.29.0",
"version": "1.28.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "mail_mjml",
"version": "1.29.0",
"version": "1.28.0",
"license": "MIT",
"dependencies": {
"@html-to/text-cli": "0.6.1",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mail_mjml",
"version": "1.29.0",
"version": "1.28.0",
"description": "An util to generate html and text django's templates from mjml templates",
"type": "module",
"dependencies": {
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "sdk",
"version": "1.29.0",
"version": "1.28.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "sdk",
"version": "1.29.0",
"version": "1.28.0",
"license": "ISC",
"workspaces": [
"./library",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "sdk",
"version": "1.29.0",
"version": "1.28.0",
"author": "",
"license": "ISC",
"description": "",
+7
View File
@@ -22,6 +22,13 @@ Configure your env values in `env.d/summary` to properly set up WhisperX and the
make run
```
When the stack is up, configure the MinIO webhook
*(TODO: add this step to `make bootstrap`)*
```sh
make minio-webhook-setup
```
If you want to develop on the Celery workers with hot reloading, run:
```sh
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "summary"
version = "1.29.0"
version = "1.28.0"
dependencies = [
"fastapi[standard]>=0.105.0",
"uvicorn>=0.24.0",