diff --git a/CHANGELOG.md b/CHANGELOG.md index 72093376..ec26eeaf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ and this project adheres to ## [Unreleased] +### Changed + +- 🔥(backend) remove the S3 storage-event webhook for recordings +- ♻️(backend) always finalize recordings using the LiveKit egress_ended webhook + ## [1.29.0] - 2026-08-25 ### Added diff --git a/UPGRADE.md b/UPGRADE.md index 9827f99d..ba5ad10b 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -16,6 +16,23 @@ the following command inside your docker container: ## [Unreleased] +### Removing S3 storage-event webhooks for recordings + +Recordings were previously confirmed as saved by an S3 storage-event webhook posting to `/api/v1.0/recordings/storage-hook/`. That endpoint has been removed: recordings are now always finalized from LiveKit's own `egress_ended` webhook, which has been the default path since v1.22.0. + +**Required for every deployment:** LiveKit must be able to deliver webhooks to the backend at `/api/v1.0/rooms/webhooks-livekit/`. This is now the only way a recording reaches a saved state; if `egress_ended` is never delivered, recordings stay in the `active` state. + +For hosters who had configured storage-event webhooks: +- Recordings reach the same final state, but they are now finalized when LiveKit reports the egress as ended rather than when the storage backend reports the upload. +- Remove the event notification from your bucket configuration: it now targets a non-existent endpoint and will fail on every delivery. + +For hosters who had **not** configured storage-event webhooks: +- Nothing changes. Recordings have been finalized from the `egress_ended` webhook since v1.22.0. + +In both cases, the following settings are no longer used and can be removed from your env: `RECORDING_EVENT_PARSER_CLASS`, `RECORDING_ENABLE_STORAGE_EVENT_AUTH`, `RECORDING_STORAGE_EVENT_ENABLE`, `RECORDING_STORAGE_EVENT_TOKEN`. + +On completion of the egress, a recording moves to `notification_succeeded`, or to `saved` if notifying external services failed. + ## v1.23.0 As part of the 1.23.0 release, the legacy `api/v1` implementation has been removed from the _experimental_ Summary service and Meet has been migrated to the new `api/v2`. diff --git a/compose.yml b/compose.yml index 76fedbd6..67b002d2 100644 --- a/compose.yml +++ b/compose.yml @@ -46,21 +46,6 @@ services: /usr/bin/mc mb meet/meet-media-storage && \ exit 0;" - createwebhook: - image: minio/mc - depends_on: - minio: - condition: service_healthy - restart: true - entrypoint: > - sh -c " - /usr/bin/mc alias set meet http://minio:9000 meet password && - /usr/bin/mc admin config set meet notify_webhook:meet-webhook endpoint='http://app-dev:8000/api/v1.0/recordings/storage-hook/' auth_token='Bearer password' && - /usr/bin/mc admin service restart meet --wait --json && - sleep 15 && - /usr/bin/mc event add meet/meet-media-storage arn:minio:sqs::meet-webhook:webhook --event put --prefix "recordings" && - exit 0;" - app-dev: build: context: . @@ -86,7 +71,6 @@ services: - mailcatcher - redis - createbuckets - - createwebhook extra_hosts: - "127.0.0.1.nip.io:host-gateway" networks: diff --git a/docs/features/recording.md b/docs/features/recording.md index 13cc06cb..6fcf82eb 100644 --- a/docs/features/recording.md +++ b/docs/features/recording.md @@ -23,14 +23,11 @@ It uses LiveKit Egress to record room sessions. For reference, see the [LiveKit To use the room recording feature, the following components are required: - A running [LiveKit Egress](https://github.com/livekit/egress) server capable of handling room composite recordings. -- A S3-compatible object storage that supports webhook events to notify the backend when recordings are uploaded. +- A S3-compatible object storage where the egress uploads the recorded files. - An email service to notify room owners when a recording is available for download. - Webhook events configured between LiveKit Server and the backend. -> [!CAUTION] -> Minio supports lifecycle events; other providers may not work out of the box. There is currently a dependency on Minio, which is planned to be refactored in the future. - > [!NOTE] > Celery isn’t in use for these async tasks yet. It’s something we’d like to add, but it’s not planned at this stage. @@ -75,7 +72,7 @@ sequenceDiagram LiveKit->>Egress: Stop recording Egress->>Storage: Upload recorded file - Storage->>Backend: Storage event notification + LiveKit->>Backend: POST /api/v1.0/rooms/webhooks-livekit/ (egress_ended) Backend->>Backend: Update Recording status to SAVED Backend->>Email: Send notification to room owner @@ -94,10 +91,6 @@ sequenceDiagram | **RECORDING_ENABLE** | Boolean | `False` | Enable or disable the room recording feature. | | **RECORDING_OUTPUT_FOLDER** | String | `"recordings"` | Folder/prefix where recordings are stored in the object storage. | | **RECORDING_WORKER_CLASSES** | Dict | `{ "screen_recording": "core.recording.worker.services.VideoCompositeEgressService", "transcript": "core.recording.worker.services.AudioCompositeEgressService" }` | Maps recording types to their worker service classes. | -| **RECORDING_EVENT_PARSER_CLASS** | String | `"core.recording.event.parsers.MinioParser"` | Class responsible for parsing storage events and updating the backend. | -| **RECORDING_ENABLE_STORAGE_EVENT_AUTH** | Boolean | `True` | Enable authentication for storage event webhook requests. | -| **RECORDING_STORAGE_EVENT_ENABLE** | Boolean | `False` | Enable handling of storage events (must configure webhook in storage). If `False`, fallback to LiveKit egress complete webhook. | -| **RECORDING_STORAGE_EVENT_TOKEN** | Secret/File | `None` | Token used to authenticate storage webhook requests, if `RECORDING_ENABLE_STORAGE_EVENT_AUTH` is enabled. | | **RECORDING_EXPIRATION_DAYS** | Integer | `None` | Number of days before recordings expire. Should match bucket lifecycle policy. Set to `None` for no expiration. | | **RECORDING_MAX_DURATION** | Integer | `None` | Maximum duration of a recording in milliseconds. Must be synced with the LiveKit Egress configuration. Set to None for unlimited duration. When the maximum duration is reached, the recording is automatically stopped and saved, and the user is prompted in the frontend with an alert message. | | **RECORDING_ENCODING_ENABLED** | Boolean | `False` | When `False`, LiveKit Egress uses its built-in `H264_720P_30` preset. When `True`, the `RECORDING_ENCODING_*` values below are sent to LiveKit as advanced `EncodingOptions`. See [Tuning recording encoding](#tuning-recording-encoding). | @@ -109,19 +102,6 @@ sequenceDiagram | **RECORDING_ENCODING_KEY_FRAME_INTERVAL_S** | Float | `4.0` | Keyframe interval in seconds. Drives seek granularity in the recorded MP4 (a player can only seek to keyframe boundaries). Larger values give the encoder slightly more bits for non-keyframe content at a fixed bitrate. `4.0` is a standard VOD value. Only applied when `RECORDING_ENCODING_ENABLED` is `True`. | -### Manual Storage Webhook - -Storage events must be configured manually; the Kubernetes chart does not do this automatically. - -1. Configure your S3 bucket to send file creation events to the backend webhook. -2. Enable events and token in settings: - -```python -RECORDING_STORAGE_EVENT_ENABLE = True -RECORDING_ENABLE_STORAGE_EVENT_AUTH = True -RECORDING_STORAGE_EVENT_TOKEN = -``` - > [!NOTE] > Questions? Open an issue on [GitHub](https://github.com/suitenumerique/meet/issues/new?assignees=&labels=bug&template=Bug_report.md) or join our [Matrix community](https://matrix.to/#/#meet-official:matrix.org). diff --git a/docs/installation/kubernetes.md b/docs/installation/kubernetes.md index c10326d7..016f2a1e 100644 --- a/docs/installation/kubernetes.md +++ b/docs/installation/kubernetes.md @@ -406,10 +406,6 @@ These are the environmental options available on meet backend. | RECORDING_ENABLE | Record meeting option | false | | RECORDING_OUTPUT_FOLDER | Folder to store meetings | recordings | | RECORDING_WORKER_CLASSES | Worker classes for recording | {"screen_recording": "core.recording.worker.services.VideoCompositeEgressService","transcript": "core.recording.worker.services.AudioCompositeEgressService"} | -| RECORDING_EVENT_PARSER_CLASS | Storage event engine for recording | core.recording.event.parsers.MinioParser | -| RECORDING_ENABLE_STORAGE_EVENT_AUTH | Enable storage event authorization | true | -| RECORDING_STORAGE_EVENT_ENABLE | Enable recording storage events. If false, fallback to egress webhook. | false | -| RECORDING_STORAGE_EVENT_TOKEN | Recording storage event token | | | RECORDING_EXPIRATION_DAYS | Recording expiration in days | | | RECORDING_MAX_DURATION | Maximum recording duration in milliseconds. Must match LiveKit Egress configuration exactly. | | | SCREEN_RECORDING_BASE_URL | Screen recording base URL | | diff --git a/env.d/development/common.dist b/env.d/development/common.dist index 161cf649..63d3ce16 100644 --- a/env.d/development/common.dist +++ b/env.d/development/common.dist @@ -63,8 +63,6 @@ ALLOW_UNREGISTERED_ROOMS=False # Recording RECORDING_ENABLE=True -RECORDING_STORAGE_EVENT_ENABLE=False -RECORDING_STORAGE_EVENT_TOKEN=password SUMMARY_SERVICE_ENDPOINT=http://app-summary-dev:8000/api/v2/async-jobs/transcribe/ SUMMARY_SERVICE_API_TOKEN=password SUMMARY_SERVICE_WEBHOOK_API_TOKEN=webhook-password diff --git a/src/backend/core/api/feature_flag.py b/src/backend/core/api/feature_flag.py index eca2a2fb..c000e225 100644 --- a/src/backend/core/api/feature_flag.py +++ b/src/backend/core/api/feature_flag.py @@ -11,7 +11,6 @@ class FeatureFlag: FLAGS = { "recording": "RECORDING_ENABLE", - "storage_event": "RECORDING_STORAGE_EVENT_ENABLE", "subtitle": "ROOM_SUBTITLE_ENABLED", "file_upload": "FILE_UPLOAD_ENABLED", "addons": "ADDONS_ENABLED", diff --git a/src/backend/core/api/viewsets.py b/src/backend/core/api/viewsets.py index 4dfb43b6..1b628057 100644 --- a/src/backend/core/api/viewsets.py +++ b/src/backend/core/api/viewsets.py @@ -45,25 +45,11 @@ from core.api import throttling from core.api.filters import ListFileFilter from core.enums import MEDIA_STORAGE_URL_PATTERN from core.recording.enums import FileExtension -from core.recording.event.authentication import ( - RecordingProcessWebhookAuthentication, - StorageEventAuthentication, -) -from core.recording.event.exceptions import ( - InvalidBucketError, - InvalidFilepathError, - InvalidFileTypeError, - ParsingEventDataError, -) -from core.recording.event.parsers import get_parser +from core.recording.event.authentication import RecordingProcessWebhookAuthentication from core.recording.services.metadata_collector import ( MetadataCollectorException, MetadataCollectorService, ) -from core.recording.services.recording_events import ( - RecordingEventsService, - RecordingNotSavableError, -) from core.recording.worker.exceptions import ( RecordingStartError, RecordingStopError, @@ -1034,56 +1020,6 @@ class RecordingViewSet( .filter(Q(accesses__user=user) | Q(accesses__team__in=user.get_teams())) ) - @decorators.action( - detail=False, - methods=["post"], - url_path="storage-hook", - authentication_classes=[StorageEventAuthentication], - ) - @FeatureFlag.require("storage_event") - def on_storage_event_received(self, request, pk=None): # pylint: disable=unused-argument - """Handle incoming storage hook events for recordings.""" - - parser = get_parser() - - try: - recording_id = parser.get_recording_id(request.data) - - except ParsingEventDataError as e: - raise drf_exceptions.PermissionDenied("Invalid request data.") from e - - except InvalidBucketError as e: - raise drf_exceptions.PermissionDenied("Invalid bucket specified.") from e - - except InvalidFilepathError: - return drf_response.Response( - {"message": "Notification ignored."}, - ) - - except InvalidFileTypeError: - return drf_response.Response( - {"message": "Notification ignored."}, - ) - - try: - recording = models.Recording.objects.get(id=recording_id) - except models.Recording.DoesNotExist as e: - raise drf_exceptions.NotFound("No recording found for this event.") from e - - # Save recording - recording_events_service = RecordingEventsService() - try: - recording_events_service.handle_complete(recording) - except RecordingNotSavableError: - raise drf_exceptions.PermissionDenied( - f"Recording with ID {recording_id} cannot be saved because it is either," - " in an error state or has already been saved." - ) from None - - return drf_response.Response( - {"message": "Event processed."}, - ) - @decorators.action( detail=False, methods=["post"], diff --git a/src/backend/core/recording/event/authentication.py b/src/backend/core/recording/event/authentication.py index eec74b5f..c68d79c8 100644 --- a/src/backend/core/recording/event/authentication.py +++ b/src/backend/core/recording/event/authentication.py @@ -1,4 +1,4 @@ -"""Authentication class for storage event token validation.""" +"""Authentication classes for server-to-server webhook token validation.""" import logging import secrets @@ -12,9 +12,9 @@ logger = logging.getLogger(__name__) class MachineUser: - """Represent a non-interactive system user for automated storage operations.""" + """Represent a non-interactive system user for automated operations.""" - def __init__(self, username: str = "storage_event_user") -> None: + def __init__(self, username: str = "machine_user") -> None: self.pk = None self.username = username self.is_active = True @@ -41,24 +41,17 @@ class HeaderBasedAuthentication(BaseAuthentication): TOKEN_TYPE = "Bearer" # noqa S105 REALM = "" - IS_ENFORCED_SETTINGS_KEY = None EXPECTED_TOKEN_SETTINGS_KEY = None def authenticate(self, request): """Validate the Bearer token from the Authorization header.""" - if self.IS_ENFORCED_SETTINGS_KEY is not None: - if not getattr(settings, self.IS_ENFORCED_SETTINGS_KEY): - return MachineUser(), None - if ( self.EXPECTED_TOKEN_SETTINGS_KEY is None or (required_token := getattr(settings, self.EXPECTED_TOKEN_SETTINGS_KEY)) is None ): - raise AuthenticationFailed( - "Authentication is enabled but token is not configured." - ) + raise AuthenticationFailed("Authentication token is not configured.") auth_header = request.headers.get(self.AUTH_HEADER) if not auth_header: @@ -88,18 +81,6 @@ class HeaderBasedAuthentication(BaseAuthentication): return f"{self.TOKEN_TYPE} realm='{self.REALM}'" -class StorageEventAuthentication(HeaderBasedAuthentication): - """Authenticate requests using a Bearer token for storage event integration. - This class validates Bearer tokens for storage events that don't map to database users. - It's designed for S3-compatible storage integrations and similar use cases. - Events are submitted when a webhook is configured on some bucket's events. - """ - - REALM = "Storage event API" - IS_ENFORCED_SETTINGS_KEY = "RECORDING_ENABLE_STORAGE_EVENT_AUTH" - EXPECTED_TOKEN_SETTINGS_KEY = "RECORDING_STORAGE_EVENT_TOKEN" # noqa S105 - - class RecordingProcessWebhookAuthentication(HeaderBasedAuthentication): """ Custom authentication class for recording process webhook requests. diff --git a/src/backend/core/recording/event/exceptions.py b/src/backend/core/recording/event/exceptions.py deleted file mode 100644 index d562a9d3..00000000 --- a/src/backend/core/recording/event/exceptions.py +++ /dev/null @@ -1,17 +0,0 @@ -"""Storage parsers specific exceptions.""" - - -class ParsingEventDataError(Exception): - """Raised when the request data is malformed, incomplete, or missing.""" - - -class InvalidBucketError(Exception): - """Raised when the bucket name in the request does not match the expected one.""" - - -class InvalidFileTypeError(Exception): - """Raised when the file type in the request is not supported.""" - - -class InvalidFilepathError(Exception): - """Raised when the filepath in the request is invalid.""" diff --git a/src/backend/core/recording/event/parsers.py b/src/backend/core/recording/event/parsers.py deleted file mode 100644 index 1bdf3995..00000000 --- a/src/backend/core/recording/event/parsers.py +++ /dev/null @@ -1,178 +0,0 @@ -"""Meet storage event parser classes.""" - -import logging -import mimetypes -import re -from dataclasses import dataclass -from functools import lru_cache -from typing import Any, Dict, Optional, Protocol -from urllib.parse import quote - -from django.conf import settings -from django.utils.module_loading import import_string - -from core.enums import FILE_EXT_REGEX, UUID_REGEX - -from .exceptions import ( - InvalidBucketError, - InvalidFilepathError, - InvalidFileTypeError, - ParsingEventDataError, -) - -# Additional MIME type mapping -mimetypes.add_type("audio/ogg", ".ogg") - -logger = logging.getLogger(__name__) - - -@dataclass -class StorageEvent: - """Represents a storage event with relevant metadata. - Attributes: - filepath: Identifier for the affected recording - filetype: Type of storage event - bucket_name: When the event occurred - metadata: Additional event data - """ - - filepath: str - filetype: str - bucket_name: str - metadata: Optional[Dict[str, Any]] - - def __post_init__(self): - if self.filepath is None: - raise TypeError("filepath cannot be None") - if self.filetype is None: - raise TypeError("filetype cannot be None") - if self.bucket_name is None: - raise TypeError("bucket_name cannot be None") - - -class EventParser(Protocol): - """Interface for parsing storage events.""" - - def __init__(self, bucket_name, allowed_filetypes=None): - """Initialize parser with bucket name and optional allowed filetypes.""" - - def parse(self, data: Dict) -> StorageEvent: - """Extract storage event data from raw dictionary input.""" - - def validate(self, data: StorageEvent) -> str: - """Verify storage event data meets all requirements.""" - - def get_recording_id(self, data: Dict) -> str: - """Extract recording ID from event dictionary.""" - - -@lru_cache(maxsize=1) -def get_parser() -> EventParser: - """Return cached instance of configured event parser. - Uses function memoization instead of a factory class since the only - varying parameter is the parser class from settings. A factory class - would add unnecessary complexity when a cached function provides the - same singleton behavior with simpler code. - """ - - event_parser_cls = import_string(settings.RECORDING_EVENT_PARSER_CLASS) - return event_parser_cls(bucket_name=settings.AWS_STORAGE_BUCKET_NAME) - - -class BaseS3Parser: - """Base class for handling parsing and validation of S3-compatible storage events.""" - - def __init__(self, bucket_name: str, allowed_filetypes=None): - """Initialize parser with target bucket name and accepted filetypes.""" - - if not bucket_name: - raise ValueError("Bucket name cannot be None or empty") - - self._bucket_name = bucket_name - self._allowed_filetypes = allowed_filetypes or {"audio/ogg", "video/mp4"} - - # pylint: disable=line-too-long - self._filepath_regex = re.compile( - rf"(?P(?:[^%]+%2F)+)?{settings.RECORDING_OUTPUT_FOLDER}%2F(?P{UUID_REGEX})\.(?P{FILE_EXT_REGEX})" - ) - - def validate(self, event_data: StorageEvent) -> str: - """Verify StorageEvent matches bucket, filetype and filepath requirements.""" - - if event_data.bucket_name != self._bucket_name: - raise InvalidBucketError( - f"Invalid bucket: expected {self._bucket_name}, got {event_data.bucket_name}" - ) - - if event_data.filetype not in self._allowed_filetypes: - raise InvalidFileTypeError( - f"Invalid file type, expected {self._allowed_filetypes}," - f"got '{event_data.filetype}'" - ) - - match = self._filepath_regex.match(event_data.filepath) - if not match: - raise InvalidFilepathError( - f"Invalid filepath structure: {event_data.filepath}" - ) - - recording_id = match.group("recording_id") - return recording_id - - def get_recording_id(self, data): - """Extract recording ID from S3 event through parsing and validation.""" - - event_data = self.parse(data) - return self.validate(event_data) - - def parse(self, data: Dict) -> StorageEvent: - """To be implemented by subclasses.""" - raise NotImplementedError("Subclasses must implement parse()") - - -class MinioParser(BaseS3Parser): - """Minio specific event parsing.""" - - def parse(self, data: Dict) -> StorageEvent: - if not data: - raise ParsingEventDataError("Received empty data.") - try: - record = data["Records"][0] - s3 = record["s3"] - return StorageEvent( - filepath=s3["object"]["key"], - filetype=s3["object"]["contentType"], # Minio-specific field - bucket_name=s3["bucket"]["name"], - metadata=None, - ) - except (KeyError, IndexError) as e: - raise ParsingEventDataError(f"Malformed Minio event: {e}") from e - except TypeError as e: - raise ParsingEventDataError(f"Missing essential data fields: {e}") from e - - -class S3Parser(BaseS3Parser): - """AWS S3 specific event parsing.""" - - def parse(self, data: Dict) -> StorageEvent: - if not data: - raise ParsingEventDataError("Received empty data.") - try: - # AWS S3 structure can slightly differ from Minio implementation - record = data["Records"][0] - s3 = record["s3"] - filepath = s3["object"]["key"] - if not filepath: - raise ParsingEventDataError("Missing object key name") - filetype, _ = mimetypes.guess_type(filepath) - # Normalize raw S3-compatible object keys without re-encoding - # already encoded AWS S3 notification keys. - filepath = quote(filepath, safe="%+") - return StorageEvent( - filepath=filepath, - filetype=filetype, - bucket_name=s3["bucket"]["name"], - metadata=None, - ) - except (KeyError, IndexError) as e: - raise ParsingEventDataError(f"Malformed S3 event: {e}") from e diff --git a/src/backend/core/services/livekit_events.py b/src/backend/core/services/livekit_events.py index 939194ff..3f396e6f 100644 --- a/src/backend/core/services/livekit_events.py +++ b/src/backend/core/services/livekit_events.py @@ -220,10 +220,8 @@ class LiveKitEventsService: f"Failed to process limit reached event for recording {recording}" ) from e - # Fallback for completion when no MinIO/S3 webhooks are configured - if ( - not settings.RECORDING_STORAGE_EVENT_ENABLE - ) and data.egress_info.status in [ + # Finalize the recording, the egress has uploaded the file to the storage + if data.egress_info.status in [ api.EgressStatus.EGRESS_COMPLETE, api.EgressStatus.EGRESS_LIMIT_REACHED, ]: diff --git a/src/backend/core/tests/recording/event/test_authentication.py b/src/backend/core/tests/recording/event/test_authentication.py index 74db3abd..19c8dd94 100644 --- a/src/backend/core/tests/recording/event/test_authentication.py +++ b/src/backend/core/tests/recording/event/test_authentication.py @@ -11,135 +11,98 @@ from rest_framework.exceptions import AuthenticationFailed from core.recording.event.authentication import ( MachineUser, - StorageEventAuthentication, + RecordingProcessWebhookAuthentication, ) def test_successful_authentication(settings): """Test successful authentication with valid token.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {"Authorization": "Bearer valid-test-token"} - user, token = StorageEventAuthentication().authenticate(request) + user, token = RecordingProcessWebhookAuthentication().authenticate(request) assert token == "valid-test-token" assert isinstance(user, MachineUser) -def test_disabled_authentication_with_header(settings): - """Authentication should pass when no auth is configured, and header is present.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = None - settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False - - request = RequestFactory().get("/") - request.headers = {"Authorization": "Bearer some-token"} - - user, token = StorageEventAuthentication().authenticate(request) - assert token is None - assert isinstance(user, MachineUser) - - -def test_disabled_authentication_without_header(settings): - """Authentication should pass when no auth is configured, and no header is present.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = None - settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False - - request = RequestFactory().get("/") - - user, token = StorageEventAuthentication().authenticate(request) - assert token is None - assert isinstance(user, MachineUser) - - -def test_authentication_when_disabled(settings): - """Authentication should pass when disabled, regardless of token configuration.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "some-token" - settings.RECORDING_ENABLE_STORAGE_EVENT_AUTH = False - - request = RequestFactory().get("/") - - user, token = StorageEventAuthentication().authenticate(request) - assert token is None - assert isinstance(user, MachineUser) - - def test_authentication_fails_when_token_not_configured(settings): - """Authentication should fail when authentication is enabled but no token is configured.""" + """Authentication should fail when no token is configured.""" - # By default RECORDING_ENABLE_STORAGE_EVENT_AUTH should be True - settings.RECORDING_STORAGE_EVENT_TOKEN = None + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = None request = RequestFactory().get("/") with pytest.raises( AuthenticationFailed, - match="Authentication is enabled but token is not configured", + match="Authentication token is not configured", ): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_missing_auth_header(settings): """Test failure when Authorization header is missing.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {} with pytest.raises(AuthenticationFailed, match="Authorization header is required"): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_invalid_auth_header_format(settings): """Test failure when Authorization header has invalid format.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {"Authorization": "InvalidFormat"} with pytest.raises(AuthenticationFailed, match="Invalid authorization header"): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_invalid_token_type(settings): """Test failure when token type is not Bearer.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {"Authorization": "Basic some-token"} with pytest.raises(AuthenticationFailed, match="Invalid authorization header"): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_invalid_token(settings): """Test failure when token is invalid.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {"Authorization": "Bearer wrong-token"} with pytest.raises(AuthenticationFailed, match="Invalid token"): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_malformed_auth_header(settings): """Test failure when Authorization header is malformed.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "valid-test-token" request = RequestFactory().get("/") request.headers = {"Authorization": "Bearer"} # Missing token part with pytest.raises(AuthenticationFailed, match="Invalid authorization header"): - StorageEventAuthentication().authenticate(request) + RecordingProcessWebhookAuthentication().authenticate(request) def test_authenticate_header(): """Test the WWW-Authenticate header value.""" request = RequestFactory().get("/") - header = StorageEventAuthentication().authenticate_header(request) - assert header == "Bearer realm='Storage event API'" + header = RecordingProcessWebhookAuthentication().authenticate_header(request) + assert header == "Bearer realm='External process webhook API'" def test_multiple_spaces_in_auth_header(settings): - """Test success when Authorization header contains multiple spaces.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "valid-test-token" + """Extra spaces between the scheme and the token should be tolerated.""" + settings.SUMMARY_SERVICE_WEBHOOK_API_TOKEN = "extra-spaces-token" request = RequestFactory().get("/") request.headers = {"Authorization": "Bearer extra-spaces-token"} - header = StorageEventAuthentication().authenticate_header(request) - assert header == "Bearer realm='Storage event API'" + user, token = RecordingProcessWebhookAuthentication().authenticate(request) + assert token == "extra-spaces-token" + assert isinstance(user, MachineUser) diff --git a/src/backend/core/tests/recording/event/test_parsers.py b/src/backend/core/tests/recording/event/test_parsers.py deleted file mode 100644 index fcbf0694..00000000 --- a/src/backend/core/tests/recording/event/test_parsers.py +++ /dev/null @@ -1,512 +0,0 @@ -""" -Test event parsers. -""" - -# pylint: disable=protected-access,redefined-outer-name,unused-argument - -from unittest import mock - -from django.conf import settings - -import pytest - -from core.recording.event.exceptions import ( - InvalidBucketError, - InvalidFilepathError, - InvalidFileTypeError, - ParsingEventDataError, -) -from core.recording.event.parsers import ( - MinioParser, - S3Parser, - StorageEvent, - get_parser, -) - -# MinioParser - - -@pytest.fixture -def valid_minio_event(): - """Mock a valid Minio event.""" - return { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - "contentType": "audio/ogg", - }, - } - } - ] - } - - -@pytest.fixture -def minio_parser(): - """Mock a Minio parser.""" - return MinioParser(bucket_name="test-bucket") - - -def test_minio_parse_valid_event(minio_parser, valid_minio_event): - """Test parsing a valid Minio event.""" - event = minio_parser.parse(valid_minio_event) - assert isinstance(event, StorageEvent) - assert event.filepath == "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg" - assert event.filetype == "audio/ogg" - assert event.bucket_name == "test-bucket" - assert event.metadata is None - - -def test_minio_parse_with_video_type(minio_parser): - """Test parsing event with video file type.""" - video_event = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "46d1a121-2426-484d-8fb3-09b5d886f7a8.mp4", - "contentType": "video/mp4", - }, - } - } - ] - } - event = minio_parser.parse(video_event) - assert event.filetype == "video/mp4" - assert event.filepath.endswith(".mp4") - - -def test_minio_parse_empty_data(minio_parser): - """Test parsing empty event data raises error.""" - with pytest.raises(ParsingEventDataError, match="Received empty data."): - minio_parser.parse({}) - - -def test_minio_parse_missing_keys(minio_parser): - """Test parsing event with missing key.""" - - invalid_minio_event = { - "Records": [ - { - "s3": { - "bucket": {"name": None}, - # Missing 'object' key - } - } - ] - } - - with pytest.raises(ParsingEventDataError, match="Malformed Minio event:"): - minio_parser.parse(invalid_minio_event) - - -def test_minio_parse_none_key(minio_parser): - """Test parsing event with None field.""" - - invalid_minio_event = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - "contentType": None, # 'contentType' should not be None - }, - } - } - ] - } - - with pytest.raises(ParsingEventDataError, match="Missing essential data fields"): - minio_parser.parse(invalid_minio_event) - - -def test_minio_validate_invalid_bucket(minio_parser): - """Test validation with wrong bucket name.""" - event = StorageEvent( - filepath="recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - filetype="audio/ogg", - bucket_name="wrong-bucket", - metadata=None, - ) - with pytest.raises(InvalidBucketError): - minio_parser.validate(event) - - -def test_minio_validate_invalid_filetype(minio_parser): - """Test validation with unsupported file type.""" - event = StorageEvent( - filepath="recording%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.txt", - filetype="text/plain", # Not included in the default allowed filetypes - bucket_name="test-bucket", - metadata=None, - ) - with pytest.raises(InvalidFileTypeError): - minio_parser.validate(event) - - -@pytest.mark.parametrize( - "invalid_filepath", - [ - "invalid_filepath", # totally invalid string - "recordings/46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - "recordings/46d1a121-2426-484d-8fb3-09b5d886f7a8", # missing extension - "46d1a121-2426-484d-8fb3-09b5d886f7a8", # missing url_encoded_folder_path and extension - "", # empty string - "46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # no folder at all - "uploads%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # wrong folder name - "folder%2Fuploads%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", # nested but no recordings/ - ], -) -def test_minio_validate_invalid_filepath(invalid_filepath, minio_parser): - """Test validation with malformed filepath.""" - event = StorageEvent( - filepath=invalid_filepath, - filetype="audio/ogg", - bucket_name="test-bucket", - metadata=None, - ) - with pytest.raises(InvalidFilepathError): - minio_parser.validate(event) - - -def test_minio_validate_valid_event(minio_parser): - """Test validation with valid event data.""" - event = StorageEvent( - filepath="recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - filetype="audio/ogg", - bucket_name="test-bucket", - metadata=None, - ) - recording_id = minio_parser.validate(event) - assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8" - - -def test_minio_get_recording_id_success(minio_parser, valid_minio_event): - """Test successful extraction of recording ID.""" - recording_id = minio_parser.get_recording_id(valid_minio_event) - assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8" - - -def test_minio_validate_filepath_with_folder(minio_parser): - """Test validation of filepath with folder structure.""" - event = StorageEvent( - filepath="parent_folder%2Frecordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - filetype="audio/ogg", - bucket_name="test-bucket", - metadata=None, - ) - recording_id = minio_parser.validate(event) - assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8" - - -def test_minio_empty_allowed_filetypes(): - """Test MinioParser with empty allowed_filetypes.""" - empty_types = set() - parser = MinioParser(bucket_name="test-bucket", allowed_filetypes=empty_types) - assert parser._allowed_filetypes == {"audio/ogg", "video/mp4"} - - -def test_minio_custom_allowed_filetypes(): - """Test MinioParser with empty allowed_filetypes.""" - custom_types = {"audio/mp3", "video/mov"} - parser = MinioParser(bucket_name="test-bucket", allowed_filetypes=custom_types) - assert parser._allowed_filetypes == {"audio/mp3", "video/mov"} - - -def test_minio_validate_custom_filetypes(): - """Test validation of filepath with folder structure.""" - - parser = MinioParser(bucket_name="test-bucket", allowed_filetypes={"audio/mp3"}) - - event = StorageEvent( - filepath="parent_folder%2Frecordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - filetype="audio/mp3", - bucket_name="test-bucket", - metadata=None, - ) - parser.validate(event) - - -def test_minio_constructor_none_bucket(): - """Test MinioParser constructor with None bucket name.""" - with pytest.raises(ValueError, match="Bucket name cannot be None or empty"): - MinioParser(bucket_name=None) - - -def test_minio_constructor_empty_bucket(): - """Test MinioParser constructor with empty bucket name.""" - with pytest.raises(ValueError, match="Bucket name cannot be None or empty"): - MinioParser(bucket_name="") - - -# S3Parser - - -@pytest.fixture -def valid_s3_event(): - """Mock a valid S3 event.""" - return { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg", - }, - } - } - ] - } - - -@pytest.fixture -def s3_parser(): - """Mock an S3 parser.""" - return S3Parser(bucket_name="test-bucket") - - -def test_s3_parse_valid_event(s3_parser, valid_s3_event): - """Test parsing a valid S3 event.""" - event = s3_parser.parse(valid_s3_event) - assert isinstance(event, StorageEvent) - assert event.filepath == "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.ogg" - assert event.filetype == "audio/ogg" - assert event.bucket_name == "test-bucket" - assert event.metadata is None - - -def test_s3_parse_empty_data(s3_parser): - """Test parsing empty S3 event data raises error.""" - with pytest.raises(ParsingEventDataError, match="Received empty data."): - s3_parser.parse({}) - - -def test_s3_parse_missing_keys(s3_parser): - """Test parsing S3 event with missing key.""" - invalid_s3_event = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - # Missing 'object' key - } - } - ] - } - - with pytest.raises(ParsingEventDataError, match="Malformed S3 event:"): - s3_parser.parse(invalid_s3_event) - - -def test_s3_parse_none_key(s3_parser): - """Test parsing S3 event with None field.""" - invalid_s3_event = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": None, - }, - } - } - ] - } - - with pytest.raises(ParsingEventDataError, match="Missing object key name"): - s3_parser.parse(invalid_s3_event) - - -def test_s3_parse_with_video_type(s3_parser): - """Test parsing S3 event with mp4 file extension.""" - video_event = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.mp4", - }, - } - } - ] - } - event = s3_parser.parse(video_event) - assert event.filetype == "video/mp4" - assert event.filepath.endswith(".mp4") - - -def test_s3_parse_unrecognized_extension(s3_parser): - """Test parsing S3 event with unrecognized file extension.""" - event_with_unknown_ext = { - "Records": [ - { - "s3": { - "bucket": {"name": "test-bucket"}, - "object": { - "key": "recordings%2F46d1a121-2426-484d-8fb3-09b5d886f7a8.zzunknown999", - }, - } - } - ] - } - - with pytest.raises(TypeError, match="filetype cannot be None"): - s3_parser.parse(event_with_unknown_ext) - - -def test_s3_parser_keeps_encoded_filepath_compatible(settings): - """Test S3 parser keeps already encoded object keys compatible.""" - settings.RECORDING_OUTPUT_FOLDER = "recordings" - - recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d" - parser = S3Parser(bucket_name="recordings-bucket") - - data = { - "Records": [ - { - "s3": { - "bucket": {"name": "recordings-bucket"}, - "object": { - "key": f"recordings%2F{recording_id}.mp4", - }, - } - } - ] - } - - assert parser.get_recording_id(data) == recording_id - - -def test_s3_parser_accepts_unencoded_filepath(settings): - """Test S3 parser accepts raw object keys with slash separators.""" - settings.RECORDING_OUTPUT_FOLDER = "recordings" - - recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d" - parser = S3Parser(bucket_name="recordings-bucket") - - data = { - "Records": [ - { - "s3": { - "bucket": {"name": "recordings-bucket"}, - "object": { - "key": f"recordings/{recording_id}.mp4", - }, - } - } - ] - } - - assert parser.get_recording_id(data) == recording_id - - -def test_s3_parser_preserves_plus_signs_in_encoded_filepath(settings): - """Test S3 parser preserves plus signs in already encoded object keys.""" - settings.RECORDING_OUTPUT_FOLDER = "recordings" - - recording_id = "80ae9fe5-639a-438b-b86e-9e3dd2d55f4d" - parser = S3Parser(bucket_name="recordings-bucket") - - data = { - "Records": [ - { - "s3": { - "bucket": {"name": "recordings-bucket"}, - "object": { - "key": f"folder+name%2Frecordings%2F{recording_id}.mp4", - }, - } - } - ] - } - - assert parser.get_recording_id(data) == recording_id - - -def test_s3_get_recording_id_success(s3_parser, valid_s3_event): - """Test successful extraction of recording ID from S3 event.""" - recording_id = s3_parser.get_recording_id(valid_s3_event) - assert recording_id == "46d1a121-2426-484d-8fb3-09b5d886f7a8" - - -# get_parser - - -@pytest.fixture -def clear_lru_cache(): - """Fixture to clear the LRU cache between tests.""" - get_parser.cache_clear() - yield - get_parser.cache_clear() - - -def test_returns_correct_instance(clear_lru_cache): - """Test if get_parser returns the correct parser instance.""" - settings.AWS_STORAGE_BUCKET_NAME = "test-bucket" - parser = get_parser() - assert isinstance(parser, MinioParser) - assert parser._bucket_name == "test-bucket" - - -def test_caching_behavior(clear_lru_cache): - """Test if the function properly caches the parser instance.""" - settings.AWS_STORAGE_BUCKET_NAME = "test-bucket" - parser1 = get_parser() - parser2 = get_parser() - assert parser1 is parser2 # Check object identity - - -def test_different_settings_new_instance(): - """Test if changing settings creates a new instance.""" - settings.AWS_STORAGE_BUCKET_NAME = "different-bucket" - parser = get_parser() - assert parser._bucket_name == "different-bucket" - - -def test_import_error_handling(clear_lru_cache): - """Test handling of import errors for invalid parser class.""" - settings.RECORDING_EVENT_PARSER_CLASS = "invalid.parser.path" - with pytest.raises(ImportError): - get_parser() - - -@mock.patch("core.recording.event.parsers.import_string") -def test_parser_instantiation_called_once(mock_import_string, clear_lru_cache): - """Test that parser class is instantiated only once due to caching.""" - mock_parser_cls = type( - "MockParser", - (), - { - "__init__": lambda self, bucket_name: setattr( - self, "_bucket_name", bucket_name - ) - }, - ) - mock_import_string.return_value = mock_parser_cls - - # First call - parser1 = get_parser() - # Second call - parser2 = get_parser() - - # Verify import_string was called only once - mock_import_string.assert_called_once_with(settings.RECORDING_EVENT_PARSER_CLASS) - assert parser1 is parser2 - - -def test_cache_clear_behavior(clear_lru_cache, settings): - """Test that cache clearing creates new instance.""" - - settings.RECORDING_EVENT_PARSER_CLASS = "core.recording.event.parsers.MinioParser" - - parser1 = get_parser() - get_parser.cache_clear() - parser2 = get_parser() - - assert parser1 is not parser2 # Should be different instances after cache clear diff --git a/src/backend/core/tests/recording/service/test_recording_events.py b/src/backend/core/tests/recording/service/test_recording_events.py index f878117f..cfd2b28a 100644 --- a/src/backend/core/tests/recording/service/test_recording_events.py +++ b/src/backend/core/tests/recording/service/test_recording_events.py @@ -12,6 +12,7 @@ from core.factories import RecordingFactory from core.recording.services.recording_events import ( RecordingEventsError, RecordingEventsService, + RecordingNotSavableError, ) from core.utils import NotificationError @@ -70,3 +71,56 @@ def test_handle_limit_reached_error(mock_notify, mode, notification_type, servic mock_notify.assert_called_once_with( room_name=str(recording.room.id), notification_data={"type": notification_type} ) + + +@pytest.mark.parametrize("status", ["active", "stopped"]) +@pytest.mark.parametrize( + ("notify_return_value", "expected_status"), + ((True, "notification_succeeded"), (False, "saved")), +) +@mock.patch( + "core.recording.services.recording_events.notification_service." + "notify_external_services" +) +def test_handle_complete_saves_recording( # pylint: disable=too-many-arguments, too-many-positional-arguments + mock_notify_external_services, + notify_return_value, + expected_status, + status, + service, +): + """Test handle_complete notifies external services and saves a savable recording.""" + + mock_notify_external_services.return_value = notify_return_value + + recording = RecordingFactory(status=status) + service.handle_complete(recording) + + mock_notify_external_services.assert_called_once_with(recording) + + recording.refresh_from_db() + assert recording.status == expected_status + + +@pytest.mark.parametrize( + "status", + ["initiated", "saved", "notification_succeeded", "aborted", "failed_to_start"], +) +@mock.patch( + "core.recording.services.recording_events.notification_service." + "notify_external_services" +) +def test_handle_complete_non_savable_recording( + mock_notify_external_services, status, service +): + """Test handle_complete refuses recordings that are already saved or in error.""" + + recording = RecordingFactory(status=status) + + with pytest.raises(RecordingNotSavableError): + service.handle_complete(recording) + + mock_notify_external_services.assert_not_called() + + recording.refresh_from_db() + assert recording.status == status diff --git a/src/backend/core/tests/recording/test_api_recordings_storage_hook.py b/src/backend/core/tests/recording/test_api_recordings_storage_hook.py deleted file mode 100644 index 5603523c..00000000 --- a/src/backend/core/tests/recording/test_api_recordings_storage_hook.py +++ /dev/null @@ -1,267 +0,0 @@ -""" -Test recordings API endpoints in the Meet core app: save recording. -""" - -# pylint: disable=redefined-outer-name,unused-argument - -import uuid -from unittest import mock - -import pytest -from rest_framework.test import APIClient - -from ...factories import RecordingFactory -from ...models import Recording, RecordingStatusChoices -from ...recording.event.exceptions import ( - InvalidBucketError, - InvalidFilepathError, - InvalidFileTypeError, - ParsingEventDataError, -) - -pytestmark = pytest.mark.django_db - - -@pytest.fixture -def recording_settings(settings): - """Configure recording-related and storage event Django settings.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken" - settings.RECORDING_STORAGE_EVENT_ENABLE = True - return settings - - -@pytest.fixture -def mock_get_parser(): - """Mock 'get_parser' factory function.""" - with mock.patch("core.api.viewsets.get_parser") as mock_parser: - yield mock_parser - - -def test_save_recording_anonymous(settings, client): - """Anonymous users should not be allowed to save room recordings.""" - settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken" - - RecordingFactory(status="active") - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - ) - - assert response.status_code == 401 - assert Recording.objects.count() == 1 - - -def test_save_recording_wrong_bearer(settings, client): - """Requests with incorrect bearer token should be rejected when auth is required.""" - - settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken" - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer wrongAuthToken", - ) - - assert response.status_code == 401 - - -def test_save_recording_permission_needed(settings, client): - """Recordings should not be saved when feature is disabled.""" - - settings.RECORDING_STORAGE_EVENT_TOKEN = "testAuthToken" - settings.RECORDING_STORAGE_EVENT_ENABLE = False - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 404 - assert response.json() == {"detail": "Not found."} - - -def test_save_recording_parsing_error(recording_settings, mock_get_parser, client): - """Test handling of parsing errors in recording event data.""" - mock_parser = mock.Mock() - mock_parser.get_recording_id.side_effect = ParsingEventDataError("Error message") - mock_get_parser.return_value = mock_parser - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 403 - assert response.json() == {"detail": "Invalid request data."} - - -def test_save_recording_bucket_error(recording_settings, mock_get_parser, client): - """Test handling of invalid storage bucket errors in recording event data.""" - - mock_parser = mock.Mock() - mock_parser.get_recording_id.side_effect = InvalidBucketError("Error message") - mock_get_parser.return_value = mock_parser - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 403 - assert response.json() == {"detail": "Invalid bucket specified."} - - -def test_save_recording_filetype_error(recording_settings, mock_get_parser): - """Test handling of unsupported file types in recording event data.""" - - mock_parser = mock.Mock() - mock_parser.get_recording_id.side_effect = InvalidFileTypeError( - "unsupported '.json'" - ) - mock_get_parser.return_value = mock_parser - - client = APIClient() - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 200 - assert response.json() == {"message": "Notification ignored."} - - -def test_save_recording_filepath_error(recording_settings, mock_get_parser): - """Test handling of unsupported filepath in recording event data.""" - - mock_parser = mock.Mock() - mock_parser.get_recording_id.side_effect = InvalidFilepathError( - "Invalid filepath structure: parent/folder/recording.jpeg" - ) - mock_get_parser.return_value = mock_parser - - client = APIClient() - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 200 - assert response.json() == {"message": "Notification ignored."} - - -def test_save_recording_unknown_recording(recording_settings, mock_get_parser, client): - """Test handling of events for non-existent recordings.""" - - RecordingFactory(status="active") - - mock_parser = mock.Mock() - mock_parser.get_recording_id.return_value = uuid.uuid4() - mock_get_parser.return_value = mock_parser - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 404 - assert response.json() == {"detail": "No recording found for this event."} - - -@pytest.mark.parametrize( - "status", ["failed_to_start", "aborted", "failed_to_stop", "saved", "initiated"] -) -def test_save_recording_non_savable_recording( - recording_settings, mock_get_parser, client, status -): - """Test that recordings in non-savable states cannot be saved.""" - - recording = RecordingFactory(status=status) - - mock_parser = mock.Mock() - mock_parser.get_recording_id.return_value = recording.id - mock_get_parser.return_value = mock_parser - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 403 - assert response.json() == { - "detail": f"Recording with ID {recording.id} cannot be saved because it is either," - " in an error state or has already been saved." - } - - -@pytest.mark.parametrize("status", ["active", "stopped"]) -def test_save_recording_success(recording_settings, mock_get_parser, client, status): - """Test successful saving of recordings in valid states.""" - - recording = RecordingFactory(status=status) - - mock_parser = mock.Mock() - mock_parser.get_recording_id.return_value = recording.id - mock_get_parser.return_value = mock_parser - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 200 - assert response.json() == {"message": "Event processed."} - - recording.refresh_from_db() - assert recording.status == RecordingStatusChoices.SAVED - - -@mock.patch( - "core.recording.services.recording_events.notification_service." - "notify_external_services" -) -@pytest.mark.parametrize("notification_succeeded", [True, False]) -def test_save_recording_notifies_external_services( - mock_notify_external_services, - recording_settings, - mock_get_parser, - client, - notification_succeeded, -): - """External services should be notified when a recording is saved.""" - - recording = RecordingFactory(status="active") - - mock_parser = mock.Mock() - mock_parser.get_recording_id.return_value = recording.id - mock_get_parser.return_value = mock_parser - - mock_notify_external_services.return_value = notification_succeeded - - response = client.post( - "/api/v1.0/recordings/storage-hook/", - {"recording_data": "valid-data"}, - HTTP_AUTHORIZATION="Bearer testAuthToken", - ) - - assert response.status_code == 200 - assert response.json() == {"message": "Event processed."} - - mock_notify_external_services.assert_called_once_with(recording) - - recording.refresh_from_db() - assert recording.status == ( - RecordingStatusChoices.NOTIFICATION_SUCCEEDED - if notification_succeeded - else RecordingStatusChoices.SAVED - ) diff --git a/src/backend/core/tests/services/test_livekit_events.py b/src/backend/core/tests/services/test_livekit_events.py index 548c71c5..2726f5c1 100644 --- a/src/backend/core/tests/services/test_livekit_events.py +++ b/src/backend/core/tests/services/test_livekit_events.py @@ -75,12 +75,11 @@ def test_initialization( ) @mock.patch("core.utils.notify_participants") @mock.patch("core.services.room_management.RoomManagement.update_metadata") -def test_handle_egress_ended_success( # noqa: PLR0913, PLR0917 # pylint: disable=too-many-arguments, too-many-positional-arguments - mock_update_metadata, mock_notify, mode, notification_type, service, settings +def test_handle_egress_ended_success( # pylint: disable=too-many-arguments, too-many-positional-arguments + mock_update_metadata, mock_notify, mode, notification_type, service ): """Should successfully stop recording and notifies all participant.""" - settings.RECORDING_STORAGE_EVENT_ENABLE = False recording = RecordingFactory(worker_id="worker-1", mode=mode, status="active") mock_data = mock.MagicMock() mock_data.egress_info.egress_id = recording.worker_id @@ -159,12 +158,11 @@ def test_handle_egress_updated_non_handled( ) @mock.patch("core.utils.notify_participants") @mock.patch("core.services.room_management.RoomManagement.update_metadata") -def test_handle_egress_ended_metadata_update_fails( # noqa: PLR0913, PLR0917 # pylint: disable=too-many-arguments, too-many-positional-arguments - mock_update_metadata, mock_notify, mode, notification_type, service, settings +def test_handle_egress_ended_metadata_update_fails( # pylint: disable=too-many-arguments, too-many-positional-arguments + mock_update_metadata, mock_notify, mode, notification_type, service ): """Should successfully stop and save recording when metadata's update fails.""" - settings.RECORDING_STORAGE_EVENT_ENABLE = False recording = RecordingFactory(worker_id="worker-1", mode=mode, status="active") mock_data = mock.MagicMock() mock_data.egress_info.egress_id = recording.worker_id @@ -358,12 +356,10 @@ def test_handle_egress_ended_finalizes_recording( # noqa: PLR0913, PLR0917 recording_status, egress_status, service, - settings, ): # pylint: disable=too-many-arguments,too-many-positional-arguments """Should notify external services and save the recording on egress completion - (EGRESS_COMPLETE or EGRESS_LIMIT_REACHED) when RECORDING_STORAGE_EVENT_ENABLE is False. + (EGRESS_COMPLETE or EGRESS_LIMIT_REACHED). """ - settings.RECORDING_STORAGE_EVENT_ENABLE = False mock_notify_external_services.return_value = notify_return_value recording = RecordingFactory(worker_id="worker-1", status="active") @@ -379,47 +375,6 @@ def test_handle_egress_ended_finalizes_recording( # noqa: PLR0913, PLR0917 assert recording.status == recording_status -@mock.patch( - "core.recording.services.recording_events.notification_service." - "notify_external_services" -) -@mock.patch("core.utils.notify_participants") -@mock.patch("core.services.room_management.RoomManagement.update_metadata") -@pytest.mark.parametrize( - "egress_status, expected_status", - [ - (EgressStatus.EGRESS_COMPLETE, "active"), - (EgressStatus.EGRESS_LIMIT_REACHED, "stopped"), - ], -) -def test_handle_egress_ended_does_not_finalize_when_webhooks_enabled( # noqa: PLR0913, PLR0917 - mock_update_metadata, - mock_notify, - mock_notify_external_services, - egress_status, - expected_status, - service, - settings, -): # pylint: disable=too-many-arguments,too-many-positional-arguments - """When storage event webhooks are enabled, egress_ended must not finalize the - recording: external services are never notified. EGRESS_LIMIT_REACHED still stops - the recording, EGRESS_COMPLETE leaves it active. - """ - settings.RECORDING_STORAGE_EVENT_ENABLE = True - - recording = RecordingFactory(worker_id="worker-1", status="active") - mock_data = mock.MagicMock() - mock_data.egress_info.egress_id = recording.worker_id - mock_data.egress_info.status = egress_status - - service._handle_egress_ended(mock_data) - - mock_notify_external_services.assert_not_called() - - recording.refresh_from_db() - assert recording.status == expected_status - - @pytest.mark.parametrize( "egress_status", [ @@ -432,10 +387,9 @@ def test_handle_egress_ended_does_not_finalize_when_webhooks_enabled( # noqa: P ) @mock.patch("core.services.room_management.RoomManagement.update_metadata") def test_handle_egress_ended_does_not_save_on_wrong_status( - mock_update_metadata, egress_status, service, settings + mock_update_metadata, egress_status, service ): """Shouldn't save on invalid status.""" - settings.RECORDING_STORAGE_EVENT_ENABLE = False recording = RecordingFactory(worker_id="worker-1", status="active") mock_data = mock.MagicMock() @@ -453,14 +407,13 @@ def test_handle_egress_ended_does_not_save_on_wrong_status( ) @mock.patch("core.services.room_management.RoomManagement.update_metadata") def test_handle_egress_ended_ignores_non_savable_recording( - mock_update_metadata, status, service, settings + mock_update_metadata, status, service ): """Should handle non-savable recordings idempotently without raising. 'egress_ended' may be redelivered (e.g. for an already-saved recording); this must not raise, otherwise the webhook would 500 and LiveKit would retry. """ - settings.RECORDING_STORAGE_EVENT_ENABLE = False recording = RecordingFactory(worker_id="worker-1", status=status) mock_data = mock.MagicMock() diff --git a/src/backend/meet/settings.py b/src/backend/meet/settings.py index d6f1e17c..1a7d32c9 100755 --- a/src/backend/meet/settings.py +++ b/src/backend/meet/settings.py @@ -729,20 +729,6 @@ class Base(Configuration): environ_name="RECORDING_WORKER_CLASSES", environ_prefix=None, ) - RECORDING_EVENT_PARSER_CLASS = values.Value( - "core.recording.event.parsers.MinioParser", - environ_name="RECORDING_EVENT_PARSER_CLASS", - environ_prefix=None, - ) - RECORDING_ENABLE_STORAGE_EVENT_AUTH = values.BooleanValue( - True, environ_name="RECORDING_ENABLE_STORAGE_EVENT_AUTH", environ_prefix=None - ) - RECORDING_STORAGE_EVENT_ENABLE = values.BooleanValue( - False, environ_name="RECORDING_STORAGE_EVENT_ENABLE", environ_prefix=None - ) - RECORDING_STORAGE_EVENT_TOKEN = SecretFileValue( - None, environ_name="RECORDING_STORAGE_EVENT_TOKEN", environ_prefix=None - ) # Number of days before recordings expire - must be synced with bucket lifecycle policy # Set to None for no expiration RECORDING_EXPIRATION_DAYS = values.IntegerValue( diff --git a/src/helm/env.d/common.yaml.gotmpl b/src/helm/env.d/common.yaml.gotmpl index b4687d97..27342910 100644 --- a/src/helm/env.d/common.yaml.gotmpl +++ b/src/helm/env.d/common.yaml.gotmpl @@ -186,8 +186,6 @@ backend: CELERY_BROKER_URL: redis://default:pass@redis-master:6379/1 # Recording & Transcription RECORDING_ENABLE: True - RECORDING_STORAGE_EVENT_ENABLE: True - RECORDING_STORAGE_EVENT_TOKEN: password SUMMARY_SERVICE_ENDPOINT: http://meet-summary:80/api/v2/async-jobs/transcribe/ SUMMARY_SERVICE_API_TOKEN: password SUMMARY_SERVICE_WEBHOOK_API_TOKEN: webhook-password diff --git a/src/helm/extra/templates/minio.yaml b/src/helm/extra/templates/minio.yaml index f07e097d..72091e12 100644 --- a/src/helm/extra/templates/minio.yaml +++ b/src/helm/extra/templates/minio.yaml @@ -113,26 +113,3 @@ spec: exit 0 restartPolicy: Never backoffLimit: 3 ---- -apiVersion: batch/v1 -kind: Job -metadata: - name: minio-webhook -spec: - template: - spec: - containers: - - name: mc - image: minio/mc - command: - - /bin/sh - - -c - - | - /usr/bin/mc alias set meet http://minio:9000 meet password && \ - /usr/bin/mc admin config set meet notify_webhook:meet-webhook endpoint="https://meet.127.0.0.1.nip.io/api/v1.0/recordings/storage-hook/" auth_token="Bearer password" && \ - /usr/bin/mc admin service restart meet --wait --json && \ - sleep 15 && \ - /usr/bin/mc event add meet/meet-media-storage arn:minio:sqs::meet-webhook:webhook --event put --prefix "recordings" && \ - exit 0 - restartPolicy: Never - backoffLimit: 3 diff --git a/src/summary/README.md b/src/summary/README.md index 34246255..e88305ce 100644 --- a/src/summary/README.md +++ b/src/summary/README.md @@ -22,13 +22,6 @@ Configure your env values in `env.d/summary` to properly set up WhisperX and the make run ``` -When the stack is up, configure the MinIO webhook -*(TODO: add this step to `make bootstrap`)* - -```sh -make minio-webhook-setup -``` - If you want to develop on the Celery workers with hot reloading, run: ```sh