Compare commits

..

9 Commits

Author SHA1 Message Date
lebaudantoine 1099815a6e ⚡️(frontend) serve preset virtual backgrounds as WebP
Why: the eight presets were 2048x1152 high-quality JPEG files,
about 1.1MB each, fetched during the meeting when applied. The
camera is published at 1080p at most, so the extra pixels and
quality were never visible.

Re-encode them as 1920x1080 WebP (q80): 8.9MB down to 2.4MB, about
300KB per background. Thumbnails become lossless WebP, pixel
identical, 163KB down to 80KB.

Trade-off: lossy re-encoding of an already lossy source, with no
visible difference at video resolution.

The saved choice is migrated from .jpg to .webp on load. JPEG files
are kept for clients still running the previous bundle, and can be
removed in a later release. WebP is added to the nginx cache rule,
otherwise the new files would be served no-store.
2026-11-15 23:53:47 +01:00
lebaudantoine c5fcf2773c ⚡️(frontend) lazy load the settings dialog
The header settings dialog statically imported room-only code
(livekit-client, @livekit/components-react, posthog-js through the
feature-flag hooks), which pulled it all into the entry chunk.

Load the dialog on demand: fetch it on hover or focus of the settings
button, mount it on first open. Drop its re-export from the settings
barrel, otherwise the dynamic import is ineffective.

Initial JS on the home page goes from 1,652 kB to 644 kB minified,
which shortens FCP for first-time visitors. The join screen saves only
a few kB, since it needs LiveKit anyway.
2026-11-15 23:40:50 +01:00
lebaudantoine da6014022f ⚡️(frontend) preload background effects assets on hover
Why: nothing was fetched before the click applying an effect, so
the first activation waited for the MediaPipe wasm (~9.6MB), the
model (~250KB) and the background (~1MB), often during a meeting.
Hovering an effects entry point or a thumbnail is an early intent
signal: warm the HTTP cache then, the processors reuse it on click.

Trade-off: the download is speculative and may compete with the
meeting media. Only light assets are always preloaded; the wasm is
low priority and skipped on data saver, slow links or poor LiveKit
quality; full-size backgrounds are loaded one at a time, on
thumbnail hover. Assets are cached, so the cost is paid once.

Only the network time is saved: wasm compilation and segmenter
initialisation still happen on click.
2026-11-15 23:25:25 +01:00
lebaudantoine b64249a6a9 ⚡️(frontend) cache mediapipe models served by nginx
The .tflite and .task models live under /assets/mediapipe/models/,
which neither the wasm location nor the static extension list
matches. They fell through to "location /" and were served with
no-store, so the segmentation model was downloaded again each time
a background processor was initialised.

Add a dedicated location with the same 30 days policy as the wasm
files, in both the default and the DINUM nginx configurations.

This saves about 250KB of download on every blur or virtual
background activation after the first one.
2026-11-15 22:45:27 +01:00
lebaudantoine 323899ea0e ⚡️(frontend) preload notification sound on the pre-join screen when idle
The notification sound and the howler chunk were only requested once
the room was mounted, so they competed with the connection setup and
the first notification could fire before the sound was ready.

Load them from the pre-join screen instead, while the user is
checking their camera and microphone. The shared instance introduced
in the previous commit is reused by the room, so nothing is fetched
twice, even if the user joins before the preload has finished.

Add runWhenIdle, which waits for the page load event, then for an
idle period of the main thread (capped at 5 s). Safari has no
requestIdleCallback, so it falls back to a 1 s timer. The task is
cancelled if the user leaves the screen first.

The preload is skipped when every notification sound is disabled or
when the browser reports Save-Data. It is best effort: on failure,
the room loads the sound itself as before.
2026-11-15 22:01:16 +01:00
lebaudantoine 30beb9aa53 ♻️(frontend) replace use-sound with a single shared howler instance
useNotificationSound is called from two components, and use-sound
creates one Howl per hook call. Howler only shares a decoded buffer
once its download has completed, so two instances created at the same
time each fetched notifications.mp3: the file was downloaded twice on
every room join.

Introduce notificationSound.ts, which lazily imports howler and holds
one Howl for the whole app. Whoever calls loadNotificationSound()
first triggers the download, later callers get the same promise. A
failed load clears the promise so a later call can retry.

Behaviour is unchanged: a sound that isn't loaded yet is skipped
rather than played late, and the volume setting is applied on each
play.

use-sound was only used by this hook, so drop it and depend on howler
directly. It was already bundled as use-sound's dependency, at the
same version, so the bundle doesn't grow. Add @types/howler.
2026-11-15 21:55:36 +01:00
lebaudantoine 970ef7ea22 ⚡️(frontend) serve sounds as content-hashed, long-term cached assets
The notification and speaker-test sounds lived in public/sounds, so
they were copied to the build output under a fixed name and served by
the catch-all nginx location with "no-store". Browsers downloaded
them again on every visit: 82 kB for notifications.mp3 and 406 kB for
uprise.mp3.

Move both files to src/assets/sounds and import them, so Vite emits
them under /assets with a content hash in the file name. Add mp3 to
the extensions cached for 30 days under /assets, in default.conf and
in the Helm nginx config map.

The hash only changes when the file content does, so a release that
doesn't touch the sounds keeps them cached.

/sounds/notifications.mp3 and /sounds/uprise.mp3 no longer exist.
Any nginx config maintained outside this repository needs the same
mp3 addition to benefit from the caching.
2026-11-15 21:44:21 +01:00
lebaudantoine 765b5e4aa1 ⚡️(frontend) import RecordingMode from types, not the feature barrel
useConfig is loaded on every screen through the layout. Importing
RecordingMode from the '@/features/recording' barrel also pulled in
the recording side panels it re-exports, and with them livekit-client,
@livekit/components-react and the settings dialog.

Import the enum from its types module so the config hook no longer
ties the entry chunk to room-only code.
2026-11-15 01:07:03 +01:00
lebaudantoine 7dbe67264e wip lazy load feature flag 2026-11-15 00:53:37 +01:00
50 changed files with 666 additions and 70 deletions
+6
View File
@@ -29,6 +29,11 @@ and this project adheres to
- 🍱(frontend) add Portuguese glyphs to Marianne subsetted fonts
- 🚸(frontend) bring new speakers to the top of the carousel
- 🚸(frontend) bring raised hands into view in the carousel
- ⚡️(frontend) preload notification sound on the pre-join screen when idle
- ⚡️(frontend) cache mediapipe models served by nginx
- ⚡️(frontend) preload background effects assets on hover
- ⚡️(frontend) lazy load the settings dialog
- ⚡️(frontend) serve preset virtual backgrounds as WebP
### Fixed
@@ -46,6 +51,7 @@ and this project adheres to
- 🐛(backend) allow muting when everyone_can_mute is stored as null #1942
- 🐛(frontend) apply the sans font token on every page
- 🐛(frontend) promote the active speaker in the carousel layout
- ⚡️(frontend) serve sounds as content-hashed, long-term cached assets
## [1.37.0] - 2026-11-01
+1 -1
View File
@@ -33,7 +33,7 @@ server {
}
# Serve static files with caching
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|mp3)$ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
+6 -1
View File
@@ -71,8 +71,13 @@ server {
add_header Cache-Control "public, max-age=2592000";
}
location ^~ /assets/mediapipe/models/ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
# Serve static files with caching
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|mp3)$ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
+6 -1
View File
@@ -10,8 +10,13 @@ server {
add_header Cache-Control "public, max-age=2592000";
}
location ^~ /assets/mediapipe/models/ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
# Serve static files with caching
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|mp3)$ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
+11 -14
View File
@@ -22,6 +22,7 @@
"@tanstack/react-query": "5.101.4",
"@timephy/rnnoise-wasm": "1.0.0",
"hoofd": "1.7.3",
"howler": "2.2.4",
"humanize-duration": "3.34.1",
"i18next": "26.3.6",
"i18next-browser-languagedetector": "8.2.1",
@@ -35,7 +36,6 @@
"react-dom": "18.3.1",
"react-i18next": "17.0.12",
"react-stately": "3.49.0",
"use-sound": "5.0.0",
"valtio": "2.3.2",
"wouter": "3.10.0"
},
@@ -44,6 +44,7 @@
"@pandacss/dev": "1.11.1",
"@tanstack/eslint-plugin-query": "5.100.14",
"@tanstack/react-query-devtools": "5.100.14",
"@types/howler": "2.2.13",
"@types/humanize-duration": "3.27.4",
"@types/node": "24.12.4",
"@types/react": "18.3.12",
@@ -2555,6 +2556,13 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/howler": {
"version": "2.2.13",
"resolved": "https://registry.npmjs.org/@types/howler/-/howler-2.2.13.tgz",
"integrity": "sha512-40+EBjqIHHrC4VShlz/7i0lBUsE3QkgzZinQQji74Hd8sBkJZUBaT7LWFLK6rcabsDOOQpoMbEJvtaFQwxOu/g==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/humanize-duration": {
"version": "3.27.4",
"resolved": "https://registry.npmjs.org/@types/humanize-duration/-/humanize-duration-3.27.4.tgz",
@@ -6458,7 +6466,8 @@
"node_modules/howler": {
"version": "2.2.4",
"resolved": "https://registry.npmjs.org/howler/-/howler-2.2.4.tgz",
"integrity": "sha512-iARIBPgcQrwtEr+tALF+rapJ8qSc+Set2GJQl7xT1MQzWaVkFebdJhR3alVlSiUf5U7nAANKuj3aWpwerocD5w=="
"integrity": "sha512-iARIBPgcQrwtEr+tALF+rapJ8qSc+Set2GJQl7xT1MQzWaVkFebdJhR3alVlSiUf5U7nAANKuj3aWpwerocD5w==",
"license": "MIT"
},
"node_modules/html-parse-stringify": {
"version": "4.0.1",
@@ -11177,18 +11186,6 @@
"punycode": "^2.1.0"
}
},
"node_modules/use-sound": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/use-sound/-/use-sound-5.0.0.tgz",
"integrity": "sha512-MNHT3FFC5HxNCrgZtrnpIMJI2cw/0D2xismcrtyht8BTuF5FhFhb57xO/jlQr2xJaFrc/0btzRQvGyHQwB7PVA==",
"license": "MIT",
"dependencies": {
"howler": "^2.2.4"
},
"peerDependencies": {
"react": ">=16.8"
}
},
"node_modules/use-sync-external-store": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/use-sync-external-store/-/use-sync-external-store-1.6.0.tgz",
+2 -1
View File
@@ -30,6 +30,7 @@
"@tanstack/react-query": "5.101.4",
"@timephy/rnnoise-wasm": "1.0.0",
"hoofd": "1.7.3",
"howler": "2.2.4",
"humanize-duration": "3.34.1",
"i18next": "26.3.6",
"i18next-browser-languagedetector": "8.2.1",
@@ -43,7 +44,6 @@
"react-dom": "18.3.1",
"react-i18next": "17.0.12",
"react-stately": "3.49.0",
"use-sound": "5.0.0",
"valtio": "2.3.2",
"wouter": "3.10.0"
},
@@ -52,6 +52,7 @@
"@pandacss/dev": "1.11.1",
"@tanstack/eslint-plugin-query": "5.100.14",
"@tanstack/react-query-devtools": "5.100.14",
"@types/howler": "2.2.13",
"@types/humanize-duration": "3.27.4",
"@types/node": "24.12.4",
"@types/react": "18.3.12",
Binary file not shown.

After

Width:  |  Height:  |  Size: 172 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 309 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 394 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 436 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 202 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 222 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 294 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 348 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

+1 -1
View File
@@ -1,7 +1,7 @@
import { fetchApi } from './fetchApi'
import { keys } from './queryKeys'
import { useQuery } from '@tanstack/react-query'
import { RecordingMode } from '@/features/recording'
import type { RecordingMode } from '@/features/recording/types'
import type { ApiAccessLevel } from '@/features/rooms/api/ApiRoom'
import type { Track, VideoCodec } from 'livekit-client'
type Source = Track.Source
+2 -1
View File
@@ -4,6 +4,7 @@ import { useTranslation } from 'react-i18next'
import { useMediaDeviceSelect } from '@livekit/components-react'
import { reportError } from '@/features/analytics/telemetry'
import { canTestAudioOutput } from '@/features/rooms/utils/canTestAudioOutput'
import upriseSoundSrc from '@/assets/sounds/uprise.mp3'
export const SoundTester = () => {
const { t } = useTranslation('settings')
@@ -55,7 +56,7 @@ export const SoundTester = () => {
{/* eslint-disable jsx-a11y/media-has-caption */}
<audio
ref={audioRef}
src="/sounds/uprise.mp3"
src={upriseSoundSrc}
onEnded={() => setIsPlaying(false)}
/>
</>
@@ -0,0 +1,32 @@
import { useEffect, useState } from 'react'
import { getPosthog } from '../utils'
import { useIsAnalyticsEnabled } from './useIsAnalyticsEnabled'
/**
* Same contract as `useFeatureFlagEnabled` from 'posthog-js/react', without a
* static import of posthog-js: that import puts the whole SDK on the critical
* path of every screen, defeating the lazy `getPosthog()`.
*/
export const useFeatureFlagEnabled = (flag: string): boolean | undefined => {
const isAnalyticsEnabled = useIsAnalyticsEnabled()
const [enabled, setEnabled] = useState<boolean | undefined>(undefined)
useEffect(() => {
if (!isAnalyticsEnabled) return
let cancelled = false
let unsubscribe: (() => void) | undefined
void getPosthog().then((ph) => {
if (cancelled) return
setEnabled(ph.isFeatureEnabled(flag))
unsubscribe = ph.onFeatureFlags(() =>
setEnabled(ph.isFeatureEnabled(flag))
)
})
return () => {
cancelled = true
unsubscribe?.()
}
}, [flag, isAnalyticsEnabled])
return enabled
}
@@ -0,0 +1,25 @@
import { useEffect } from 'react'
import { notificationsStore } from '@/stores/notifications'
import { runWhenIdle } from '@/utils/runWhenIdle'
import { loadNotificationSound } from '../notificationSound'
/**
* Fetch and decode the notification sound while the browser is idle, so it
* doesn't compete with the room connection and is ready for the first
* notification. Meant for screens shown before the meeting, e.g. pre-join.
*/
export const usePreloadNotificationSound = () => {
useEffect(() => {
const isAnySoundEnabled = Array.from(
notificationsStore.soundNotifications.values()
).some(Boolean)
const saveData = (navigator as { connection?: { saveData?: boolean } })
.connection?.saveData
if (!isAnySoundEnabled || saveData) return
return runWhenIdle(() => {
// Best effort: the room loads the sound itself if this fails.
loadNotificationSound().catch(() => {})
})
}, [])
}
@@ -1,24 +1,25 @@
import useSound from 'use-sound'
import { useEffect } from 'react'
import { useSnapshot } from 'valtio'
import { notificationsStore } from '@/stores/notifications'
import type { NotificationType } from '@/features/notifications/NotificationType'
import {
loadNotificationSound,
playNotificationSound,
} from '@/features/notifications/notificationSound'
// fixme - handle dynamic audio output changes
export const useNotificationSound = () => {
const notificationsSnap = useSnapshot(notificationsStore)
const [play] = useSound('/sounds/notifications.mp3', {
sprite: {
participantJoined: [0, 1150],
handRaised: [1400, 180],
messageReceived: [1580, 300],
participantWaiting: [2039, 710],
success: [2740, 1304],
},
volume: notificationsSnap.soundNotificationVolume,
})
useEffect(() => {
loadNotificationSound().catch(() => {})
}, [])
const triggerNotificationSound = (type: NotificationType) => {
const isSoundEnabled = notificationsSnap.soundNotifications.get(type)
if (isSoundEnabled) play({ id: type })
if (isSoundEnabled) {
playNotificationSound(type, notificationsSnap.soundNotificationVolume)
}
}
return { triggerNotificationSound }
}
@@ -0,0 +1,66 @@
import type { Howl } from 'howler'
import type { NotificationType } from './NotificationType'
import notificationSoundSrc from '@/assets/sounds/notifications.mp3'
const NOTIFICATION_SOUND_SPRITE: Record<string, [number, number]> = {
participantJoined: [0, 1150],
handRaised: [1400, 180],
messageReceived: [1580, 300],
participantWaiting: [2039, 710],
success: [2740, 1304],
}
let howl: Howl | undefined
let howlPromise: Promise<Howl> | undefined
/**
* Lazily load howler and the notification sound sprite.
*
* The Howl instance is shared by every consumer: whoever calls first triggers
* the (single) download and decoding of the file, later calls reuse it. It can
* thus be called ahead of time, e.g. from the pre-join screen, so the sound is
* ready before the first notification of the meeting.
*/
export const loadNotificationSound = (): Promise<Howl> => {
if (!howlPromise) {
howlPromise = import('howler')
.then(
({ Howl }) =>
new Promise<Howl>((resolve, reject) => {
const instance = new Howl({
src: [notificationSoundSrc],
// The extension is kept by the build, but don't rely on it.
format: ['mp3'],
sprite: NOTIFICATION_SOUND_SPRITE,
onload: () => {
howl = instance
resolve(instance)
},
onloaderror: (_id, error) => {
instance.unload()
reject(new Error(`Notification sound failed to load: ${error}`))
},
})
})
)
.catch((error) => {
// Let a later call retry.
howlPromise = undefined
throw error
})
}
return howlPromise
}
/**
* Play a notification sound, if the sprite is already loaded. A sound which
* isn't ready is skipped rather than played late.
*/
export const playNotificationSound = (
type: NotificationType,
volume: number
) => {
if (!howl) return
howl.volume(volume)
howl.play(type)
}
@@ -1,14 +1,18 @@
import React from 'react'
import { useTranslation } from 'react-i18next'
import { useSidePanel } from '@/features/rooms/livekit/hooks/useSidePanel'
import { useEffectsPreload } from '@/features/rooms/livekit/hooks/useEffectsPreload'
import { Button } from '@/primitives'
import { RiImageCircleAiFill } from '@remixicon/react'
export const EffectsButton = React.memo(() => {
const { t } = useTranslation('rooms', { keyPrefix: 'participantTileFocus' })
const { isEffectsOpen, toggleEffects } = useSidePanel()
const { hoverProps, focusProps } = useEffectsPreload()
return (
<Button
{...hoverProps}
{...focusProps}
size={'sm'}
variant={'primaryTextDark'}
square
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import type { RecordingMode } from '../types'
import { useIsRecordingModeEnabled } from './useIsRecordingModeEnabled'
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import type { RecordingMode } from '../types'
import { useIsRecordingModeEnabled } from './useIsRecordingModeEnabled'
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import { FeatureFlags } from '@/features/analytics/enums'
@@ -15,6 +15,7 @@ import {
import { SelectDevice } from '../livekit/components/controls/Device/SelectDevice'
import { ToggleDevice } from '../livekit/components/controls/Device/ToggleDevice'
import { BackgroundProcessorFactory } from '../livekit/components/blur'
import { useEffectsPreload } from '../livekit/hooks/useEffectsPreload'
import { Lobby } from './Lobby'
import { openPermissionsDialog } from '@/stores/permissions'
import { isSafari } from '@/utils/livekit'
@@ -33,6 +34,7 @@ import { useDeviceMissing } from '../livekit/hooks/useDeviceMissing'
import { useJoinTracks } from '../livekit/hooks/useJoinTracks'
import { SilentMicDetector } from './SilentMicDetector'
import { type PreviewTrack, toggleTrack } from '../utils/toggleTrack'
import { usePreloadNotificationSound } from '@/features/notifications/hooks/usePreloadNotificationSound'
const styles = {
page: css({
@@ -272,6 +274,7 @@ const Effects = ({
}: Pick<EffectsConfigurationProps, 'videoTrack'>) => {
const { t } = useTranslation('rooms', { keyPrefix: 'join.effects' })
const [isDialogOpen, setIsDialogOpen] = useState(false)
const { hoverProps, focusProps } = useEffectsPreload()
if (!BackgroundProcessorFactory.isSupported() || isMobileBrowser()) {
return null
@@ -299,6 +302,8 @@ const Effects = ({
onPress={() => setIsDialogOpen(true)}
tooltip={t('description')}
aria-label={t('description')}
{...hoverProps}
{...focusProps}
>
<RiImageCircleAiFill size={24} />
</Button>
@@ -458,6 +463,7 @@ export const Join = ({
roomId: string
}) => {
const { audioTrack, videoTrack } = useJoinTracks()
usePreloadNotificationSound()
return (
<Screen footer={false}>
@@ -0,0 +1,147 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const MODEL = '/assets/mediapipe/models/selfie_segmenter_landscape.tflite'
const WASM_LOADER = '/wasm/vision_wasm_internal.js'
const WASM_BINARY = '/wasm/vision_wasm_internal.wasm'
const isSupported = vi.fn(() => true)
vi.mock('.', () => ({
BackgroundProcessorFactory: { isSupported: () => isSupported() },
MEDIAPIPE_PATH_WASM: '/wasm',
SELFIE_SEGMENTER_MODEL_PATH: MODEL,
}))
vi.mock('@mediapipe/tasks-vision', () => ({
FilesetResolver: {
forVisionTasks: async () => ({
wasmLoaderPath: WASM_LOADER,
wasmBinaryPath: WASM_BINARY,
}),
},
}))
vi.mock('@/features/files/utils/resolveMediaUrl', () => ({
resolveMediaUrlOrFallback: async (url: string) => `resolved:${url}`,
}))
class FakeImage {
static instances: FakeImage[] = []
src = ''
crossOrigin: string | null = null
onerror: (() => void) | null = null
constructor() {
FakeImage.instances.push(this)
}
}
const fetchMock = vi.fn()
const fetchedUrls = () => fetchMock.mock.calls.map(([url]) => url)
const imageUrls = () => FakeImage.instances.map((image) => image.src)
// Let the fire-and-forget preloads settle.
const flush = () => new Promise((resolve) => setTimeout(resolve))
const load = () => import('./preload')
beforeEach(() => {
vi.resetModules()
isSupported.mockReturnValue(true)
FakeImage.instances = []
fetchMock.mockReset()
fetchMock.mockImplementation(async () => ({ ok: true, body: null }))
vi.stubGlobal('fetch', fetchMock)
vi.stubGlobal('Image', FakeImage)
vi.stubGlobal('navigator', {})
})
describe('preloadEffectsAssets', () => {
it('preloads the thumbnails, the model and the wasm', async () => {
const { preloadEffectsAssets } = await load()
preloadEffectsAssets()
await flush()
expect(imageUrls()).toHaveLength(8)
expect(imageUrls()[0]).toBe('/assets/backgrounds/thumbnails/1.webp')
expect(fetchedUrls().sort()).toEqual([MODEL, WASM_LOADER, WASM_BINARY])
expect(fetchMock).toHaveBeenCalledWith(WASM_BINARY, { priority: 'low' })
})
it('preloads only once', async () => {
const { preloadEffectsAssets } = await load()
preloadEffectsAssets()
await flush()
preloadEffectsAssets()
await flush()
expect(imageUrls()).toHaveLength(8)
expect(fetchMock).toHaveBeenCalledTimes(3)
})
it('skips the wasm when the caller reports a degraded connection', async () => {
const { preloadEffectsAssets } = await load()
preloadEffectsAssets({ isConstrained: true })
await flush()
expect(imageUrls()).toHaveLength(8)
expect(fetchedUrls()).toEqual([MODEL])
})
it.each([{ saveData: true }, { effectiveType: '3g' }])(
'skips the wasm when the browser reports %o',
async (connection) => {
vi.stubGlobal('navigator', { connection })
const { preloadEffectsAssets } = await load()
preloadEffectsAssets()
await flush()
expect(fetchedUrls()).toEqual([MODEL])
}
)
it('retries after a failure', async () => {
vi.spyOn(console, 'warn').mockImplementation(() => {})
fetchMock.mockImplementation(async () => ({ ok: false, status: 503 }))
const { preloadEffectsAssets } = await load()
preloadEffectsAssets({ isConstrained: true })
await flush()
preloadEffectsAssets({ isConstrained: true })
await flush()
expect(fetchedUrls()).toEqual([MODEL, MODEL])
})
it('does nothing when background processors are not supported', async () => {
isSupported.mockReturnValue(false)
const { preloadEffectsAssets } = await load()
preloadEffectsAssets()
await flush()
expect(imageUrls()).toHaveLength(0)
expect(fetchMock).not.toHaveBeenCalled()
})
})
describe('preloadBackgroundImage', () => {
it('loads the resolved image once, in CORS mode', async () => {
const { preloadBackgroundImage } = await load()
await preloadBackgroundImage('/assets/backgrounds/1.webp')
await preloadBackgroundImage('/assets/backgrounds/1.webp')
expect(FakeImage.instances).toHaveLength(1)
expect(FakeImage.instances[0].src).toBe(
'resolved:/assets/backgrounds/1.webp'
)
expect(FakeImage.instances[0].crossOrigin).toBe('anonymous')
})
it('skips local files and degraded connections', async () => {
const { preloadBackgroundImage } = await load()
await preloadBackgroundImage('blob:https://meet.example.com/1234')
await preloadBackgroundImage('/assets/backgrounds/1.webp', {
isConstrained: true,
})
expect(FakeImage.instances).toHaveLength(0)
})
})
@@ -0,0 +1,148 @@
import { FilesetResolver } from '@mediapipe/tasks-vision'
import { resolveMediaUrlOrFallback } from '@/features/files/utils/resolveMediaUrl'
import {
BackgroundProcessorFactory,
MEDIAPIPE_PATH_WASM,
SELFIE_SEGMENTER_MODEL_PATH,
} from '.'
/**
* Warm the browser caches with what a background effect needs, ahead of
* the click that applies it. Nothing is instantiated here: the processors
* issue the very same requests later and find them in the HTTP cache.
*
* A meeting may be running, so what is preloaded is arbitrated by cost:
*
* | Asset | Size | Preloaded |
* | ----------------------- | --------- | --------------------------------- |
* | Preset thumbnails | ~80KB | always |
* | Segmentation model | ~250KB | always |
* | MediaPipe wasm + loader | ~9.6MB | low priority, unless constrained |
* | One background image | ~300KB | on thumbnail hover, unless const. |
* | All background images | ~2.4MB | never |
*/
export const PRESET_BACKGROUNDS_COUNT = 8
export const getPresetBackgroundPaths = (index: number) => ({
imagePath: `/assets/backgrounds/${index + 1}.webp`,
thumbnailPath: `/assets/backgrounds/thumbnails/${index + 1}.webp`,
})
type NetworkInformation = {
saveData?: boolean
effectiveType?: string
}
/**
* Data saver or a slow link, as reported by the browser (Chromium only,
* other browsers are considered unconstrained).
*/
const isNetworkConstrained = () => {
const connection = (navigator as { connection?: NetworkInformation })
.connection
if (!connection) return false
return (
connection.saveData === true ||
['slow-2g', '2g', '3g'].includes(connection.effectiveType ?? '')
)
}
/**
* Download a same-origin asset into the HTTP cache. The body is read to
* completion, as an aborted response is not stored, and discarded chunk
* by chunk to avoid buffering several megabytes.
*/
const warmHttpCache = async (url: string, priority: RequestPriority) => {
const response = await fetch(url, { priority })
if (!response.ok) {
throw new Error(`Failed to preload ${url}: HTTP ${response.status}`)
}
const reader = response.body?.getReader()
if (!reader) return
while (!(await reader.read()).done) {
// discard
}
}
// Held so that in-flight and decoded images are not garbage collected,
// and can be reused from the memory cache. Bounded by the presets and
// the user's custom backgrounds.
const preloadedImages = new Map<string, HTMLImageElement>()
const preloadImage = (src: string, key = src, crossOrigin?: 'anonymous') => {
if (preloadedImages.has(key)) return
const image = new Image()
if (crossOrigin) image.crossOrigin = crossOrigin
image.onerror = () => preloadedImages.delete(key)
image.src = src
preloadedImages.set(key, image)
}
// Run each step once, and allow another attempt after a failure.
const once = (run: () => Promise<unknown>) => {
let pending: Promise<void> | undefined
return () => {
pending ??= run().then(
() => undefined,
(error) => {
console.warn(error)
pending = undefined
}
)
return pending
}
}
const preloadModel = once(() =>
warmHttpCache(SELFIE_SEGMENTER_MODEL_PATH, 'auto')
)
const preloadWasm = once(async () => {
// Resolves to the SIMD or non-SIMD build, as the processors will.
const { wasmLoaderPath, wasmBinaryPath } =
await FilesetResolver.forVisionTasks(MEDIAPIPE_PATH_WASM)
await Promise.all([
warmHttpCache(wasmLoaderPath, 'low'),
warmHttpCache(wasmBinaryPath, 'low'),
])
})
/**
* Preload what any background effect needs. Meant to be called when the
* user shows an intent to open the effects (hover or focus of an entry
* point).
*
* @param isConstrained caller's knowledge of a degraded connection (e.g.
* LiveKit connection quality): only the light assets are then preloaded.
*/
export const preloadEffectsAssets = ({ isConstrained = false } = {}) => {
if (!BackgroundProcessorFactory.isSupported()) return
for (let index = 0; index < PRESET_BACKGROUNDS_COUNT; index++) {
preloadImage(getPresetBackgroundPaths(index).thumbnailPath)
}
void preloadModel()
if (isConstrained || isNetworkConstrained()) return
void preloadWasm()
}
/**
* Preload a single full-size background. Meant to be called when the
* user dwells on its thumbnail.
*/
export const preloadBackgroundImage = async (
imagePath: string,
{ isConstrained = false } = {}
) => {
// Local files (not logged-in users) are already in memory.
if (imagePath.startsWith('blob:')) return
if (isConstrained || isNetworkConstrained()) return
if (preloadedImages.has(imagePath)) return
// Same resolution and CORS mode as the processors' own image load, so
// that the cached response can be reused.
const src = await resolveMediaUrlOrFallback(imagePath)
preloadImage(src, imagePath, 'anonymous')
}
@@ -5,6 +5,7 @@ import { styled } from '@/styled-system/jsx'
import { Button } from '@/primitives'
import { canTestAudioOutput } from '@/features/rooms/utils/canTestAudioOutput'
import { resolveAudioOutputDeviceId } from '@/features/rooms/utils/resolveAudioOutputDeviceId'
import upriseSoundSrc from '@/assets/sounds/uprise.mp3'
// Speaker test in the audiooutput menu footer (Meet-style UX). Outputs have
// no track: the test plays a bundled file through the selected sink, and
@@ -129,7 +130,7 @@ export const OutputSoundTester = ({
{/* eslint-disable-next-line jsx-a11y/media-has-caption */}
<audio
ref={audioRef}
src="/sounds/uprise.mp3"
src={upriseSoundSrc}
onEnded={() => setIsPlaying(false)}
/>
</StyledContainer>
@@ -9,6 +9,7 @@ import { css } from '@/styled-system/css'
import { useCanPublishTrack } from '../../../hooks/useCanPublishTrack'
import { useCannotUseDevice } from '../../../hooks/useCannotUseDevice'
import { useSidePanel } from '../../../hooks/useSidePanel'
import { useEffectsPreload } from '../../../hooks/useEffectsPreload'
import { BackgroundProcessorFactory } from '../../blur'
import Source = Track.Source
import * as React from 'react'
@@ -27,10 +28,13 @@ import {
const EffectsButton = ({ onPress }: { onPress: () => void }) => {
const { t } = useTranslation('rooms', { keyPrefix: 'selectDevice' })
const { isEffectsOpen, toggleEffects } = useSidePanel()
const { hoverProps, focusProps } = useEffectsPreload()
return (
<Button
size="sm"
square
{...hoverProps}
{...focusProps}
tooltip={t('effects')}
aria-label={t('effects')}
variant="primaryDark"
@@ -3,14 +3,17 @@ import { MenuItem } from 'react-aria-components'
import { useTranslation } from 'react-i18next'
import { menuRecipe } from '@/primitives/menuRecipe'
import { useSidePanel } from '../../../hooks/useSidePanel'
import { useEffectsPreload } from '../../../hooks/useEffectsPreload'
export const EffectsMenuItem = () => {
const { t } = useTranslation('rooms', { keyPrefix: 'options.items' })
const { toggleEffects } = useSidePanel()
const { hoverProps } = useEffectsPreload()
return (
<MenuItem
onAction={() => toggleEffects()}
{...hoverProps}
className={menuRecipe({ icon: true, variant: 'dark' }).item}
>
<RiImageCircleAiFill size={20} />
@@ -7,6 +7,11 @@ import {
ProcessorConfig,
ProcessorType,
} from '../blur'
import {
getPresetBackgroundPaths,
PRESET_BACKGROUNDS_COUNT,
} from '../blur/preload'
import { useEffectsPreload } from '../../hooks/useEffectsPreload'
import { css } from '@/styled-system/css'
import { useResolvedMediaUrls } from '@/features/files/hooks/useResolvedMediaUrls'
import { Button, Dialog, H, P, Text, ToggleButton } from '@/primitives'
@@ -116,6 +121,8 @@ export const EffectsConfiguration = ({
> | null>(null)
const effectAnnouncementId = useRef(0)
const { preloadBackground, cancel: cancelPreload } = useEffectsPreload()
const { processorConfig } = useSnapshot(userChoicesStore)
const selectedId = useMemo(
@@ -146,7 +153,7 @@ export const EffectsConfiguration = ({
const getVirtualBackgroundName = useCallback(
(imagePath?: string) => {
if (!imagePath) return ''
const match = imagePath.match(/\/backgrounds\/(\d+)\.jpg$/)
const match = imagePath.match(/\/backgrounds\/(\d+)\.webp$/)
if (!match) return ''
const index = Number(match[1]) - 1
if (Number.isNaN(index)) return ''
@@ -405,6 +412,7 @@ export const EffectsConfiguration = ({
tooltip: string
ariaLabel: string
thumbnailPath: string
imagePath: string
index: number
}[]
remoteCustomVirtualBackgrounds: {
@@ -450,31 +458,33 @@ export const EffectsConfiguration = ({
config,
}
}),
virtualBackgrounds: [...Array(8).keys()].map((index) => {
const imagePath = `/assets/backgrounds/${index + 1}.jpg`
const thumbnailPath = `/assets/backgrounds/thumbnails/${index + 1}.jpg`
const config: ProcessorConfig = {
type: ProcessorType.VIRTUAL,
imagePath,
}
const id = deriveIdFromProcessorConfig(config)
const isSelected = selectedId === id
const backgroundName = t(`virtual.presets.descriptions.${index}`)
const ariaLabelPrefix = t(
isSelected ? `virtual.selectedLabel` : `virtual.apply`
)
const ariaLabel = `${ariaLabelPrefix} ${backgroundName}`
virtualBackgrounds: [...Array(PRESET_BACKGROUNDS_COUNT).keys()].map(
(index) => {
const { imagePath, thumbnailPath } = getPresetBackgroundPaths(index)
const config: ProcessorConfig = {
type: ProcessorType.VIRTUAL,
imagePath,
}
const id = deriveIdFromProcessorConfig(config)
const isSelected = selectedId === id
const backgroundName = t(`virtual.presets.descriptions.${index}`)
const ariaLabelPrefix = t(
isSelected ? `virtual.selectedLabel` : `virtual.apply`
)
const ariaLabel = `${ariaLabelPrefix} ${backgroundName}`
return {
tooltip: backgroundName,
id,
config,
isSelected: selectedId === id,
thumbnailPath,
ariaLabel,
index,
return {
tooltip: backgroundName,
id,
config,
isSelected: selectedId === id,
thumbnailPath,
imagePath,
ariaLabel,
index,
}
}
}),
),
remoteCustomVirtualBackgrounds: (filesQ.data?.results ?? [])
.filter((file) => file.url)
.map((file) => {
@@ -761,6 +771,10 @@ export const EffectsConfiguration = ({
onChange={getHandleSelectChangeFile(
option.file
)}
onHoverStart={() =>
preloadBackground(option.file.url!)
}
onHoverEnd={cancelPreload}
isSelected={option.isSelected}
className={css({
bgSize: 'cover',
@@ -921,6 +935,10 @@ export const EffectsConfiguration = ({
aria-label={option.ariaLabel}
isDisabled={processorOptions.isDisabled}
onChange={() => toggleEffect(option.config)}
onHoverStart={() =>
preloadBackground(option.imagePath)
}
onHoverEnd={cancelPreload}
isSelected={option.isSelected}
className={css({
bgSize: 'cover',
@@ -0,0 +1,63 @@
import { useCallback, useEffect, useMemo, useRef } from 'react'
import { useMaybeRoomContext } from '@livekit/components-react'
import { ConnectionQuality } from 'livekit-client'
import {
preloadBackgroundImage,
preloadEffectsAssets,
} from '../components/blur/preload'
// Time the pointer must stay on an element before preloading, so that
// sweeping over a menu or a row of thumbnails does not trigger anything.
const HOVER_INTENT_DELAY_MS = 150
/**
* Preload the background effects assets when the user shows an intent to
* use them, to save the download time once they actually click.
*
* In a meeting, the heavy assets are skipped when the connection quality
* is degraded, to avoid competing with the media streams.
*/
export const useEffectsPreload = () => {
const room = useMaybeRoomContext()
const timeout = useRef<ReturnType<typeof setTimeout>>()
const cancel = useCallback(() => clearTimeout(timeout.current), [])
useEffect(() => cancel, [cancel])
const schedule = useCallback(
(preload: (options: { isConstrained: boolean }) => void) => {
cancel()
timeout.current = setTimeout(() => {
// Read when firing: no need to re-render on quality changes.
const quality = room?.localParticipant.connectionQuality
preload({
isConstrained:
quality === ConnectionQuality.Poor ||
quality === ConnectionQuality.Lost,
})
}, HOVER_INTENT_DELAY_MS)
},
[cancel, room]
)
const preloadEffects = useCallback(
() => schedule(preloadEffectsAssets),
[schedule]
)
const preloadBackground = useCallback(
(imagePath: string) =>
schedule((options) => void preloadBackgroundImage(imagePath, options)),
[schedule]
)
return useMemo(
() => ({
hoverProps: { onHoverStart: preloadEffects, onHoverEnd: cancel },
focusProps: { onFocus: preloadEffects, onBlur: cancel },
preloadBackground,
cancel,
}),
[cancel, preloadBackground, preloadEffects]
)
}
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import { FeatureFlags } from '@/features/analytics/enums'
import useKonami from '@/features/rooms/livekit/hooks/useKonami'
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { FeatureFlags } from '@/features/analytics/enums'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import { isMobileBrowser } from '@livekit/components-core'
@@ -1,22 +1,38 @@
import { lazy, Suspense, useState } from 'react'
import { useTranslation } from 'react-i18next'
import { DialogTrigger } from 'react-aria-components'
import { RiSettings3Line } from '@remixicon/react'
import { Button } from '@/primitives'
import { SettingsDialog } from './SettingsDialog'
const loadSettingsDialog = () => import('./SettingsDialog')
const SettingsDialog = lazy(() =>
loadSettingsDialog().then((m) => ({ default: m.SettingsDialog }))
)
export const SettingsButton = () => {
const { t } = useTranslation('settings')
const [hasOpened, setHasOpened] = useState(false)
const [isOpen, setIsOpen] = useState(false)
return (
<DialogTrigger>
<>
<Button
square
variant="secondaryText"
aria-label={t('settingsButtonLabel')}
tooltip={t('settingsButtonLabel')}
onHoverStart={() => void loadSettingsDialog()}
onFocus={() => void loadSettingsDialog()}
onPress={() => {
setHasOpened(true)
setIsOpen(true)
}}
>
<RiSettings3Line />
</Button>
<SettingsDialog />
</DialogTrigger>
{hasOpened && (
<Suspense fallback={null}>
<SettingsDialog isOpen={isOpen} onOpenChange={setIsOpen} />
</Suspense>
)}
</>
)
}
@@ -1,5 +1,4 @@
export { SettingsButton } from './components/SettingsButton'
export { SettingsDialog } from './components/SettingsDialog'
export { useTranscriptionLanguage } from './hook/useTranscriptionLanguage'
@@ -1,4 +1,4 @@
import { useFeatureFlagEnabled } from 'posthog-js/react'
import { useFeatureFlagEnabled } from '@/features/analytics/hooks/useFeatureFlagEnabled'
import { FeatureFlags } from '@/features/analytics/enums'
import { useIsAnalyticsEnabled } from '@/features/analytics/hooks/useIsAnalyticsEnabled'
import { useConfig } from '@/api/useConfig'
+9
View File
@@ -52,8 +52,17 @@ subscribe(userChoicesStore, () => {
saveUserChoices(userChoicesStore as LocalUserChoicesLK, false)
})
// Preset backgrounds used to be JPEG files: point the saved choice to their
// WebP replacement. The JPEG files can be removed once this has shipped.
const LEGACY_PRESET_BACKGROUND = /^(\/assets\/backgrounds\/\d+)\.jpg$/
// we run some logic on store loading to check if the processor config is still valid
if (userChoicesStore.processorConfig?.type === ProcessorType.VIRTUAL) {
userChoicesStore.processorConfig.imagePath =
userChoicesStore.processorConfig.imagePath.replace(
LEGACY_PRESET_BACKGROUND,
'$1.webp'
)
if (userChoicesStore.processorConfig.imagePath.startsWith('blob:')) {
// this happens when a not authenticated user had changed their background image
// we restore clear the processor config to avoid displaying a black screen.
+34
View File
@@ -0,0 +1,34 @@
/**
* Run a non-urgent task once the page has finished loading and the main
* thread is idle. Returns a function cancelling the task if it didn't run yet.
*
* @param task
* @param timeout max delay (ms) to wait for an idle period once the page is loaded
*/
export const runWhenIdle = (task: () => void, timeout: number = 5000) => {
let cancelSchedule: (() => void) | undefined
const schedule = () => {
// Safari doesn't support requestIdleCallback
if (typeof window.requestIdleCallback === 'function') {
const handle = window.requestIdleCallback(task, { timeout })
cancelSchedule = () => window.cancelIdleCallback(handle)
} else {
const handle = window.setTimeout(task, 1000)
cancelSchedule = () => window.clearTimeout(handle)
}
}
// Initial resources (scripts, styles, fonts...) are still being fetched
// until the load event.
if (document.readyState === 'complete') {
schedule()
} else {
window.addEventListener('load', schedule, { once: true })
}
return () => {
window.removeEventListener('load', schedule)
cancelSchedule?.()
}
}
@@ -79,8 +79,13 @@ data:
add_header Cache-Control "public, max-age=2592000";
}
location ^~ /assets/mediapipe/models/ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}
# Serve static files with caching
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
location ~* ^/assets/.*\.(css|js|json|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|mp3)$ {
expires 30d;
add_header Cache-Control "public, max-age=2592000";
}