temetro/.agents/skills/ai-elements/references/speech-input.md

Speech Input

A button component that captures voice input and converts it to text, with cross-browser support.

The SpeechInput component provides an easy-to-use interface for capturing voice input in your application. It uses the Web Speech API for real-time transcription in supported browsers (Chrome, Edge), and falls back to MediaRecorder with an external transcription service for browsers that don't support Web Speech API (Firefox, Safari).

See scripts/speech-input.tsx for this example.

Installation

npx ai-elements@latest add speech-input

Features

• Built on Web Speech API (SpeechRecognition) with MediaRecorder fallback
• Cross-browser support (Chrome, Edge, Firefox, Safari)
• Continuous speech recognition with interim results
• Visual feedback with pulse animation when listening
• Loading state during transcription processing
• Automatic browser compatibility detection
• Final transcript extraction and callbacks
• Error handling and automatic state management
• Extends shadcn/ui Button component
• Full TypeScript support

Props

<SpeechInput />

The component extends the shadcn/ui Button component, so all Button props are available.

Prop	Type	Default	Description
onTranscriptionChange	(text: string) => void	-	Callback fired when final transcription text is available. Only fires for completed phrases, not interim results.
onAudioRecorded	(audioBlob: Blob) => Promise<string>	-	Callback for MediaRecorder fallback. Required for Firefox/Safari support. Receives recorded audio blob and should return transcribed text from an external service (e.g., OpenAI Whisper).
lang	string	-	Language for speech recognition.
...props	React.ComponentProps<typeof Button>	-	Any other props are spread to the Button component, including variant, size, disabled, etc.

Behavior

Speech Recognition Modes

The component automatically detects browser capabilities and uses the best available method:

Browser	Mode	Behavior
Chrome, Edge	Web Speech API	Real-time transcription, no server required
Firefox, Safari	MediaRecorder	Records audio, sends to external transcription service
Unsupported	Disabled	Button is disabled

Web Speech API Mode (Chrome, Edge)

Uses the Web Speech API with the following configuration:

• Continuous: Set to true to keep recognition active until manually stopped
• Interim Results: Set to true to receive partial results during speech
• Language: Configurable via lang prop, defaults to "en-US"

MediaRecorder Mode (Firefox, Safari)

When the Web Speech API is unavailable, the component falls back to recording audio:

1. Records audio using MediaRecorder API
2. On stop, creates an audio blob (audio/webm)
3. Calls onAudioRecorded with the blob
4. Waits for transcription result
5. Passes result to onTranscriptionChange

Note: The onAudioRecorded prop is required for this mode to work. Without it, the button will be disabled in Firefox/Safari.

Transcription Processing

The component only calls onTranscriptionChange with final transcripts. Interim results (Web Speech API) are ignored to prevent incomplete text from being processed.

Visual States

• Default State: Standard button appearance with microphone icon
• Listening State: Pulsing animation with accent colors to indicate active listening
• Processing State: Loading spinner while waiting for transcription (MediaRecorder mode)
• Disabled State: Button is disabled when no API is available or required props are missing

Lifecycle

1. Mount: Detects available APIs and initializes appropriate mode
2. Click: Toggles between listening/recording and stopped states
3. Stop (MediaRecorder): Processes audio and waits for transcription
4. Unmount: Stops recognition/recording and releases microphone

Browser Support

The component provides cross-browser support through a two-tier system:

Browser	API Used	Requirements
Chrome	Web Speech API	None
Edge	Web Speech API	None
Firefox	MediaRecorder	onAudioRecorded prop
Safari	MediaRecorder	onAudioRecorded prop

For full cross-browser support, provide the onAudioRecorded callback that sends audio to a transcription service like OpenAI Whisper, Google Cloud Speech-to-Text, or AssemblyAI.

Accessibility

• Uses semantic button element via shadcn/ui Button
• Visual feedback for listening state
• Keyboard accessible (can be triggered with Space/Enter)
• Screen reader friendly with proper button semantics

Usage with MediaRecorder Fallback

To support Firefox and Safari, provide an onAudioRecorded callback that sends audio to a transcription service:

const handleAudioRecorded = async (audioBlob: Blob): Promise<string> => {
  const formData = new FormData();
  formData.append("file", audioBlob, "audio.webm");
  formData.append("model", "whisper-1");

  const response = await fetch(
    "https://api.openai.com/v1/audio/transcriptions",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      },
      body: formData,
    }
  );

  const data = await response.json();
  return data.text;
};

<SpeechInput
  onTranscriptionChange={(text) => console.log(text)}
  onAudioRecorded={handleAudioRecorded}
/>;

Notes

• Requires a secure context (HTTPS or localhost)
• Browser may prompt user for microphone permission
• Only final transcripts trigger the onTranscriptionChange callback
• Language is configurable via the lang prop
• Continuous recognition continues until button is clicked again
• Errors are logged to console and automatically stop recognition/recording
• MediaRecorder fallback requires the onAudioRecorded prop to be provided
• Audio is recorded in audio/webm format for the MediaRecorder fallback

TypeScript

The component includes full TypeScript definitions for the Web Speech API:

• SpeechRecognition
• SpeechRecognitionEvent
• SpeechRecognitionResult
• SpeechRecognitionAlternative
• SpeechRecognitionErrorEvent

These types are properly declared for both standard and webkit-prefixed implementations.