Adding TTS Audio to a Feature

This skill walks you through adding text-to-speech audio to a feature in the app. The TTS system plays audio via a voice chain (pregenerated mp3 → on-demand generation via OpenAI → browser SpeechSynthesis → subtitles). Clips are collected at runtime, persisted to the database, and generated as high-quality OpenAI TTS mp3s — either on-the-fly during playback (if the generate chain entry is configured) or in batch from the admin panel.

Before You Start

Read the integration guide: apps/web/.claude/reference/tts-audio-system.md

It contains the full API reference, patterns, anti-patterns, and existing implementations.

Your Job

1. Understand what text the feature needs spoken and when
2. Create a feature-specific audio hook
3. Wire it into the component
4. Verify with TypeScript

Step 1: Design the Utterances

For each piece of audio the feature needs, determine:

• What text to speak (static string or dynamic from state/props)
• When to speak it (on mount, on state change, on user action, on completion)
• How it should sound (tone — write as voice-actor stage directions)

Step 2: Create a Feature Audio Hook

Create a hook in the feature's hooks/ directory. This hook owns text construction, tone strings, auto-play logic, and cleanup.

Read the reference implementation first:

apps/web/src/components/practice/hooks/usePracticeAudioHelp.ts

Key rules:

1. Tone strings must be module-level constants — never compute them dynamically per render
2. Always clean up on unmount — call stop() in a cleanup effect
3. Use refs to track previous values — prevents re-playing on every render
4. Guard with isEnabled — respect the user's audio toggle

Template:

typescript
1'use client'
2
3import { useEffect, useRef } from 'react'
4import { useTTS } from '@/hooks/useTTS'
5import { useAudioManager } from '@/hooks/useAudioManager'
6
7// Stable tone constants — changing these creates new clips
8const INSTRUCTION_TONE =
9  'Patiently guiding a young child. Clear, slow, friendly.'
10const CELEBRATION_TONE =
11  'Warmly congratulating a child. Genuinely encouraging and happy.'
12
13interface UseMyFeatureAudioHelpOptions {
14  currentStep: string
15  isComplete: boolean
16}
17
18export function useMyFeatureAudioHelp({
19  currentStep,
20  isComplete,
21}: UseMyFeatureAudioHelpOptions) {
22  const { isEnabled, stop } = useAudioManager()
23
24  // Declare utterances
25  const sayInstruction = useTTS(currentStep, { tone: INSTRUCTION_TONE })
26  const sayCelebration = useTTS(
27    isComplete ? 'Well done!' : '',
28    { tone: CELEBRATION_TONE },
29  )
30
31  // Auto-play when step changes
32  const prevStepRef = useRef<string>('')
33  useEffect(() => {
34    if (!isEnabled || !currentStep || currentStep === prevStepRef.current) return
35    prevStepRef.current = currentStep
36    sayInstruction()
37  }, [isEnabled, currentStep, sayInstruction])
38
39  // Auto-play celebration on completion
40  useEffect(() => {
41    if (!isEnabled || !isComplete) return
42    sayCelebration()
43  }, [isEnabled, isComplete, sayCelebration])
44
45  // Stop audio on unmount
46  useEffect(() => {
47    return () => stop()
48  }, [stop])
49
50  return { replay: sayInstruction }
51}

Step 3: Wire Into the Component

typescript
1import { useMyFeatureAudioHelp } from './hooks/useMyFeatureAudioHelp'
2import { useAudioManager } from '@/hooks/useAudioManager'
3
4function MyFeature() {
5  const { isEnabled, isPlaying } = useAudioManager()
6  const { replay } = useMyFeatureAudioHelp({
7    currentStep: 'Tap the bead to move it up',
8    isComplete: false,
9  })
10
11  return (
12    <div>
13      {isEnabled && (
14        <button onClick={replay} disabled={isPlaying}>
15          {isPlaying ? 'Speaking...' : 'Replay'}
16        </button>
17      )}
18    </div>
19  )
20}

Step 4: Verify

bash
1cd apps/web && npx tsc --noEmit

Common Patterns

Dynamic text from state

typescript
1const text = useMemo(
2  () => (terms ? termsToSentence(terms) : ''),
3  [terms],
4)
5const sayProblem = useTTS(text, { tone: MATH_TONE })

One-shot playback (play once, don't repeat)

typescript
1const playedRef = useRef(false)
2useEffect(() => {
3  if (!shouldPlay || playedRef.current) return
4  playedRef.current = true
5  sayIt()
6}, [shouldPlay, sayIt])
7
8// Reset when trigger resets
9useEffect(() => {
10  if (!shouldPlay) playedRef.current = false
11}, [shouldPlay])

Multiple utterances — play the right one

typescript
1const sayStep1 = useTTS('First, look at the abacus', { tone: INST })
2const sayStep2 = useTTS('Now tap the bead', { tone: INST })
3
4// speak() stops previous before starting
5if (step === 0) sayStep1()
6if (step === 1) sayStep2()

Tone String Guidelines

Write tones as voice-actor stage directions. Be specific about emotion, pace, and audience.

Good examples:

• 'Speaking clearly and steadily, reading a math problem to a young child. Pause slightly between each number and operator.'
• 'Warmly congratulating a child. Genuinely encouraging and happy.'
• 'Gently guiding a child after a wrong answer. Kind, not disappointed.'
• 'Patiently guiding a young child through an abacus tutorial. Clear, slow, friendly.'

Bad examples:

• 'Read this text' — too vague
• `Speaking ${mood}` — dynamic per render, creates new clips every time

Anti-Patterns to Avoid

1. Never use raw speechSynthesis — always go through useTTS so the voice chain and collection work
2. Never forget cleanup — always useEffect(() => () => stop(), [stop])
3. Never use dynamic tone strings — keep them as module-level constants
4. Never call speak() unconditionally in render — always guard with refs and isEnabled

Key Files

Voice Chain

Audio plays through the voice chain in order. The typical chain is:

pregenerated voice (nova) → auto-generate → browser TTS → subtitles

• Pregenerated: instant playback from pre-generated mp3 on disk
• Auto-generate: calls OpenAI on-the-fly if the pregenerated mp3 is missing, caches result
• Browser TTS: uses the browser's built-in speech synthesis
• Subtitles: shows text on screen with a reading-time timer

You don't need to think about this when adding TTS to a feature — just use useTTS() and the chain handles fallback automatically. The admin configures the chain at /admin/audio.

Reference Implementations

Follow usePracticeAudioHelp as the most complete example.