Lesson 5 of 7
Speaking every language
7 min read
A video recorded in English, now speaking fluent Ukrainian — same person, same voice, lips moving to match the new words. How does one clip learn to speak another language?
Three steps in one
Dubbing a video into another language used to mean a studio, translators and voice actors. AI folds it into a few steps you can run yourself. First it transcribes and translates what's said. Then it re-voices the translation — often in a voice that matches the original speaker. Finally it re-times the mouth so the lips fit the new words.
Translating a clip's speech and re-voicing it in another language is dubbing. AI can also match the new audio to the speaker's mouth.
Matching the mouth
That last step is the one that sells it. Every sound has a mouth shape — lips together for an m, round for an o, teeth for an f. When the language changes, the sounds change, so the mouth has to change too, or the video looks like a badly dubbed old movie. Aligning the mouth shape to each new sound is lip-sync. Play the scene and watch the mouth take the shape of each sound.
Lip-sync matches each mouth shape to the sound being spoken, so a re-voiced clip in a new language still looks natural.
Handy — and easy to fake with
The upside is huge: a course, an ad or a film can reach the whole world without reshoots, and a creator can 'speak' a language they don't. But the same pipeline — translate, re-voice, lip-match — is exactly what makes a convincing deepfake. The tools (HeyGen, ElevenLabs' dubbing and others) are powerful; use them on content you have the right to.
Dubbing puts new words in someone's mouth — literally. Only dub content you own or have permission to, be clear when a video is AI-dubbed, and remember the same trick makes deepfakes. Consent and disclosure keep it on the right side of the line.
The gist
- —AI dubbing = translate the speech, re-voice it in the new language, then lip-match the mouth to the new sounds
- —Lip-sync — matching each mouth shape to its sound — is what makes a dubbed clip look real, not like an old bad dub
- —It unlocks global reach without reshoots — but the same pipeline makes deepfakes, so use it with rights and disclosure
You dub an English explainer into Ukrainian and the audio is perfect, but the speaker's lips still move to the old English words. What's missing?
Continue in the app
Take the whole AI Voice & Speech course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.