Dubbing
Dubbing takes a video, translates its dialogue, speaks the translation in the original speakers' own voices and mixes it over the film's music and effects. Everything runs on your computer and nothing is uploaded; the only network use is the translation step, which works as it does on Translate (your key, or credits). It needs a license and, for now, Windows.
Install the Dubbing runtime
Dubbing needs the Dubbing runtime, a private Python environment with PyTorch and four voice models. Install it from Settings → Dependencies; the New dubbing dialog also tells you when it is missing. Finding who speaks when uses the separate Speaker detection download (about 55 MB).
- About 6.4 GB to download with an NVIDIA graphics card (3.5 GB of that is voice models) and about 9 GB of disk once installed.
- With an NVIDIA card it uses the card. Without one a smaller CPU build is installed, which is far slower.
- An interrupted install never damages a working one. Run Install again and only the missing files are fetched.
Starting a project
Open Dubbing in the main menu and choose New dubbing. It asks for:
- The video and its audio track — pick the one with the dialogue.
- The spoken language and Dub into: two different languages.
- The dialogue: a subtitle file you already have, or transcribe the audio. Your edits to the dialogue are never replaced later. Subtitles for the deaf and hard of hearing are fine: sound descriptions and name tags (
[sighs],(laughs),GOKU:) and song lyrics (♪) are taken out, and a subtitle that is only a sound is not a line, so the original sound plays there. A project made before this shows a Remove them notice that does it once. - The first time only, a confirmation that you have the rights and permission for the material.
Creating the project only writes its file. Dub on the next screen does the work in stages: prepare the audio, separate the dialogue from music and effects, find the speakers, choose voice samples, translate, speak every line, mix, and export. A percentage appears only when the tool really reports one. You can leave the screen and the run continues; the app asks before quitting during it. If you stop, close the app or it crashes, Dub continues from the first stage that is not finished, and lines already spoken are kept.
Speakers and voice samples
The app works out who speaks when and numbers the speakers. Rename a speaker to a character name, merge two that are the same person, or reassign single lines in the table. A film with overlapping speech, crowds or music will not be split perfectly — this screen is where you correct it, and your choices survive a re-run. Detect speakers again re-runs detection on an older project.
Each speaker’s voice is copied from one sample, 5 to 10 seconds of that person alone and clear. The app picks the best candidate and warns when the best it found is short, noisy or faint. Play it from the speaker’s card. A sample you choose yourself is never overridden. To keep voices for every video, see the voice library.
Translation and shorter alternatives
Lines are translated with the same service as Translate. Each line has a length budget from its time slot and the target language, because a voice speaks only so many characters a second at a natural pace. A line that comes out longer gets a shorter alternative beside it — click it to use it; it is never applied for you. You can edit any translation by hand, or import a translated subtitle file to use instead of the AI translation: each cue goes to the line it overlaps most.
Review states
| Not started | No translation or no voice sample yet. |
|---|---|
| Ready | Can be spoken, not yet generated. |
| Processing | Being spoken now. |
| Generated | Spoken and fits its slot. |
| Needs review | Spoken but longer than its slot, even sped up a little. |
| Approved | You listened and kept it. |
| Failed | The last attempt failed; the line says why. |
| Out of date | The audio was made from a text, sample or setting that has since changed. |
Approved audio is never replaced on its own. If its text or sample changes it becomes Out of date and waits for you; other out-of-date lines are regenerated on the next run. Lines you leave out are skipped, and the export refuses while any other line has no audio. Regenerate this line makes a new take, which differs a little from the last. A clip may be sped up by up to 15% or slowed by 8% to fit its slot; beyond that the line goes to Needs review rather than sounding processed.
Levels and the mix
Levels in the toolbar sets how far the original is lowered where a dubbed line plays, and the dub level relative to the original line it replaces. Each line is first matched to the loudness of the original, so this is one nudge for the whole film. Music and effects are never lowered. In a 5.1 film only the centre channel is replaced and the other channels pass through. The original dialogue is silenced only while a dubbed line plays; if separation fails, the whole original is lowered under each line instead. Changing a level re-mixes without regenerating any line.
Export
Export writes a video with the dub as an audio track. Video and audio are copied, not re-encoded, so it takes seconds. Choose Save as a new file beside the video (the default) or Replace the video; nothing is written over the original until the result has been checked. Keep the original audio adds the dub beside the original track, and Play the dub by default makes it the track a player starts with. AVI, WMV, FLV and MPEG cannot hold the result, so a new MKV is made and Replace is refused.
The rights notice and the watermark
Dubbing copies the voices in your video. You are responsible for having the rights and permission to use the video and those voices, and for how you use the result — copying a person’s voice without permission can be illegal or harmful. Every dubbed clip carries an inaudible watermark added by the voice engine (Chatterbox, MIT-licensed) that marks it as machine-generated; it cannot be turned off.
Limits
- Not lip sync. A line fits its time slot; mouths will not match the new words.
- Voice similarity depends on the sample: a clean 8 seconds of one person works best, and emotion is only partly carried over.
- Windows only for now, and an NVIDIA card is strongly recommended.
- Chatterbox speaks Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish and Turkish.
- Switching to another project restarts the voice engine, which loads again.