New product promises to capture pitch, volume, and speed of speech

Dubformer_Professional Dubbing_4

AI dubbing startup Dubformer has launched Emotion Transfer, a proprietary AI dubbing technology that aims to produce more emotional and expressive dubbing.

The product aims to capture pitch, volume, and speed of speech, instead of the voice cloning or pre-recorded videos currently used by many companies. It separates these aspects from the timbre, which it says is the unique part of an individual’s voice, and transfers them to the software.

It processes each phrase individually, using the original voice as a reference to capture the emotional essence of the performance, similar to the process professional dubbing actors follow before delivering their lines.

The technology cannot operate beyond the specific content it is used to process, and it doesn’t use actor data for training. Emotion Transfer is capable of translating 20 languages into English, German, Spanish, and French.

This aims to make the resulting voice much closer to the original, giving more emotional depth. Also, as it doesn’t use the original actor’s voice, it hopes to avoid issues around actors’ voices being cloned without consent or being used for other projects than the one they were originally recorded for.

Anton Dvorkovich, Dubformer’s CEO and founder, said: “Until now, AI dubbing has struggled to capture the emotional nuances that human voice actors bring to their performances. With Emotion Transfer, we elevate the entire AI dubbing process by focusing on the emotions behind each phrase, enhancing the expressiveness of the output. By transferring emotional dynamics instead of cloning voices, we’re able to create a result that’s both more engaging and true to the original performance. This approach represents a significant leap in AI dubbing technology,”