Dubbed content is an alien concept to the American or British television viewer, that is unless you watch Chinese martial arts films. However, for many countries around the world, such as Russia, China, France, Italy, and Spain for example, experience voice-over, dubbed content on a daily basis, heck it’s even part of the culture. This resulted in streaming services (like Netflix) or Hollywood film distributors to adapt their content and provide voice-over versions - even for famous Hollywood stars - in the native language. While we might recognize the distinctive voices of famous Hollywood actors, to people living in dubbed over countries, these famous voices mean nothing.
Now, an Israeli startup wants to toy with AI algorithms to create a dubbing platform for foreign languages but in the actor’s original voice - eliminating the need for local voice-over actors. Or if you want, “Deep Fake” the human voice.
Morgan Freeman but in Spanish
The software developed by Israeli company Deepdub allows content creators and production companies to localize content from one language to another by injecting the different phases with Deep Learning. In order to keep the actor’s unique voice, the platform learns and trains on different clips of the actor’s original voice, which it then converts to the target language. The service’s initial launch will include deep dubs in English, Spanish, French, and German.
While chatting with Geektime, CEO and co-founder Ofir Krakowski explains that development requires the combination of different technologies like advanced research of learning models, Deep Learning, signal processing, unique actor voice modeling, Neural style transfer, and NLP. The company, which is taking part in the NVIDIA Inception program, has access to the advanced NVIDIA hardware, GPUs, and software developments, such as Apex for PyTorch and CUDA, in addition to utilizing a multi-node training system: “Through these hardware and software solutions, we are able to achieve 100x the acceleration, both during research and machine learning, as well as in quality,” said Krakowski.
The Israeli startup has already successfully modeled one of the more well-known voices out there - American actor Morgan Freeman, whose familiar and unique deep tone voice now speaks fluent Spanish. Moreover, the company is super proud that their customers can choose if the originally American actor will speak proper Italian, or maybe Italian with an American accent, or maybe English with an Italian accent.
Eventually, the company plans to cut the long - 3 months - dubbing production into a two week process, at a fraction of the current cost.
deepdub was founded by Ofir Krakowski and his brother Nir Krakowski, who helped found the cyber sector of Shin Bet, and also participated in the Y Combinator accelerator program. The two tell that, as avid movie lovers, and after talking with key players in the industry, they realized that there is an actual need for a dubbing solution, and from here the rest is history.
The company, which was founded in 2019, emerged from stealth mode and has already secured Seed funding. Just recently, the company announced the recruitment of Kevin Reilly, former Chief Content Officer at HBO Max, to the advisory board. Reilly joins Emiliano Calemzuk, who was the former President of Fox Studios, and Eran Etam, co-founder of Kaltura and ICQ.