AI might help edit the next generation of blockbusters - The Verge

Artist: William Joel

The next few Tuesdays, The Verge’s flagship podcast The Vergecast is showcasing a miniseries dedicated to the use of artificial intelligence in industries that are often overlooked, hosted by Verge senior reporter Ashley Carman. This week, the series focuses on AI for the video world.

More specifically, we’re looking at how AI is being used as a tool to help people streamline the process of creating video content. Yes, this might mean software taking on a bigger role in the very human act of creativity, but what if instead of replacing us, machine learning tools could be used to assist our work?

That’s what Scott Prevost, VP of Adobe Sensei — Adobe’s machine learning platform — envisions for Adobe’s AI products. “Sensei was founded on this firm belief that we have that AI is going to democratize and amplify human creativity, but not replace it,” Prevost says. “Ultimately, enabling the creator to do things that maybe they couldn’t do before. But also to automate and speed up some of the mundane and repetitive tasks that are parts of creativity.”

Adobe has already built Sensei’s initiatives into its current products. Last fall, the company released a feature called Neural Filters for Photoshop, which can be used to remove artifacts from compressed images, change the lighting in a photo, or even alter a subject’s face, giving them a smile instead of a frown, for example, or adjusting their “facial age.” From the user’s perspective, all this is done by just moving a few sliders.

Adobe’s Neural Filter

Image: Adobe

Adobe also has features like Content Aware Fill, which is built into its video editing software After Effects and can seamlessly remove objects from videos — a task that would take hours or even days to do manually. Prevost shared a story about a small team of documentary filmmakers who ran into trouble with their footage when they realized there were unwanted specks on their visual caused by a dirty camera lens. With Content Aware Fill, the team was able to remove the unwanted blemishes from the video after identifying the object in only a single frame. Without software like Adobe’s, the team would have had to edit thousands of frames individually or reshoot the footage entirely.

Adobe’s content-aware fill for video

GIF: Adobe

Another feature from Adobe called Auto Reframe uses AI to reformat and reframe video for different aspect ratios, keeping the important objects in frame that may have been cut out using a regular static crop.

GIF: Adobe

Adobe’s Auto Reframe feature

Technology in this field is clearly advancing for consumers, but also for the big-budget professionals, too. While AI video editing techniques like deepfakes have not really made it onto the big screen just yet — most studios still rely on traditional CGI — the place where directors and Hollywood studios are on the way to using AI is for dubbing.

A company called Flawless, which specializes in AI-driven VFX and film-making tools, is currently working on something they call TrueSync, which uses machine learning to create realistic, lip-synced visualizations on actors for multiple languages. Co-CEO and co-founder of Flawless Scott Mann told The Verge that this technique works significantly better than traditional CGI to reconstruct an actor’s mouth movements.

“You’re training a network to understand how one person speaks, so the mouth movements of an ooh and aah, different visemes and phonemes that make up our language are very person specific,” says Mann. “And that’s why it requires such detail in the process to really get something authentic that speaks like that person spoke like.”

An example Flawless shared that really stood out was a scene from the movie Forrest Gump, with a dub of Tom Hanks’ character speaking Japanese. The emotion of the character is still present and the end results are definitely more believable than a traditional overdub because the movement of the mouth is synchronized to the new dialogue. There are points where you almost forget that it’s another voice actor behind the scenes.

But as with any AI changing any industry, we also have to think about job replacement.

If someone is creating, editing, and publishing projects by themselves, then Adobe’s AI tools should save them a lot of time. But in larger production houses where each role is delegated to a specific specialist — retouchers, colorists, editors, social media managers — those teams may end up downsizing.

Adobe’s Prevost believes the technology will more likely shift jobs than completely destroy them. “We think some of the work that creatives used to do in production, they’re not going to do as much of that anymore,” he says. “They may become more like art directors. We think it actually allows the humans to focus more on the creative aspects of their work and to explore this broader creative space, where Sensei does some of the more mundane work.”

Will video teams end up downsizing?

Scott Mann at Flawless has a similar sentiment. Though the company’s technology may result in less of a need for script rewriters for translated movies, it may open up doors for new job opportunities, he argues. “I would say, truthfully, that role is kind of a director. What you’re doing is you’re trying to convey that performance. But I think with tech and really with this process, it’s going to be a case of taking that side of the industry and growing that side of the industry.”

Will script supervisors end up becoming directors? Or photo retouchers end up becoming art directors? Maybe. But what we are seeing for certain today is that a lot of these tools are already combining workflows from various points of the creative process. Audio mixing, coloring, and graphics are all becoming one part of multipurpose software. So, if you’re working in the visual media space, instead of specializing in specific creative talents, your creative job may instead require you to be more of a generalist in the future.

“I think the boundaries between images, and videos, and audio, and 3D, and augmented reality are going to start to blur,” says Prevost. “It used to be that there are people who specialized in images, and people who specialized in video, and now you see people working across all of these mediums. And so we think that Sensei will have a big role in basically helping to connect these things together in meaningful ways.”

The Vergecast The podcast you need to make sense of the week in tech news Subscribe